Compare commits
10 Commits
v0.7.2
...
1665359486
| Author | SHA1 | Date | |
|---|---|---|---|
| 1665359486 | |||
| 03f2543927 | |||
| fe9c348092 | |||
| f7f8f1a949 | |||
| d40c91acde | |||
| ed4dcf1ef7 | |||
| 24cce49a70 | |||
| 1e6db89dd4 | |||
| 0454296c81 | |||
| 58c6ab2d54 |
99
README.md
99
README.md
@@ -16,7 +16,6 @@ Implemented now:
|
||||
|
||||
Not implemented yet:
|
||||
|
||||
- `archive` stage behavior
|
||||
- `notify` stage behavior
|
||||
- additional analyze artifacts beyond `session_recap`
|
||||
- generic DAG orchestration
|
||||
@@ -47,6 +46,77 @@ Optional secrets-from-files config:
|
||||
|
||||
YAML decoding is strict (`KnownFields(true)`), so unknown fields fail fast.
|
||||
|
||||
## Storage And Archive Foundations
|
||||
|
||||
Narratio now includes configuration and path-model foundations for archive support, plus implemented prepare-stage S3 audio input.
|
||||
|
||||
Implemented foundations:
|
||||
|
||||
- `pipeline.storage.s3` config shape (`bucket`, `root_prefix`, `region`, `endpoint`, `force_path_style`)
|
||||
- `pipeline.spool` config shape (`root`, `delete_audio_after_archive`)
|
||||
- `pipeline.archive` config shape (`enabled`, `upload_run`, `promote_artifacts`)
|
||||
- promotion-rule validation (`from`/`to` required, relative-only paths, traversal rejected)
|
||||
- `session.campaign` requirement for campaign-aware path construction
|
||||
- optional `session.inputs.audio_s3.prefix` modeling and prepare-stage S3 audio download
|
||||
- run ID generation and S3/local path helper foundations
|
||||
- manifest run/path identity fields
|
||||
|
||||
Current defaults:
|
||||
|
||||
- `pipeline.storage.s3.root_prefix`: `dnd`
|
||||
- `pipeline.spool.root`: `/var/spool/narratio`
|
||||
- `pipeline.spool.delete_audio_after_archive`: `false`
|
||||
- `pipeline.archive.enabled`: `true`
|
||||
- `pipeline.archive.upload_run`: `true`
|
||||
- default `pipeline.archive.promote_artifacts`:
|
||||
- `transcripts/trimmed.json` -> `transcripts/trimmed.json` (`required: true`)
|
||||
- `artifacts/session_recap.md` -> `artifacts/session_recap.md` (`required: true`)
|
||||
|
||||
Current boundaries:
|
||||
|
||||
- local development audio (`audio_dir` / `audio_files`) still works
|
||||
- `audio_dir`/`audio_files` and `audio_s3` are mutually exclusive
|
||||
- real S3-compatible backend now exists in the storage adapter package
|
||||
- storage backend tests use fake storage and do not require live S3
|
||||
- archive uploads successful run records under `runs/{run_id}/`
|
||||
- archive does not upload local audio by default
|
||||
- archive uploads promoted outputs to session-level keys using `archive.promote_artifacts`
|
||||
- archive uploads `current/manifest.json`
|
||||
- archive uploads `current/run_id.txt` last as the effective commit marker
|
||||
- required missing promotions fail archive
|
||||
- optional missing promotions are skipped and recorded
|
||||
|
||||
S3 input details and current boundaries are documented in [docs/s3-audio-input.md](docs/s3-audio-input.md).
|
||||
|
||||
## Remote Storage Backend
|
||||
|
||||
Narratio includes an object-store backend layer for future prepare/archive work:
|
||||
|
||||
- `List(ctx, prefix)`
|
||||
- `Download(ctx, key, localPath)`
|
||||
- `Upload(ctx, localPath, key, opts)`
|
||||
- `Exists(ctx, key)`
|
||||
|
||||
Implemented backends:
|
||||
|
||||
- fake storage backend for deterministic tests
|
||||
- S3-compatible backend built from `pipeline.storage.s3`
|
||||
|
||||
Key invariant:
|
||||
|
||||
- callers pass full bucket-relative object keys
|
||||
- storage backends do not prepend `root_prefix` and do not infer session/campaign paths
|
||||
|
||||
Current boundary:
|
||||
|
||||
- `prepare` uses `List` + `Download` through the backend when `session.inputs.audio_s3` is configured
|
||||
- `archive` uses `Upload` through the backend for successful run-record uploads
|
||||
- `archive` also uses `Upload` for promotion writes and current pointers
|
||||
- no failed or incomplete runs are uploaded
|
||||
- local audio is not re-uploaded by default
|
||||
|
||||
Archive run-upload details and boundaries are documented in [docs/archive-storage.md](docs/archive-storage.md).
|
||||
|
||||
## Canonical Stage Order
|
||||
|
||||
1. `prepare`
|
||||
@@ -66,6 +136,33 @@ YAML decoding is strict (`KnownFields(true)`), so unknown fields fail fast.
|
||||
- `transcripts/normalized.json`: Seriatim-normalized transcript from the normalize stage
|
||||
- `transcripts/trimmed.json`: gameplay-only normalized polished transcript from trim stage
|
||||
|
||||
## Audita Configuration
|
||||
|
||||
`pipeline.audita` configures the real Audita subprocess adapter used by `polish`.
|
||||
|
||||
Required:
|
||||
|
||||
- `binary`
|
||||
- `timeout`
|
||||
- `base_url`
|
||||
- `model`
|
||||
|
||||
Optional:
|
||||
|
||||
- `llm_api_key_env` (when set, Narratio requires that env var and passes it to Audita as `AUDITA_LLM_API_KEY`)
|
||||
- `modules` override list (when empty/omitted, Narratio does not pass `--modules`)
|
||||
- `transcript_description`
|
||||
- `config_path`
|
||||
- `output_schema` (`bare-segments` or `audita-v1`)
|
||||
- `work_dir_retention` (`always`, `auto`, or `never`)
|
||||
- `total_llm_concurrency` (> 0 when provided)
|
||||
- `proposal_llm_concurrency` (> 0 when provided)
|
||||
- `validation_model`
|
||||
- `validation_llm_concurrency` (> 0 when provided)
|
||||
- `report` (defaults to `true`)
|
||||
|
||||
Narratio passes only configured optional Audita flags. Omitted optional values are left to Audita runtime defaults.
|
||||
|
||||
## Normalize Configuration
|
||||
|
||||
`pipeline.normalize` is optional. When omitted, Narratio defaults to:
|
||||
|
||||
129
architecture.md
129
architecture.md
@@ -22,10 +22,24 @@ Implemented:
|
||||
- real `trim` stage producing `transcripts/trimmed.json`
|
||||
- real `analyze` stage for initial `session_recap` generation
|
||||
- optional Scriptorium render diagnostics (`render_debug`) before production run
|
||||
- storage/archive configuration and validation foundations for:
|
||||
- `pipeline.storage.s3`
|
||||
- `pipeline.spool`
|
||||
- `pipeline.archive` promotion rules
|
||||
- `session.inputs.audio_s3`
|
||||
- run identity and path-model foundations:
|
||||
- run ID generation (`YYYYMMDDTHHMMSSZ-xxxxxxxx`)
|
||||
- S3 session/run/current key builders
|
||||
- campaign/session/run local work/spool path helpers
|
||||
- manifest run/path identity fields (`campaign`, `run_id`, local and S3 prefixes)
|
||||
- remote storage backend layer:
|
||||
- narrow object-store interface (`List`, `Download`, `Upload`, `Exists`)
|
||||
- fake storage backend for deterministic tests (no network dependency)
|
||||
- S3-compatible backend using AWS SDK v2
|
||||
- config-based object-store construction helper
|
||||
|
||||
Still placeholder/future:
|
||||
|
||||
- `archive` stage behavior
|
||||
- `notify` stage behavior
|
||||
- additional Scriptorium artifact types beyond `session_recap`
|
||||
- artifact-to-artifact workflows beyond the initial single-artifact implementation
|
||||
@@ -110,12 +124,125 @@ Optional pipeline secrets directory:
|
||||
- if configured, unreadable/missing `env_dir` fails command execution early
|
||||
- relative `env_dir` values are resolved from current working directory
|
||||
|
||||
Storage and archive foundations:
|
||||
|
||||
- `pipeline.storage.s3` is available for modeling S3 coordinates:
|
||||
- `bucket`
|
||||
- `root_prefix` (default `dnd`)
|
||||
- `region`
|
||||
- `endpoint`
|
||||
- `force_path_style` (default `false`)
|
||||
- `pipeline.spool.root` defaults to `/var/spool/narratio`
|
||||
- `pipeline.spool.delete_audio_after_archive` defaults to `false` (cleanup behavior not implemented yet)
|
||||
- `pipeline.archive` is optional and defaults to:
|
||||
- `enabled: true`
|
||||
- `upload_run: true`
|
||||
- default `promote_artifacts`:
|
||||
- `transcripts/trimmed.json`
|
||||
- `artifacts/session_recap.md`
|
||||
- archive promotion rules enforce safe relative paths:
|
||||
- `from` and `to` are required
|
||||
- absolute paths are rejected
|
||||
- traversal segments such as `..` are rejected
|
||||
|
||||
Session input foundations:
|
||||
|
||||
- `session.campaign` is required
|
||||
- local audio remains supported through `session.inputs.audio_dir` or `session.inputs.audio_files`
|
||||
- optional S3 audio input shape is `session.inputs.audio_s3.prefix`
|
||||
- `audio_dir`/`audio_files` and `audio_s3` are mutually exclusive
|
||||
- when `audio_s3` is configured, `prepare` lists and downloads `.flac` objects through the object-store backend
|
||||
|
||||
Cross-config validation scope:
|
||||
|
||||
- `pipeline.storage.s3.bucket` is required only when an S3-dependent feature is explicitly configured (for current foundations, that includes `session.inputs.audio_s3`, and archive upload intent when using `storage.backend: s3`)
|
||||
- no AWS credentials are stored in Narratio config; credential resolution remains an external runtime concern
|
||||
|
||||
Remote object-store backend scope:
|
||||
|
||||
- remote storage APIs are isolated to `internal/adapters/storage`
|
||||
- AWS SDK types remain contained within the S3 backend implementation package
|
||||
- S3 key/session path semantics remain outside the backend, with this invariant:
|
||||
- callers pass full bucket-relative object keys
|
||||
- backend methods do not prepend `root_prefix` or infer campaign/session/run paths
|
||||
- `prepare` now uses object-store `List` and `Download` for S3 audio input
|
||||
- `archive` now uses object-store `Upload` for successful run-record upload under the run prefix
|
||||
- `archive` now uses object-store `Upload` for promoted outputs and current pointers
|
||||
|
||||
Prepare S3 audio behavior (implemented):
|
||||
|
||||
- compute session prefix as `{root_prefix}/campaigns/{campaign}/sessions/{session_id}/`
|
||||
- resolve `session.inputs.audio_s3.prefix` under that session prefix
|
||||
- list objects under the computed audio prefix and filter `.flac` keys
|
||||
- fail clearly when no `.flac` objects are found
|
||||
- download selected objects to spool audio path:
|
||||
- `{spool.root}/{campaign}/{session_id}/{run_id}/audio/`
|
||||
- materialize audio files into workdir audio path:
|
||||
- `{workspace.root}/work/{campaign}/{session_id}/{run_id}/audio/`
|
||||
- record S3 provenance in manifest input records (bucket/key/metadata/local paths/checksum)
|
||||
- no AWS SDK types are used in stage code; storage implementation details stay in storage adapter packages
|
||||
|
||||
Archive publishing behavior (implemented):
|
||||
|
||||
- `archive` verifies prerequisite stage success before upload:
|
||||
- `prepare`, `transcribe`, `merge`, `polish`, `normalize`, `trim`, `analyze`
|
||||
- only successful/completed runs are uploaded
|
||||
- uploaded run record destination is:
|
||||
- `{root_prefix}/campaigns/{campaign}/sessions/{session_id}/runs/{run_id}/`
|
||||
- uploaded existing local paths include:
|
||||
- `inputs/`, `transcripts/`, `artifacts/`, optional `reports/`, `config/`, `logs/`, and `manifest.json`
|
||||
- local `audio/` is intentionally excluded from upload by default
|
||||
- file upload order is deterministic (sorted relative paths)
|
||||
- `archive.enabled: false` and `archive.upload_run: false` skip upload cleanly
|
||||
- stage metadata records non-secret upload context:
|
||||
- run upload details, promoted output details, current manifest key, current pointer key
|
||||
- no secrets, transcript contents, prompt contents, or environment dumps
|
||||
- promotion rules:
|
||||
- `from` resolves from local workdir
|
||||
- `to` resolves under session-level S3 root
|
||||
- missing required source fails archive
|
||||
- missing optional source is skipped and recorded
|
||||
- default promoted outputs:
|
||||
- `transcripts/trimmed.json`
|
||||
- `artifacts/session_recap.md`
|
||||
- current pointers:
|
||||
- `current/manifest.json` uploaded after run upload and promotions
|
||||
- `current/run_id.txt` uploaded last with `{run_id}\n`
|
||||
- `current/run_id.txt` is the effective commit marker
|
||||
- if promotion or current-manifest upload fails, archive returns failure and does not write `current/run_id.txt`
|
||||
- failed/incomplete runs remain local and are not uploaded
|
||||
|
||||
`pipeline.scriptorium` is optional. Existing pipelines without Scriptorium continue to work.
|
||||
|
||||
`pipeline.trim` is optional. Existing pipelines without trim config continue to work.
|
||||
|
||||
`pipeline.normalize` is optional. Existing pipelines without normalize config continue to work.
|
||||
|
||||
`pipeline.audita` drives the real Audita subprocess adapter for the `polish` stage.
|
||||
|
||||
Audita required fields:
|
||||
|
||||
- `binary`
|
||||
- `timeout`
|
||||
- `base_url`
|
||||
- `model`
|
||||
|
||||
Audita optional fields:
|
||||
|
||||
- `llm_api_key_env` (enforced only when configured)
|
||||
- `modules` override list (when omitted/empty, Narratio does not pass `--modules`)
|
||||
- `transcript_description`
|
||||
- `config_path`
|
||||
- `output_schema` (`bare-segments` or `audita-v1`)
|
||||
- `work_dir_retention` (`always`, `auto`, `never`)
|
||||
- `total_llm_concurrency` (> 0 when provided)
|
||||
- `proposal_llm_concurrency` (> 0 when provided)
|
||||
- `validation_model`
|
||||
- `validation_llm_concurrency` (> 0 when provided)
|
||||
- `report` (default `true`)
|
||||
|
||||
Narratio passes only configured optional Audita flags; omitted optional values defer to Audita defaults.
|
||||
|
||||
When `pipeline.normalize` is omitted, defaults are applied:
|
||||
|
||||
- `output_path: transcripts/normalized.json`
|
||||
|
||||
115
docs/archive-storage.md
Normal file
115
docs/archive-storage.md
Normal file
@@ -0,0 +1,115 @@
|
||||
# Archive Storage
|
||||
|
||||
This document describes implemented archive-stage publish behavior.
|
||||
|
||||
## S3 Paths
|
||||
|
||||
Session root:
|
||||
|
||||
`{root_prefix}/campaigns/{campaign}/sessions/{session_id}/`
|
||||
|
||||
Run prefix:
|
||||
|
||||
`{root_prefix}/campaigns/{campaign}/sessions/{session_id}/runs/{run_id}/`
|
||||
|
||||
## Scope
|
||||
|
||||
Implemented:
|
||||
|
||||
- archive uploads successful run records to remote object storage through the storage backend abstraction.
|
||||
- archive uploads configured promoted outputs to session-level keys.
|
||||
- archive uploads `current/manifest.json`.
|
||||
- archive uploads `current/run_id.txt` last as the effective commit marker.
|
||||
- tests use fake storage and do not require live S3.
|
||||
|
||||
Future work:
|
||||
|
||||
- spool audio cleanup/deletion behavior
|
||||
- `notify` stage behavior
|
||||
- stale detection
|
||||
- optional future source-audio upload mode
|
||||
- additional artifact generation beyond current implemented set
|
||||
|
||||
## Prerequisites
|
||||
|
||||
Archive verifies these stages succeeded before upload:
|
||||
|
||||
- `prepare`
|
||||
- `transcribe`
|
||||
- `merge`
|
||||
- `polish`
|
||||
- `normalize`
|
||||
- `trim`
|
||||
- `analyze`
|
||||
|
||||
If any prerequisite is missing or not succeeded, archive fails and does not upload.
|
||||
Failed or incomplete runs remain local only.
|
||||
|
||||
## Run Upload
|
||||
|
||||
Archive uploads existing files from the run workdir when present:
|
||||
|
||||
- `inputs/`
|
||||
- `transcripts/`
|
||||
- `artifacts/`
|
||||
- `reports/` (optional)
|
||||
- `config/`
|
||||
- `logs/`
|
||||
- `manifest.json`
|
||||
|
||||
Relative paths are preserved under `runs/{run_id}/`.
|
||||
|
||||
## Promotion Rules
|
||||
|
||||
Archive applies `archive.promote_artifacts` in config order.
|
||||
|
||||
Rule behavior:
|
||||
|
||||
- `from`: local workdir-relative source path
|
||||
- `to`: session-root-relative destination key
|
||||
- `required: true`: missing source fails archive
|
||||
- `required: false`: missing source is skipped and recorded
|
||||
|
||||
Default promoted outputs:
|
||||
|
||||
- `transcripts/trimmed.json`
|
||||
- `artifacts/session_recap.md`
|
||||
|
||||
## Current Pointers
|
||||
|
||||
Archive writes:
|
||||
|
||||
1. `current/manifest.json` (after run upload + promotions)
|
||||
2. `current/run_id.txt` last
|
||||
|
||||
`current/run_id.txt` contains exactly:
|
||||
|
||||
- `{run_id}` plus trailing newline
|
||||
|
||||
Writing `current/run_id.txt` last makes it the effective commit marker for published session state.
|
||||
|
||||
If any required run upload, promotion upload, or current-manifest upload fails, archive returns failure and does not write `current/run_id.txt`.
|
||||
|
||||
## Audio Upload Policy
|
||||
|
||||
Archive does not upload local `audio/` by default.
|
||||
Original audio is expected at the session-level audio prefix and is not duplicated under `runs/{run_id}/`.
|
||||
|
||||
## Config Controls
|
||||
|
||||
- `archive.enabled: false` skips archive cleanly.
|
||||
- `archive.upload_run: false` skips run upload cleanly.
|
||||
|
||||
## Metadata
|
||||
|
||||
Archive stage metadata includes non-secret upload context (for example):
|
||||
|
||||
- `s3_bucket`
|
||||
- `s3_run_prefix`
|
||||
- run upload counts/paths
|
||||
- promoted upload counts/paths
|
||||
- skipped optional promotions
|
||||
- `current_manifest_key`
|
||||
- `current_run_id_key`
|
||||
- `current_pointer_written`
|
||||
- `audio_upload_skipped`
|
||||
@@ -79,6 +79,51 @@ current/run_id.txt:
|
||||
written last as the effective S3 commit pointer
|
||||
```
|
||||
|
||||
### 2.1 Implementation Status (2026-05-16)
|
||||
|
||||
Implemented in repository:
|
||||
|
||||
- storage/archive configuration and validation foundations:
|
||||
- `storage.s3`
|
||||
- `spool`
|
||||
- `archive`
|
||||
- promotion-rule safety checks
|
||||
- `inputs.audio_s3` modeling
|
||||
- run and path-model foundations:
|
||||
- run ID generation (`YYYYMMDDTHHMMSSZ-xxxxxxxx`)
|
||||
- S3 key builders for session/run/current/promoted destinations
|
||||
- campaign/session/run local work and spool path helpers
|
||||
- manifest run/path identity fields
|
||||
- examples and tests for the above foundations
|
||||
- remote storage backend layer:
|
||||
- object-store abstraction with `List`, `Download`, `Upload`, and `Exists`
|
||||
- fake storage backend for deterministic, no-network testing
|
||||
- S3-compatible backend built from `storage.s3` config
|
||||
- backend construction helper from resolved config
|
||||
- archive run upload behavior:
|
||||
- archive validates required prior stage success before uploading
|
||||
- archive uploads successful run records under `runs/{run_id}/`
|
||||
- upload set includes run-record files (`inputs`, `transcripts`, `artifacts`, optional `reports`, `config`, `logs`, `manifest.json`)
|
||||
- local audio is not uploaded by default
|
||||
- upload uses storage backend abstraction and deterministic ordering
|
||||
- archive skips cleanly when `archive.enabled` or `archive.upload_run` is false
|
||||
- archive promotion and current publish behavior:
|
||||
- promotion rules upload configured outputs to session-level destinations
|
||||
- required missing promotion sources fail archive
|
||||
- optional missing promotion sources are skipped and recorded
|
||||
- `current/manifest.json` is uploaded after run upload and promotions
|
||||
- `current/run_id.txt` is uploaded last as the effective commit marker
|
||||
- current pointer content is `{run_id}` plus trailing newline
|
||||
- if promotion/current manifest upload fails, current pointer is not written
|
||||
|
||||
Not implemented yet:
|
||||
|
||||
- spool audio cleanup / deletion behavior
|
||||
- `notify` stage behavior
|
||||
- generic stale detection based on input/config checksums
|
||||
- optional future mode for uploading source audio from local workspace/spool
|
||||
- additional artifact generation beyond current implemented set
|
||||
|
||||
## 3. S3 Layout
|
||||
|
||||
The canonical S3 layout should be:
|
||||
@@ -285,7 +330,7 @@ work/audio
|
||||
transcribe
|
||||
```
|
||||
|
||||
The `prepare` stage should:
|
||||
Implemented `prepare` behavior:
|
||||
|
||||
1. list `.flac` objects under the configured S3 audio prefix,
|
||||
2. fail clearly if none are found,
|
||||
@@ -302,7 +347,11 @@ spool:
|
||||
delete_audio_after_archive: true
|
||||
```
|
||||
|
||||
For v1, prefer retaining local workdir audio until the run has successfully archived. The source of truth remains S3, but local diagnostics are valuable during development.
|
||||
Current boundary:
|
||||
|
||||
- downloaded audio is retained in spool/workdir
|
||||
- spool cleanup policy remains future work
|
||||
- archive does not upload source audio by default
|
||||
|
||||
## 6. Configuration Design
|
||||
|
||||
@@ -801,9 +850,9 @@ Test:
|
||||
- `run-stage` can find or require a run ID according to final CLI policy
|
||||
- multiple local runs are handled deterministically
|
||||
|
||||
## 15. Implementation Phases
|
||||
## 15. Implementation Sequence
|
||||
|
||||
### Phase 1: Config and Path Model
|
||||
### Config and Path Model (Implemented)
|
||||
|
||||
Implement:
|
||||
|
||||
@@ -823,9 +872,9 @@ Expected commit:
|
||||
Add archive storage path configuration
|
||||
```
|
||||
|
||||
### Phase 2: Storage Backend Interface and S3 Backend
|
||||
### Storage Backend Interface and S3 Backend (Implemented)
|
||||
|
||||
Implement:
|
||||
Implemented:
|
||||
|
||||
- storage backend interface
|
||||
- object metadata type
|
||||
@@ -833,7 +882,7 @@ Implement:
|
||||
- real S3 backend using AWS SDK or existing project dependency policy
|
||||
- backend construction from config
|
||||
|
||||
No stage behavior yet.
|
||||
No prepare/archive stage behavior yet.
|
||||
|
||||
Expected commit:
|
||||
|
||||
@@ -841,7 +890,7 @@ Expected commit:
|
||||
Add S3 storage backend abstraction
|
||||
```
|
||||
|
||||
### Phase 3: Prepare Stage S3 Audio Download
|
||||
### Prepare Stage S3 Audio Download
|
||||
|
||||
Implement:
|
||||
|
||||
@@ -858,7 +907,7 @@ Expected commit:
|
||||
Download S3 audio during prepare"
|
||||
```
|
||||
|
||||
### Phase 4: Real Archive Stage Run Upload
|
||||
### Real Archive Stage Run Upload
|
||||
|
||||
Implement:
|
||||
|
||||
@@ -873,7 +922,7 @@ Expected commit:
|
||||
Upload successful run records to S3
|
||||
```
|
||||
|
||||
### Phase 5: Promotion Rules and Current Pointer
|
||||
### Promotion Rules and Current Pointer
|
||||
|
||||
Implement:
|
||||
|
||||
@@ -890,7 +939,7 @@ Expected commit:
|
||||
Promote current session artifacts to S3
|
||||
```
|
||||
|
||||
### Phase 6: Documentation and Examples
|
||||
### Documentation and Examples
|
||||
|
||||
Update:
|
||||
|
||||
@@ -907,7 +956,7 @@ Expected commit:
|
||||
Document S3 archive workflow
|
||||
```
|
||||
|
||||
### Phase 7: Architectural Review
|
||||
### Architectural Review
|
||||
|
||||
Review:
|
||||
|
||||
|
||||
35
docs/runbooks/s3-archive-foundations.md
Normal file
35
docs/runbooks/s3-archive-foundations.md
Normal file
@@ -0,0 +1,35 @@
|
||||
# S3 Archive Foundations Runbook
|
||||
|
||||
This runbook documents the currently implemented storage/archive foundations and the boundaries of current behavior.
|
||||
|
||||
## Implemented Now
|
||||
|
||||
- config modeling for:
|
||||
- `pipeline.storage.s3`
|
||||
- `pipeline.spool`
|
||||
- `pipeline.archive`
|
||||
- `session.inputs.audio_s3`
|
||||
- promotion rule validation for safe relative paths
|
||||
- run ID generation and path/key helper functions
|
||||
- manifest run/path identity fields
|
||||
- remote storage backend layer:
|
||||
- object-store interface (`List`, `Download`, `Upload`, `Exists`)
|
||||
- fake backend for deterministic tests
|
||||
- S3-compatible backend using AWS SDK v2
|
||||
- config-based backend construction helper
|
||||
|
||||
## Not Implemented Yet
|
||||
|
||||
- prepare-stage S3 object listing or download
|
||||
- archive-stage S3 upload or promotion writes
|
||||
- writing `current/manifest.json` or `current/run_id.txt` in S3
|
||||
|
||||
## Operational Notes
|
||||
|
||||
- local audio workflows remain the active development path (`audio_dir` or `audio_files`)
|
||||
- `audio_s3` and local audio config are mutually exclusive
|
||||
- do not place AWS credentials in Narratio config files
|
||||
|
||||
## Next Implementation Target
|
||||
|
||||
Use the storage backend layer in prepare-stage session audio discovery/download flow, while preserving local audio input support.
|
||||
83
docs/s3-audio-input.md
Normal file
83
docs/s3-audio-input.md
Normal file
@@ -0,0 +1,83 @@
|
||||
# S3 Audio Input
|
||||
|
||||
This document describes implemented S3 audio input behavior in `prepare`.
|
||||
|
||||
## Scope
|
||||
|
||||
Implemented:
|
||||
|
||||
- `prepare` can acquire source audio from S3 when `session.inputs.audio_s3.prefix` is configured.
|
||||
- object listing and download go through the storage backend abstraction.
|
||||
- tests use fake storage; no live S3 service is required for test runs.
|
||||
|
||||
Not implemented:
|
||||
|
||||
- spool cleanup/deletion behavior
|
||||
- uploads of failed runs
|
||||
|
||||
## Required Configuration
|
||||
|
||||
`pipeline.yml`:
|
||||
|
||||
- `storage.s3.bucket` must be set when S3 audio input is used.
|
||||
- `storage.s3.root_prefix` defaults to `dnd`.
|
||||
- `spool.root` defaults to `/var/spool/narratio`.
|
||||
|
||||
`session.yml`:
|
||||
|
||||
- configure `session.campaign` and `session.session_id`.
|
||||
- configure `session.inputs.audio_s3.prefix` for S3 audio input.
|
||||
- do not configure `inputs.audio_dir` or `inputs.audio_files` at the same time as `inputs.audio_s3`.
|
||||
|
||||
## Prefix Shape
|
||||
|
||||
Session S3 root:
|
||||
|
||||
`{root_prefix}/campaigns/{campaign}/sessions/{session_id}/`
|
||||
|
||||
Audio prefix:
|
||||
|
||||
`{session_root}/{audio_s3.prefix}`
|
||||
|
||||
Example:
|
||||
|
||||
`dnd/campaigns/forsaken/sessions/2026-04-19/audio/`
|
||||
|
||||
Audio files must already exist in S3 before running Narratio.
|
||||
|
||||
## Prepare Behavior
|
||||
|
||||
When `inputs.audio_s3.prefix` is configured, `prepare`:
|
||||
|
||||
1. lists objects under the computed S3 audio prefix
|
||||
2. filters to `.flac` objects
|
||||
3. fails when no `.flac` objects are found
|
||||
4. downloads selected objects to spool audio:
|
||||
- `{spool.root}/{campaign}/{session_id}/{run_id}/audio/`
|
||||
5. materializes audio into workdir audio:
|
||||
- `{workspace.root}/work/{campaign}/{session_id}/{run_id}/audio/`
|
||||
6. records input provenance in the manifest (bucket, key, metadata, local paths, checksum)
|
||||
|
||||
Notes:
|
||||
|
||||
- `.flac` filtering is case-insensitive.
|
||||
- ETag is recorded as provider metadata only and is not treated as a checksum.
|
||||
|
||||
## Local Audio Development
|
||||
|
||||
Local audio workflows remain supported:
|
||||
|
||||
- `inputs.audio_dir`
|
||||
- `inputs.audio_files`
|
||||
|
||||
These options are mutually exclusive with `inputs.audio_s3`.
|
||||
|
||||
## Archive Boundary
|
||||
|
||||
Current archive behavior relevant to S3 audio input:
|
||||
|
||||
- successful runs are uploaded by archive under `runs/{run_id}/`
|
||||
- configured promotions are uploaded to session-level destinations
|
||||
- `current/manifest.json` and `current/run_id.txt` are published
|
||||
- local source audio is not re-uploaded by default
|
||||
- failed or incomplete runs are not uploaded
|
||||
58
docs/storage-backends.md
Normal file
58
docs/storage-backends.md
Normal file
@@ -0,0 +1,58 @@
|
||||
# Storage Backends
|
||||
|
||||
This document describes the currently implemented remote object storage backend layer used by Narratio, and its intended role in later prepare/archive work.
|
||||
|
||||
## Implemented
|
||||
|
||||
Remote object store abstraction:
|
||||
|
||||
- `List(ctx, prefix)`
|
||||
- `Download(ctx, key, localPath)`
|
||||
- `Upload(ctx, localPath, key, opts)`
|
||||
- `Exists(ctx, key)`
|
||||
|
||||
Object metadata model includes:
|
||||
|
||||
- key
|
||||
- size
|
||||
- ETag (provider metadata only)
|
||||
- last modified time when available
|
||||
|
||||
Backends:
|
||||
|
||||
- fake storage backend for deterministic tests
|
||||
- S3-compatible backend implemented with AWS SDK for Go v2
|
||||
|
||||
Construction:
|
||||
|
||||
- config-based constructor builds S3 backend from `pipeline.storage.s3` values:
|
||||
- bucket
|
||||
- region
|
||||
- endpoint
|
||||
- force_path_style
|
||||
|
||||
## Key Invariant
|
||||
|
||||
- callers pass full bucket-relative object keys
|
||||
- storage backends do not prepend `root_prefix`
|
||||
- storage backends do not infer campaign/session/run paths
|
||||
|
||||
S3 session/run key builders remain separate and continue to live outside backend implementations.
|
||||
|
||||
## Security Boundary
|
||||
|
||||
- do not store AWS credentials in Narratio config
|
||||
- AWS credentials are resolved through standard AWS SDK credential chains
|
||||
- AWS SDK-specific types remain isolated to the storage adapter package
|
||||
|
||||
## Testing
|
||||
|
||||
- fake storage tests cover list/download/upload/exists and error paths
|
||||
- S3 backend tests use injected fake S3 API clients
|
||||
- tests do not require live S3 services, AWS credentials, or network access
|
||||
|
||||
## Not Implemented Yet
|
||||
|
||||
- prepare-stage S3 object listing or downloads
|
||||
- archive-stage S3 uploads or promotion writes
|
||||
- writing `current/manifest.json` or `current/run_id.txt` to S3
|
||||
@@ -3,6 +3,27 @@ workspace:
|
||||
|
||||
storage:
|
||||
backend: local
|
||||
s3:
|
||||
bucket: "my-dnd-archive"
|
||||
root_prefix: "dnd"
|
||||
region: "us-east-1"
|
||||
endpoint: ""
|
||||
force_path_style: false
|
||||
|
||||
spool:
|
||||
root: "/var/spool/narratio"
|
||||
delete_audio_after_archive: false
|
||||
|
||||
archive:
|
||||
enabled: true
|
||||
upload_run: true
|
||||
promote_artifacts:
|
||||
- from: "transcripts/trimmed.json"
|
||||
to: "transcripts/trimmed.json"
|
||||
required: true
|
||||
- from: "artifacts/session_recap.md"
|
||||
to: "artifacts/session_recap.md"
|
||||
required: true
|
||||
|
||||
secrets:
|
||||
# Optional: load environment variables from files in this directory.
|
||||
@@ -33,17 +54,16 @@ audita:
|
||||
binary: "audita"
|
||||
timeout: "3h"
|
||||
llm_api_key_env: "AUDITA_LLM_API_KEY"
|
||||
modules:
|
||||
- glossary
|
||||
- homophones
|
||||
- glossary
|
||||
- spoken_word
|
||||
- grammar
|
||||
- homophones
|
||||
- glossary
|
||||
# Optional: pass only when overriding Audita's default module sequence.
|
||||
modules: []
|
||||
base_url: "https://openrouter.ai/api/v1"
|
||||
model: "openrouter/google/gemma-4-31b-it"
|
||||
llm_concurrency: 1
|
||||
transcript_description: ""
|
||||
config_path: ""
|
||||
output_schema: "audita-v1"
|
||||
work_dir_retention: "auto"
|
||||
total_llm_concurrency: 1
|
||||
proposal_llm_concurrency: 1
|
||||
validation_model: ""
|
||||
validation_llm_concurrency: 1
|
||||
report: true
|
||||
|
||||
@@ -4,7 +4,10 @@ date: 2026-05-03
|
||||
title: Sample Session
|
||||
inputs:
|
||||
audio_dir: ./audio
|
||||
# Optional S3 input alternative. Do not configure with audio_dir/audio_files.
|
||||
# Narratio prepare lists this prefix and downloads .flac files.
|
||||
# audio_s3:
|
||||
# prefix: "audio/"
|
||||
speakers_file: ./speakers.yml
|
||||
autocorrect_file: ./autocorrect.yml
|
||||
glossary_file: ./glossary.yml
|
||||
|
||||
|
||||
25
go.mod
25
go.mod
@@ -2,4 +2,27 @@ module gitea.maximumdirect.net/eric/narratio
|
||||
|
||||
go 1.25.0
|
||||
|
||||
require gopkg.in/yaml.v3 v3.0.1
|
||||
require (
|
||||
github.com/aws/aws-sdk-go-v2/config v1.32.17
|
||||
github.com/aws/aws-sdk-go-v2/service/s3 v1.101.0
|
||||
github.com/aws/smithy-go v1.25.1
|
||||
gopkg.in/yaml.v3 v3.0.1
|
||||
)
|
||||
|
||||
require (
|
||||
github.com/aws/aws-sdk-go-v2 v1.41.7 // indirect
|
||||
github.com/aws/aws-sdk-go-v2/aws/protocol/eventstream v1.7.10 // indirect
|
||||
github.com/aws/aws-sdk-go-v2/credentials v1.19.16 // indirect
|
||||
github.com/aws/aws-sdk-go-v2/feature/ec2/imds v1.18.23 // indirect
|
||||
github.com/aws/aws-sdk-go-v2/internal/configsources v1.4.23 // indirect
|
||||
github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.7.23 // indirect
|
||||
github.com/aws/aws-sdk-go-v2/internal/v4a v1.4.24 // indirect
|
||||
github.com/aws/aws-sdk-go-v2/service/internal/accept-encoding v1.13.9 // indirect
|
||||
github.com/aws/aws-sdk-go-v2/service/internal/checksum v1.9.15 // indirect
|
||||
github.com/aws/aws-sdk-go-v2/service/internal/presigned-url v1.13.23 // indirect
|
||||
github.com/aws/aws-sdk-go-v2/service/internal/s3shared v1.19.23 // indirect
|
||||
github.com/aws/aws-sdk-go-v2/service/signin v1.0.11 // indirect
|
||||
github.com/aws/aws-sdk-go-v2/service/sso v1.30.17 // indirect
|
||||
github.com/aws/aws-sdk-go-v2/service/ssooidc v1.35.21 // indirect
|
||||
github.com/aws/aws-sdk-go-v2/service/sts v1.42.1 // indirect
|
||||
)
|
||||
|
||||
36
go.sum
36
go.sum
@@ -1,3 +1,39 @@
|
||||
github.com/aws/aws-sdk-go-v2 v1.41.7 h1:DWpAJt66FmnnaRIOT/8ASTucrvuDPZASqhhLey6tLY8=
|
||||
github.com/aws/aws-sdk-go-v2 v1.41.7/go.mod h1:4LAfZOPHNVNQEckOACQx60Y8pSRjIkNZQz1w92xpMJc=
|
||||
github.com/aws/aws-sdk-go-v2/aws/protocol/eventstream v1.7.10 h1:gx1AwW1Iyk9Z9dD9F4akX5gnN3QZwUB20GGKH/I+Rho=
|
||||
github.com/aws/aws-sdk-go-v2/aws/protocol/eventstream v1.7.10/go.mod h1:qqY157uZoqm5OXq/amuaBJyC9hgBCBQnsaWnPe905GY=
|
||||
github.com/aws/aws-sdk-go-v2/config v1.32.17 h1:FpL4/758/diKwqbytU0prpuiu60fgXKUWCpDJtApclU=
|
||||
github.com/aws/aws-sdk-go-v2/config v1.32.17/go.mod h1:OXqUMzgXytfoF9JaKkhrOYsyh72t9G+MJH8mMRaexOE=
|
||||
github.com/aws/aws-sdk-go-v2/credentials v1.19.16 h1:r3RJBuU7X9ibt8RHbMjWE6y60QbKBiII6wSrXnapxSU=
|
||||
github.com/aws/aws-sdk-go-v2/credentials v1.19.16/go.mod h1:6cx7zqDENJDbBIIWX6P8s0h6hqHC8Avbjh9Dseo27ug=
|
||||
github.com/aws/aws-sdk-go-v2/feature/ec2/imds v1.18.23 h1:UuSfcORqNSz/ey3VPRS8TcVH2Ikf0/sC+Hdj400QI6U=
|
||||
github.com/aws/aws-sdk-go-v2/feature/ec2/imds v1.18.23/go.mod h1:+G/OSGiOFnSOkYloKj/9M35s74LgVAdJBSD5lsFfqKg=
|
||||
github.com/aws/aws-sdk-go-v2/internal/configsources v1.4.23 h1:GpT/TrnBYuE5gan2cZbTtvP+JlHsutdmlV2YfEyNde0=
|
||||
github.com/aws/aws-sdk-go-v2/internal/configsources v1.4.23/go.mod h1:xYWD6BS9ywC5bS3sz9Xh04whO/hzK2plt2Zkyrp4JuA=
|
||||
github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.7.23 h1:bpd8vxhlQi2r1hiueOw02f/duEPTMK59Q4QMAoTTtTo=
|
||||
github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.7.23/go.mod h1:15DfR2nw+CRHIk0tqNyifu3G1YdAOy68RftkhMDDwYk=
|
||||
github.com/aws/aws-sdk-go-v2/internal/v4a v1.4.24 h1:OQqn11BtaYv1WLUowvcA30MpzIu8Ti4pcLPIIyoKZrA=
|
||||
github.com/aws/aws-sdk-go-v2/internal/v4a v1.4.24/go.mod h1:X5ZJyfwVrWA96GzPmUCWFQaEARPR7gCrpq2E92PJwAE=
|
||||
github.com/aws/aws-sdk-go-v2/service/internal/accept-encoding v1.13.9 h1:FLudkZLt5ci0ozzgkVo8BJGwvqNaZbTWb3UcucAateA=
|
||||
github.com/aws/aws-sdk-go-v2/service/internal/accept-encoding v1.13.9/go.mod h1:w7wZ/s9qK7c8g4al+UyoF1Sp/Z45UwMGcqIzLWVQHWk=
|
||||
github.com/aws/aws-sdk-go-v2/service/internal/checksum v1.9.15 h1:ieLCO1JxUWuxTZ1cRd0GAaeX7O6cIxnwk7tc1LsQhC4=
|
||||
github.com/aws/aws-sdk-go-v2/service/internal/checksum v1.9.15/go.mod h1:e3IzZvQ3kAWNykvE0Tr0RDZCMFInMvhku3qNpcIQXhM=
|
||||
github.com/aws/aws-sdk-go-v2/service/internal/presigned-url v1.13.23 h1:pbrxO/kuIwgEsOPLkaHu0O+m4fNgLU8B3vxQ+72jTPw=
|
||||
github.com/aws/aws-sdk-go-v2/service/internal/presigned-url v1.13.23/go.mod h1:/CMNUqoj46HpS3MNRDEDIwcgEnrtZlKRaHNaHxIFpNA=
|
||||
github.com/aws/aws-sdk-go-v2/service/internal/s3shared v1.19.23 h1:03xatSQO4+AM1lTAbnRg5OK528EUg744nW7F73U8DKw=
|
||||
github.com/aws/aws-sdk-go-v2/service/internal/s3shared v1.19.23/go.mod h1:M8l3mwgx5ToK7wot2sBBce/ojzgnPzZXUV445gTSyE8=
|
||||
github.com/aws/aws-sdk-go-v2/service/s3 v1.101.0 h1:etqBTKY581iwLL/H/S2sVgk3C9lAsTJFeXWFDsDcWOU=
|
||||
github.com/aws/aws-sdk-go-v2/service/s3 v1.101.0/go.mod h1:L2dcoOgS2VSgbPLvpak2NyUPsO1TBN7M45Z4H7DlRc4=
|
||||
github.com/aws/aws-sdk-go-v2/service/signin v1.0.11 h1:TdJ+HdzOBhU8+iVAOGUTU63VXopcumCOF1paFulHWZc=
|
||||
github.com/aws/aws-sdk-go-v2/service/signin v1.0.11/go.mod h1:R82ZRExE/nheo0N+T8zHPcLRTcH8MGsnR3BiVGX0TwI=
|
||||
github.com/aws/aws-sdk-go-v2/service/sso v1.30.17 h1:7byT8HUWrgoRp6sXjxtZwgOKfhss5fW6SkLBtqzgRoE=
|
||||
github.com/aws/aws-sdk-go-v2/service/sso v1.30.17/go.mod h1:xNWknVi4Ezm1vg1QsB/5EWpAJURq22uqd38U8qKvOJc=
|
||||
github.com/aws/aws-sdk-go-v2/service/ssooidc v1.35.21 h1:+1Kl1zx6bWi4X7cKi3VYh29h8BvsCoHQEQ6ST9X8w7w=
|
||||
github.com/aws/aws-sdk-go-v2/service/ssooidc v1.35.21/go.mod h1:4vIRDq+CJB2xFAXZ+YgGUTiEft7oAQlhIs71xcSeuVg=
|
||||
github.com/aws/aws-sdk-go-v2/service/sts v1.42.1 h1:F/M5Y9I3nwr2IEpshZgh1GeHpOItExNM9L1euNuh/fk=
|
||||
github.com/aws/aws-sdk-go-v2/service/sts v1.42.1/go.mod h1:mTNxImtovCOEEuD65mKW7DCsL+2gjEH+RPEAexAzAio=
|
||||
github.com/aws/smithy-go v1.25.1 h1:J8ERsGSU7d+aCmdQur5Txg6bVoYelvQJgtZehD12GkI=
|
||||
github.com/aws/smithy-go v1.25.1/go.mod h1:YE2RhdIuDbA5E5bTdciG9KrW3+TiEONeUWCqxX9i1Fc=
|
||||
gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405 h1:yhCVgyC4o1eVCa2tZl7eS0r+SDo693bJlVdllGtEeKM=
|
||||
gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
|
||||
gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA=
|
||||
|
||||
@@ -24,6 +24,12 @@ type PolishRequest struct {
|
||||
Modules []string
|
||||
BaseURL string
|
||||
Model string
|
||||
TranscriptDescription string
|
||||
ConfigPath string
|
||||
OutputSchema string
|
||||
WorkDirRetention string
|
||||
TotalLLMConcurrency *int
|
||||
ProposalLLMConcurrency *int
|
||||
ValidationModel string
|
||||
ValidationLLMConcurrency *int
|
||||
StdoutLogPath string
|
||||
|
||||
@@ -21,7 +21,12 @@ type SubprocessRunnerConfig struct {
|
||||
Modules []string
|
||||
BaseURL string
|
||||
Model string
|
||||
LLMConcurrency *int
|
||||
TranscriptDescription string
|
||||
ConfigPath string
|
||||
OutputSchema string
|
||||
WorkDirRetention string
|
||||
TotalLLMConcurrency *int
|
||||
ProposalLLMConcurrency *int
|
||||
ValidationModel string
|
||||
ValidationLLMConcurrency *int
|
||||
Report bool
|
||||
@@ -35,7 +40,12 @@ type SubprocessRunner struct {
|
||||
modules []string
|
||||
baseURL string
|
||||
model string
|
||||
llmConcurrency *int
|
||||
transcriptDescription string
|
||||
configPath string
|
||||
outputSchema string
|
||||
workDirRetention string
|
||||
totalLLMConcurrency *int
|
||||
proposalLLMConcurrency *int
|
||||
validationModel string
|
||||
validationLLMConcurrency *int
|
||||
report bool
|
||||
@@ -49,7 +59,12 @@ func NewSubprocessRunnerFromConfigValues(
|
||||
modules []string,
|
||||
baseURL string,
|
||||
model string,
|
||||
llmConcurrency *int,
|
||||
transcriptDescription string,
|
||||
configPath string,
|
||||
outputSchema string,
|
||||
workDirRetention string,
|
||||
totalLLMConcurrency *int,
|
||||
proposalLLMConcurrency *int,
|
||||
validationModel string,
|
||||
validationLLMConcurrency *int,
|
||||
report bool,
|
||||
@@ -68,7 +83,12 @@ func NewSubprocessRunnerFromConfigValues(
|
||||
Modules: modules,
|
||||
BaseURL: baseURL,
|
||||
Model: model,
|
||||
LLMConcurrency: llmConcurrency,
|
||||
TranscriptDescription: transcriptDescription,
|
||||
ConfigPath: configPath,
|
||||
OutputSchema: outputSchema,
|
||||
WorkDirRetention: workDirRetention,
|
||||
TotalLLMConcurrency: totalLLMConcurrency,
|
||||
ProposalLLMConcurrency: proposalLLMConcurrency,
|
||||
ValidationModel: validationModel,
|
||||
ValidationLLMConcurrency: validationLLMConcurrency,
|
||||
Report: report,
|
||||
@@ -83,9 +103,6 @@ func NewSubprocessRunner(cfg SubprocessRunnerConfig) (*SubprocessRunner, error)
|
||||
if cfg.Timeout <= 0 {
|
||||
return nil, fmt.Errorf("audita timeout must be > 0")
|
||||
}
|
||||
if len(cfg.Modules) == 0 {
|
||||
return nil, fmt.Errorf("audita modules must include at least one module")
|
||||
}
|
||||
for i, module := range cfg.Modules {
|
||||
if strings.TrimSpace(module) == "" {
|
||||
return nil, fmt.Errorf("audita module at index %d is empty", i)
|
||||
@@ -104,12 +121,25 @@ func NewSubprocessRunner(cfg SubprocessRunnerConfig) (*SubprocessRunner, error)
|
||||
if strings.TrimSpace(cfg.Model) == "" {
|
||||
return nil, fmt.Errorf("audita model is required")
|
||||
}
|
||||
if cfg.LLMConcurrency != nil && *cfg.LLMConcurrency <= 0 {
|
||||
return nil, fmt.Errorf("audita llm concurrency must be > 0 when provided")
|
||||
if cfg.TotalLLMConcurrency != nil && *cfg.TotalLLMConcurrency <= 0 {
|
||||
return nil, fmt.Errorf("audita total llm concurrency must be > 0 when provided")
|
||||
}
|
||||
if cfg.ProposalLLMConcurrency != nil && *cfg.ProposalLLMConcurrency <= 0 {
|
||||
return nil, fmt.Errorf("audita proposal llm concurrency must be > 0 when provided")
|
||||
}
|
||||
if cfg.ValidationLLMConcurrency != nil && *cfg.ValidationLLMConcurrency <= 0 {
|
||||
return nil, fmt.Errorf("audita validation llm concurrency must be > 0 when provided")
|
||||
}
|
||||
switch strings.TrimSpace(cfg.OutputSchema) {
|
||||
case "", "bare-segments", "audita-v1":
|
||||
default:
|
||||
return nil, fmt.Errorf("audita output schema must be one of: bare-segments, audita-v1")
|
||||
}
|
||||
switch strings.TrimSpace(cfg.WorkDirRetention) {
|
||||
case "", "always", "auto", "never":
|
||||
default:
|
||||
return nil, fmt.Errorf("audita work dir retention must be one of: always, auto, never")
|
||||
}
|
||||
|
||||
modules := make([]string, len(cfg.Modules))
|
||||
for i, m := range cfg.Modules {
|
||||
@@ -123,7 +153,12 @@ func NewSubprocessRunner(cfg SubprocessRunnerConfig) (*SubprocessRunner, error)
|
||||
modules: modules,
|
||||
baseURL: strings.TrimSpace(cfg.BaseURL),
|
||||
model: strings.TrimSpace(cfg.Model),
|
||||
llmConcurrency: cfg.LLMConcurrency,
|
||||
transcriptDescription: strings.TrimSpace(cfg.TranscriptDescription),
|
||||
configPath: strings.TrimSpace(cfg.ConfigPath),
|
||||
outputSchema: strings.TrimSpace(cfg.OutputSchema),
|
||||
workDirRetention: strings.TrimSpace(cfg.WorkDirRetention),
|
||||
totalLLMConcurrency: cfg.TotalLLMConcurrency,
|
||||
proposalLLMConcurrency: cfg.ProposalLLMConcurrency,
|
||||
validationModel: strings.TrimSpace(cfg.ValidationModel),
|
||||
validationLLMConcurrency: cfg.ValidationLLMConcurrency,
|
||||
report: cfg.Report,
|
||||
@@ -152,7 +187,7 @@ func (r *SubprocessRunner) Run(ctx context.Context, req PolishRequest) (PolishRe
|
||||
}
|
||||
|
||||
reqModules := req.Modules
|
||||
if len(reqModules) == 0 {
|
||||
if reqModules == nil {
|
||||
reqModules = append([]string(nil), r.modules...)
|
||||
}
|
||||
args := r.buildArgs(req, reqModules)
|
||||
@@ -168,14 +203,9 @@ func (r *SubprocessRunner) Run(ctx context.Context, req PolishRequest) (PolishRe
|
||||
env["AUDITA_LLM_API_KEY"] = credential
|
||||
credentialPresent = true
|
||||
}
|
||||
primaryConcurrencyViaEnv := false
|
||||
if r.llmConcurrency != nil {
|
||||
env["AUDITA_LLM_CONCURRENCY"] = strconv.Itoa(*r.llmConcurrency)
|
||||
primaryConcurrencyViaEnv = true
|
||||
}
|
||||
|
||||
if req.GeneratedConfigPath != "" {
|
||||
if err := r.writeInvocationConfig(req, args, reqModules, credentialPresent, primaryConcurrencyViaEnv); err != nil {
|
||||
if err := r.writeInvocationConfig(req, args, reqModules, credentialPresent); err != nil {
|
||||
return PolishResult{}, fmt.Errorf("write audita invocation config %q: %w", req.GeneratedConfigPath, err)
|
||||
}
|
||||
}
|
||||
@@ -196,7 +226,7 @@ func (r *SubprocessRunner) Run(ctx context.Context, req PolishRequest) (PolishRe
|
||||
req.StderrLogPath,
|
||||
)
|
||||
wrappedMessage = addSubprocessStreamHint(wrappedMessage, err)
|
||||
return r.failureResult(req, reqModules, runRes, credentialPresent, primaryConcurrencyViaEnv), fmt.Errorf(
|
||||
return r.failureResult(req, reqModules, runRes, credentialPresent), fmt.Errorf(
|
||||
"%s: %w",
|
||||
wrappedMessage,
|
||||
err,
|
||||
@@ -204,11 +234,11 @@ func (r *SubprocessRunner) Run(ctx context.Context, req PolishRequest) (PolishRe
|
||||
}
|
||||
|
||||
if err := validateProcessedOutput(req.OutputProcessedPath); err != nil {
|
||||
return r.failureResult(req, reqModules, runRes, credentialPresent, primaryConcurrencyViaEnv), fmt.Errorf("validate audita processed output %q: %w", req.OutputProcessedPath, err)
|
||||
return r.failureResult(req, reqModules, runRes, credentialPresent), fmt.Errorf("validate audita processed output %q: %w", req.OutputProcessedPath, err)
|
||||
}
|
||||
if r.report {
|
||||
if err := validateJSONFile(req.ReportPath); err != nil {
|
||||
return r.failureResult(req, reqModules, runRes, credentialPresent, primaryConcurrencyViaEnv), fmt.Errorf("validate audita report output %q: %w", req.ReportPath, err)
|
||||
return r.failureResult(req, reqModules, runRes, credentialPresent), fmt.Errorf("validate audita report output %q: %w", req.ReportPath, err)
|
||||
}
|
||||
}
|
||||
|
||||
@@ -223,21 +253,25 @@ func (r *SubprocessRunner) Run(ctx context.Context, req PolishRequest) (PolishRe
|
||||
Duration: runRes.Duration,
|
||||
InvokedBinary: r.binary,
|
||||
Metadata: map[string]any{
|
||||
"adapter": "audita_subprocess",
|
||||
"modules": reqModules,
|
||||
"base_url": r.baseURL,
|
||||
"model": r.model,
|
||||
"validation_model": r.validationModel,
|
||||
"validation_llm_concurrency": r.validationLLMConcurrency,
|
||||
"credential_env_var": r.llmAPIKeyEnv,
|
||||
"credential_present": credentialPresent,
|
||||
"primary_llm_concurrency_via_env": primaryConcurrencyViaEnv,
|
||||
"primary_llm_concurrency_env_name": "AUDITA_LLM_CONCURRENCY",
|
||||
"adapter": "audita_subprocess",
|
||||
"modules": reqModules,
|
||||
"base_url": r.baseURL,
|
||||
"model": r.model,
|
||||
"transcript_description": r.transcriptDescription,
|
||||
"config_path": r.configPath,
|
||||
"output_schema": r.outputSchema,
|
||||
"work_dir_retention": r.workDirRetention,
|
||||
"validation_model": r.validationModel,
|
||||
"total_llm_concurrency": r.totalLLMConcurrency,
|
||||
"proposal_llm_concurrency": r.proposalLLMConcurrency,
|
||||
"validation_llm_concurrency": r.validationLLMConcurrency,
|
||||
"credential_env_var": r.llmAPIKeyEnv,
|
||||
"credential_present": credentialPresent,
|
||||
},
|
||||
}, nil
|
||||
}
|
||||
|
||||
func (r *SubprocessRunner) failureResult(req PolishRequest, modules []string, runRes subprocess.RunResult, credentialPresent bool, primaryConcurrencyViaEnv bool) PolishResult {
|
||||
func (r *SubprocessRunner) failureResult(req PolishRequest, modules []string, runRes subprocess.RunResult, credentialPresent bool) PolishResult {
|
||||
return PolishResult{
|
||||
ProcessedTranscriptPath: req.OutputProcessedPath,
|
||||
ReportPath: req.ReportPath,
|
||||
@@ -249,16 +283,20 @@ func (r *SubprocessRunner) failureResult(req PolishRequest, modules []string, ru
|
||||
Duration: runRes.Duration,
|
||||
InvokedBinary: r.binary,
|
||||
Metadata: map[string]any{
|
||||
"adapter": "audita_subprocess",
|
||||
"modules": modules,
|
||||
"base_url": r.baseURL,
|
||||
"model": r.model,
|
||||
"validation_model": r.validationModel,
|
||||
"validation_llm_concurrency": r.validationLLMConcurrency,
|
||||
"credential_env_var": r.llmAPIKeyEnv,
|
||||
"credential_present": credentialPresent,
|
||||
"primary_llm_concurrency_via_env": primaryConcurrencyViaEnv,
|
||||
"primary_llm_concurrency_env_name": "AUDITA_LLM_CONCURRENCY",
|
||||
"adapter": "audita_subprocess",
|
||||
"modules": modules,
|
||||
"base_url": r.baseURL,
|
||||
"model": r.model,
|
||||
"transcript_description": r.transcriptDescription,
|
||||
"config_path": r.configPath,
|
||||
"output_schema": r.outputSchema,
|
||||
"work_dir_retention": r.workDirRetention,
|
||||
"validation_model": r.validationModel,
|
||||
"total_llm_concurrency": r.totalLLMConcurrency,
|
||||
"proposal_llm_concurrency": r.proposalLLMConcurrency,
|
||||
"validation_llm_concurrency": r.validationLLMConcurrency,
|
||||
"credential_env_var": r.llmAPIKeyEnv,
|
||||
"credential_present": credentialPresent,
|
||||
},
|
||||
}
|
||||
}
|
||||
@@ -269,14 +307,34 @@ func (r *SubprocessRunner) buildArgs(req PolishRequest, modules []string) []stri
|
||||
req.MergedTranscriptPath,
|
||||
"--glossary", req.GlossaryPath,
|
||||
"--output", req.OutputProcessedPath,
|
||||
"--modules", strings.Join(modules, ","),
|
||||
"--base-url", r.baseURL,
|
||||
"--model", r.model,
|
||||
"--work-dir", req.WorkDir,
|
||||
}
|
||||
if len(modules) > 0 {
|
||||
args = append(args, "--modules", strings.Join(modules, ","))
|
||||
}
|
||||
if r.report {
|
||||
args = append(args, "--report-json", req.ReportPath)
|
||||
}
|
||||
if r.transcriptDescription != "" {
|
||||
args = append(args, "--transcript-description", r.transcriptDescription)
|
||||
}
|
||||
if r.configPath != "" {
|
||||
args = append(args, "--config", r.configPath)
|
||||
}
|
||||
if r.outputSchema != "" {
|
||||
args = append(args, "--output-schema", r.outputSchema)
|
||||
}
|
||||
if r.workDirRetention != "" {
|
||||
args = append(args, "--work-dir-retention", r.workDirRetention)
|
||||
}
|
||||
if r.totalLLMConcurrency != nil {
|
||||
args = append(args, "--total-llm-concurrency", strconv.Itoa(*r.totalLLMConcurrency))
|
||||
}
|
||||
if r.proposalLLMConcurrency != nil {
|
||||
args = append(args, "--proposal-llm-concurrency", strconv.Itoa(*r.proposalLLMConcurrency))
|
||||
}
|
||||
if r.validationModel != "" {
|
||||
args = append(args, "--validation-model", r.validationModel)
|
||||
}
|
||||
@@ -286,29 +344,31 @@ func (r *SubprocessRunner) buildArgs(req PolishRequest, modules []string) []stri
|
||||
return args
|
||||
}
|
||||
|
||||
func (r *SubprocessRunner) writeInvocationConfig(req PolishRequest, args []string, modules []string, credentialPresent bool, primaryConcurrencyViaEnv bool) error {
|
||||
func (r *SubprocessRunner) writeInvocationConfig(req PolishRequest, args []string, modules []string, credentialPresent bool) error {
|
||||
payload := map[string]any{
|
||||
"schema": "audita.generated.v1",
|
||||
"binary": r.binary,
|
||||
"args": args,
|
||||
"timeout": r.timeout.String(),
|
||||
"modules": modules,
|
||||
"base_url": r.baseURL,
|
||||
"model": r.model,
|
||||
"validation_model": r.validationModel,
|
||||
"validation_llm_concurrency": r.validationLLMConcurrency,
|
||||
"report_enabled": r.report,
|
||||
"merged_transcript_path": req.MergedTranscriptPath,
|
||||
"glossary_path": req.GlossaryPath,
|
||||
"output_path": req.OutputProcessedPath,
|
||||
"report_path": req.ReportPath,
|
||||
"work_dir": req.WorkDir,
|
||||
"credential_env_var": r.llmAPIKeyEnv,
|
||||
"credential_present": credentialPresent,
|
||||
"primary_llm_concurrency_via_env": primaryConcurrencyViaEnv,
|
||||
}
|
||||
if r.llmConcurrency != nil {
|
||||
payload["llm_concurrency"] = *r.llmConcurrency
|
||||
"schema": "audita.generated.v1",
|
||||
"binary": r.binary,
|
||||
"args": args,
|
||||
"timeout": r.timeout.String(),
|
||||
"modules": modules,
|
||||
"base_url": r.baseURL,
|
||||
"model": r.model,
|
||||
"transcript_description": r.transcriptDescription,
|
||||
"config_path": r.configPath,
|
||||
"output_schema": r.outputSchema,
|
||||
"work_dir_retention": r.workDirRetention,
|
||||
"validation_model": r.validationModel,
|
||||
"total_llm_concurrency": r.totalLLMConcurrency,
|
||||
"proposal_llm_concurrency": r.proposalLLMConcurrency,
|
||||
"validation_llm_concurrency": r.validationLLMConcurrency,
|
||||
"report_enabled": r.report,
|
||||
"merged_transcript_path": req.MergedTranscriptPath,
|
||||
"glossary_path": req.GlossaryPath,
|
||||
"output_path": req.OutputProcessedPath,
|
||||
"report_path": req.ReportPath,
|
||||
"work_dir": req.WorkDir,
|
||||
"credential_env_var": r.llmAPIKeyEnv,
|
||||
"credential_present": credentialPresent,
|
||||
}
|
||||
return subprocess.WriteYAMLAtomic(req.GeneratedConfigPath, payload, 0o644)
|
||||
}
|
||||
|
||||
@@ -25,7 +25,8 @@ func TestSubprocessRunnerSuccessArgsEnvAndValidation(t *testing.T) {
|
||||
t.Setenv("AUDITA_HELPER_RECORD_PATH", recordPath)
|
||||
|
||||
wrapper := writeAuditaHelperWrapper(t)
|
||||
llmConcurrency := 1
|
||||
totalLLMConcurrency := 3
|
||||
proposalLLMConcurrency := 2
|
||||
validationLLMConcurrency := 2
|
||||
runner, err := NewSubprocessRunner(SubprocessRunnerConfig{
|
||||
Binary: wrapper,
|
||||
@@ -34,7 +35,12 @@ func TestSubprocessRunnerSuccessArgsEnvAndValidation(t *testing.T) {
|
||||
Modules: []string{"glossary", "homophones", "glossary"},
|
||||
BaseURL: "https://openrouter.ai/api/v1",
|
||||
Model: "openrouter/google/gemma-4-31b-it",
|
||||
LLMConcurrency: &llmConcurrency,
|
||||
TranscriptDescription: "Campaign Session 42",
|
||||
ConfigPath: "/etc/audita/config.yml",
|
||||
OutputSchema: "audita-v1",
|
||||
WorkDirRetention: "auto",
|
||||
TotalLLMConcurrency: &totalLLMConcurrency,
|
||||
ProposalLLMConcurrency: &proposalLLMConcurrency,
|
||||
ValidationModel: "openrouter/google/gemma-4-31b-it",
|
||||
ValidationLLMConcurrency: &validationLLMConcurrency,
|
||||
Report: true,
|
||||
@@ -93,11 +99,17 @@ func TestSubprocessRunnerSuccessArgsEnvAndValidation(t *testing.T) {
|
||||
"process", req.MergedTranscriptPath,
|
||||
"--glossary", req.GlossaryPath,
|
||||
"--output", req.OutputProcessedPath,
|
||||
"--modules", "glossary,homophones,glossary",
|
||||
"--base-url", "https://openrouter.ai/api/v1",
|
||||
"--model", "openrouter/google/gemma-4-31b-it",
|
||||
"--work-dir", req.WorkDir,
|
||||
"--modules", "glossary,homophones,glossary",
|
||||
"--report-json", req.ReportPath,
|
||||
"--transcript-description", "Campaign Session 42",
|
||||
"--config", "/etc/audita/config.yml",
|
||||
"--output-schema", "audita-v1",
|
||||
"--work-dir-retention", "auto",
|
||||
"--total-llm-concurrency", "3",
|
||||
"--proposal-llm-concurrency", "2",
|
||||
"--validation-model", "openrouter/google/gemma-4-31b-it",
|
||||
"--validation-llm-concurrency", "2",
|
||||
}
|
||||
@@ -107,8 +119,8 @@ func TestSubprocessRunnerSuccessArgsEnvAndValidation(t *testing.T) {
|
||||
if rec.Env["AUDITA_LLM_API_KEY"] != "super-secret" {
|
||||
t.Fatalf("AUDITA_LLM_API_KEY = %q, want propagated secret", rec.Env["AUDITA_LLM_API_KEY"])
|
||||
}
|
||||
if rec.Env["AUDITA_LLM_CONCURRENCY"] != "1" {
|
||||
t.Fatalf("AUDITA_LLM_CONCURRENCY = %q, want 1", rec.Env["AUDITA_LLM_CONCURRENCY"])
|
||||
if rec.Env["AUDITA_LLM_CONCURRENCY"] != "" {
|
||||
t.Fatalf("AUDITA_LLM_CONCURRENCY = %q, want empty/omitted", rec.Env["AUDITA_LLM_CONCURRENCY"])
|
||||
}
|
||||
|
||||
cfgData, err := os.ReadFile(req.GeneratedConfigPath)
|
||||
@@ -124,16 +136,14 @@ func TestSubprocessRunnerMissingConfiguredCredentialFails(t *testing.T) {
|
||||
if runtime.GOOS == "windows" {
|
||||
t.Skip("helper wrapper script uses /bin/sh")
|
||||
}
|
||||
llmConcurrency := 1
|
||||
runner := mustAuditaRunner(t, SubprocessRunnerConfig{
|
||||
Binary: writeAuditaHelperWrapper(t),
|
||||
Timeout: mustParseAuditaDuration(t, "2s"),
|
||||
LLMAPIKeyEnv: "MISSING_AUDITA_KEY",
|
||||
Modules: []string{"glossary"},
|
||||
BaseURL: "https://openrouter.ai/api/v1",
|
||||
Model: "openrouter/google/gemma-4-31b-it",
|
||||
LLMConcurrency: &llmConcurrency,
|
||||
Report: false,
|
||||
Binary: writeAuditaHelperWrapper(t),
|
||||
Timeout: mustParseAuditaDuration(t, "2s"),
|
||||
LLMAPIKeyEnv: "MISSING_AUDITA_KEY",
|
||||
Modules: []string{"glossary"},
|
||||
BaseURL: "https://openrouter.ai/api/v1",
|
||||
Model: "openrouter/google/gemma-4-31b-it",
|
||||
Report: false,
|
||||
})
|
||||
req := auditaReqForTest(t, false)
|
||||
|
||||
@@ -155,16 +165,14 @@ func TestSubprocessRunnerUnconfiguredCredentialEnvOmitsCredential(t *testing.T)
|
||||
recordPath := filepath.Join(t.TempDir(), "record.json")
|
||||
t.Setenv("AUDITA_HELPER_RECORD_PATH", recordPath)
|
||||
|
||||
llmConcurrency := 1
|
||||
runner := mustAuditaRunner(t, SubprocessRunnerConfig{
|
||||
Binary: writeAuditaHelperWrapper(t),
|
||||
Timeout: mustParseAuditaDuration(t, "2s"),
|
||||
LLMAPIKeyEnv: "",
|
||||
Modules: []string{"glossary"},
|
||||
BaseURL: "https://openrouter.ai/api/v1",
|
||||
Model: "openrouter/google/gemma-4-31b-it",
|
||||
LLMConcurrency: &llmConcurrency,
|
||||
Report: false,
|
||||
Binary: writeAuditaHelperWrapper(t),
|
||||
Timeout: mustParseAuditaDuration(t, "2s"),
|
||||
LLMAPIKeyEnv: "",
|
||||
Modules: []string{"glossary"},
|
||||
BaseURL: "https://openrouter.ai/api/v1",
|
||||
Model: "openrouter/google/gemma-4-31b-it",
|
||||
Report: false,
|
||||
})
|
||||
req := auditaReqForTest(t, false)
|
||||
|
||||
@@ -211,6 +219,35 @@ func TestSubprocessRunnerInheritsParentEnvironment(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
func TestSubprocessRunnerOmitsModulesFlagWhenNotConfigured(t *testing.T) {
|
||||
if runtime.GOOS == "windows" {
|
||||
t.Skip("helper wrapper script uses /bin/sh")
|
||||
}
|
||||
t.Setenv("GO_WANT_AUDITA_HELPER", "1")
|
||||
t.Setenv("AUDITA_HELPER_MODE", "success")
|
||||
recordPath := filepath.Join(t.TempDir(), "record.json")
|
||||
t.Setenv("AUDITA_HELPER_RECORD_PATH", recordPath)
|
||||
|
||||
runner := mustAuditaRunner(t, SubprocessRunnerConfig{
|
||||
Binary: writeAuditaHelperWrapper(t),
|
||||
Timeout: mustParseAuditaDuration(t, "2s"),
|
||||
LLMAPIKeyEnv: "",
|
||||
BaseURL: "https://openrouter.ai/api/v1",
|
||||
Model: "openrouter/google/gemma-4-31b-it",
|
||||
Report: false,
|
||||
})
|
||||
req := auditaReqForTest(t, false)
|
||||
if _, err := runner.Run(context.Background(), req); err != nil {
|
||||
t.Fatalf("Run() error = %v", err)
|
||||
}
|
||||
rec := readAuditaHelperRecord(t, recordPath)
|
||||
for i := 0; i < len(rec.Args); i++ {
|
||||
if rec.Args[i] == "--modules" {
|
||||
t.Fatalf("args contained --modules unexpectedly: %#v", rec.Args)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestSubprocessRunnerSubprocessFailure(t *testing.T) {
|
||||
if runtime.GOOS == "windows" {
|
||||
t.Skip("helper wrapper script uses /bin/sh")
|
||||
@@ -220,16 +257,14 @@ func TestSubprocessRunnerSubprocessFailure(t *testing.T) {
|
||||
t.Setenv("OPENAI_KEY_SOURCE", "super-secret")
|
||||
t.Setenv("AUDITA_HELPER_RECORD_PATH", filepath.Join(t.TempDir(), "record.json"))
|
||||
|
||||
llmConcurrency := 1
|
||||
runner := mustAuditaRunner(t, SubprocessRunnerConfig{
|
||||
Binary: writeAuditaHelperWrapper(t),
|
||||
Timeout: mustParseAuditaDuration(t, "2s"),
|
||||
LLMAPIKeyEnv: "OPENAI_KEY_SOURCE",
|
||||
Modules: []string{"glossary"},
|
||||
BaseURL: "https://openrouter.ai/api/v1",
|
||||
Model: "openrouter/google/gemma-4-31b-it",
|
||||
LLMConcurrency: &llmConcurrency,
|
||||
Report: true,
|
||||
Binary: writeAuditaHelperWrapper(t),
|
||||
Timeout: mustParseAuditaDuration(t, "2s"),
|
||||
LLMAPIKeyEnv: "OPENAI_KEY_SOURCE",
|
||||
Modules: []string{"glossary"},
|
||||
BaseURL: "https://openrouter.ai/api/v1",
|
||||
Model: "openrouter/google/gemma-4-31b-it",
|
||||
Report: true,
|
||||
})
|
||||
req := auditaReqForTest(t, true)
|
||||
_, err := runner.Run(context.Background(), req)
|
||||
@@ -256,16 +291,14 @@ func TestSubprocessRunnerSubprocessFailureAddsStderrDescriptorHint(t *testing.T)
|
||||
t.Setenv("OPENAI_KEY_SOURCE", "super-secret")
|
||||
t.Setenv("AUDITA_HELPER_RECORD_PATH", filepath.Join(t.TempDir(), "record.json"))
|
||||
|
||||
llmConcurrency := 1
|
||||
runner := mustAuditaRunner(t, SubprocessRunnerConfig{
|
||||
Binary: writeAuditaHelperWrapper(t),
|
||||
Timeout: mustParseAuditaDuration(t, "2s"),
|
||||
LLMAPIKeyEnv: "OPENAI_KEY_SOURCE",
|
||||
Modules: []string{"glossary"},
|
||||
BaseURL: "https://openrouter.ai/api/v1",
|
||||
Model: "openrouter/google/gemma-4-31b-it",
|
||||
LLMConcurrency: &llmConcurrency,
|
||||
Report: true,
|
||||
Binary: writeAuditaHelperWrapper(t),
|
||||
Timeout: mustParseAuditaDuration(t, "2s"),
|
||||
LLMAPIKeyEnv: "OPENAI_KEY_SOURCE",
|
||||
Modules: []string{"glossary"},
|
||||
BaseURL: "https://openrouter.ai/api/v1",
|
||||
Model: "openrouter/google/gemma-4-31b-it",
|
||||
Report: true,
|
||||
})
|
||||
req := auditaReqForTest(t, true)
|
||||
_, err := runner.Run(context.Background(), req)
|
||||
@@ -286,16 +319,14 @@ func TestSubprocessRunnerMissingOutputFails(t *testing.T) {
|
||||
t.Setenv("OPENAI_KEY_SOURCE", "super-secret")
|
||||
t.Setenv("AUDITA_HELPER_RECORD_PATH", filepath.Join(t.TempDir(), "record.json"))
|
||||
|
||||
llmConcurrency := 1
|
||||
runner := mustAuditaRunner(t, SubprocessRunnerConfig{
|
||||
Binary: writeAuditaHelperWrapper(t),
|
||||
Timeout: mustParseAuditaDuration(t, "2s"),
|
||||
LLMAPIKeyEnv: "OPENAI_KEY_SOURCE",
|
||||
Modules: []string{"glossary"},
|
||||
BaseURL: "https://openrouter.ai/api/v1",
|
||||
Model: "openrouter/google/gemma-4-31b-it",
|
||||
LLMConcurrency: &llmConcurrency,
|
||||
Report: false,
|
||||
Binary: writeAuditaHelperWrapper(t),
|
||||
Timeout: mustParseAuditaDuration(t, "2s"),
|
||||
LLMAPIKeyEnv: "OPENAI_KEY_SOURCE",
|
||||
Modules: []string{"glossary"},
|
||||
BaseURL: "https://openrouter.ai/api/v1",
|
||||
Model: "openrouter/google/gemma-4-31b-it",
|
||||
Report: false,
|
||||
})
|
||||
req := auditaReqForTest(t, false)
|
||||
_, err := runner.Run(context.Background(), req)
|
||||
@@ -316,16 +347,14 @@ func TestSubprocessRunnerInvalidOutputJSONFails(t *testing.T) {
|
||||
t.Setenv("OPENAI_KEY_SOURCE", "super-secret")
|
||||
t.Setenv("AUDITA_HELPER_RECORD_PATH", filepath.Join(t.TempDir(), "record.json"))
|
||||
|
||||
llmConcurrency := 1
|
||||
runner := mustAuditaRunner(t, SubprocessRunnerConfig{
|
||||
Binary: writeAuditaHelperWrapper(t),
|
||||
Timeout: mustParseAuditaDuration(t, "2s"),
|
||||
LLMAPIKeyEnv: "OPENAI_KEY_SOURCE",
|
||||
Modules: []string{"glossary"},
|
||||
BaseURL: "https://openrouter.ai/api/v1",
|
||||
Model: "openrouter/google/gemma-4-31b-it",
|
||||
LLMConcurrency: &llmConcurrency,
|
||||
Report: false,
|
||||
Binary: writeAuditaHelperWrapper(t),
|
||||
Timeout: mustParseAuditaDuration(t, "2s"),
|
||||
LLMAPIKeyEnv: "OPENAI_KEY_SOURCE",
|
||||
Modules: []string{"glossary"},
|
||||
BaseURL: "https://openrouter.ai/api/v1",
|
||||
Model: "openrouter/google/gemma-4-31b-it",
|
||||
Report: false,
|
||||
})
|
||||
req := auditaReqForTest(t, false)
|
||||
_, err := runner.Run(context.Background(), req)
|
||||
@@ -346,16 +375,14 @@ func TestSubprocessRunnerSegmentsMissingFails(t *testing.T) {
|
||||
t.Setenv("OPENAI_KEY_SOURCE", "super-secret")
|
||||
t.Setenv("AUDITA_HELPER_RECORD_PATH", filepath.Join(t.TempDir(), "record.json"))
|
||||
|
||||
llmConcurrency := 1
|
||||
runner := mustAuditaRunner(t, SubprocessRunnerConfig{
|
||||
Binary: writeAuditaHelperWrapper(t),
|
||||
Timeout: mustParseAuditaDuration(t, "2s"),
|
||||
LLMAPIKeyEnv: "OPENAI_KEY_SOURCE",
|
||||
Modules: []string{"glossary"},
|
||||
BaseURL: "https://openrouter.ai/api/v1",
|
||||
Model: "openrouter/google/gemma-4-31b-it",
|
||||
LLMConcurrency: &llmConcurrency,
|
||||
Report: false,
|
||||
Binary: writeAuditaHelperWrapper(t),
|
||||
Timeout: mustParseAuditaDuration(t, "2s"),
|
||||
LLMAPIKeyEnv: "OPENAI_KEY_SOURCE",
|
||||
Modules: []string{"glossary"},
|
||||
BaseURL: "https://openrouter.ai/api/v1",
|
||||
Model: "openrouter/google/gemma-4-31b-it",
|
||||
Report: false,
|
||||
})
|
||||
req := auditaReqForTest(t, false)
|
||||
_, err := runner.Run(context.Background(), req)
|
||||
@@ -376,16 +403,14 @@ func TestSubprocessRunnerInvalidReportJSONFails(t *testing.T) {
|
||||
t.Setenv("OPENAI_KEY_SOURCE", "super-secret")
|
||||
t.Setenv("AUDITA_HELPER_RECORD_PATH", filepath.Join(t.TempDir(), "record.json"))
|
||||
|
||||
llmConcurrency := 1
|
||||
runner := mustAuditaRunner(t, SubprocessRunnerConfig{
|
||||
Binary: writeAuditaHelperWrapper(t),
|
||||
Timeout: mustParseAuditaDuration(t, "2s"),
|
||||
LLMAPIKeyEnv: "OPENAI_KEY_SOURCE",
|
||||
Modules: []string{"glossary"},
|
||||
BaseURL: "https://openrouter.ai/api/v1",
|
||||
Model: "openrouter/google/gemma-4-31b-it",
|
||||
LLMConcurrency: &llmConcurrency,
|
||||
Report: true,
|
||||
Binary: writeAuditaHelperWrapper(t),
|
||||
Timeout: mustParseAuditaDuration(t, "2s"),
|
||||
LLMAPIKeyEnv: "OPENAI_KEY_SOURCE",
|
||||
Modules: []string{"glossary"},
|
||||
BaseURL: "https://openrouter.ai/api/v1",
|
||||
Model: "openrouter/google/gemma-4-31b-it",
|
||||
Report: true,
|
||||
})
|
||||
req := auditaReqForTest(t, true)
|
||||
_, err := runner.Run(context.Background(), req)
|
||||
@@ -398,11 +423,11 @@ func TestSubprocessRunnerInvalidReportJSONFails(t *testing.T) {
|
||||
}
|
||||
|
||||
func TestSubprocessRunnerConstructorValidation(t *testing.T) {
|
||||
_, err := NewSubprocessRunnerFromConfigValues("", "3h", "AUDITA_LLM_API_KEY", []string{"glossary"}, "https://openrouter.ai/api/v1", "openrouter/google/gemma-4-31b-it", nil, "", nil, true)
|
||||
_, err := NewSubprocessRunnerFromConfigValues("", "3h", "AUDITA_LLM_API_KEY", []string{"glossary"}, "https://openrouter.ai/api/v1", "openrouter/google/gemma-4-31b-it", "", "", "", "", nil, nil, "", nil, true)
|
||||
if err == nil {
|
||||
t.Fatal("expected binary validation error")
|
||||
}
|
||||
_, err = NewSubprocessRunnerFromConfigValues("audita", "bad", "AUDITA_LLM_API_KEY", []string{"glossary"}, "https://openrouter.ai/api/v1", "openrouter/google/gemma-4-31b-it", nil, "", nil, true)
|
||||
_, err = NewSubprocessRunnerFromConfigValues("audita", "bad", "AUDITA_LLM_API_KEY", []string{"glossary"}, "https://openrouter.ai/api/v1", "openrouter/google/gemma-4-31b-it", "", "", "", "", nil, nil, "", nil, true)
|
||||
if err == nil {
|
||||
t.Fatal("expected timeout parse error")
|
||||
}
|
||||
|
||||
29
internal/adapters/storage/factory.go
Normal file
29
internal/adapters/storage/factory.go
Normal file
@@ -0,0 +1,29 @@
|
||||
package storage
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"strings"
|
||||
|
||||
"gitea.maximumdirect.net/eric/narratio/internal/config"
|
||||
)
|
||||
|
||||
// NewObjectStoreFromConfig constructs a remote object store from resolved config.
|
||||
func NewObjectStoreFromConfig(ctx context.Context, cfg *config.Config) (ObjectStore, error) {
|
||||
if cfg == nil || cfg.Pipeline == nil {
|
||||
return nil, fmt.Errorf("pipeline config is required")
|
||||
}
|
||||
|
||||
if strings.EqualFold(strings.TrimSpace(cfg.Pipeline.Storage.Backend), "s3") {
|
||||
if cfg.Pipeline.Storage.S3 == nil {
|
||||
return nil, fmt.Errorf("pipeline.storage.s3 is required when pipeline.storage.backend is s3")
|
||||
}
|
||||
return NewS3BackendFromConfig(ctx, *cfg.Pipeline.Storage.S3)
|
||||
}
|
||||
|
||||
if cfg.Pipeline.Storage.S3 != nil && strings.TrimSpace(cfg.Pipeline.Storage.S3.Bucket) != "" {
|
||||
return NewS3BackendFromConfig(ctx, *cfg.Pipeline.Storage.S3)
|
||||
}
|
||||
|
||||
return nil, fmt.Errorf("no remote object store backend is configured")
|
||||
}
|
||||
56
internal/adapters/storage/factory_test.go
Normal file
56
internal/adapters/storage/factory_test.go
Normal file
@@ -0,0 +1,56 @@
|
||||
package storage
|
||||
|
||||
import (
|
||||
"context"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"gitea.maximumdirect.net/eric/narratio/internal/config"
|
||||
)
|
||||
|
||||
func TestNewObjectStoreFromConfigBuildsS3WhenBackendIsS3(t *testing.T) {
|
||||
original := newS3Client
|
||||
t.Cleanup(func() { newS3Client = original })
|
||||
newS3Client = func(_ context.Context, _ s3ClientOptions) (s3API, error) {
|
||||
return &fakeS3API{}, nil
|
||||
}
|
||||
|
||||
store, err := NewObjectStoreFromConfig(context.Background(), &config.Config{
|
||||
Pipeline: &config.PipelineConfig{
|
||||
Storage: config.StorageConfig{
|
||||
Backend: "s3",
|
||||
S3: &config.StorageS3Config{
|
||||
Bucket: "my-archive",
|
||||
},
|
||||
},
|
||||
},
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatalf("NewObjectStoreFromConfig() error = %v", err)
|
||||
}
|
||||
if _, ok := store.(*S3Backend); !ok {
|
||||
t.Fatalf("store type = %T, want *S3Backend", store)
|
||||
}
|
||||
}
|
||||
|
||||
func TestNewObjectStoreFromConfigRequiresS3ConfigWhenBackendIsS3(t *testing.T) {
|
||||
_, err := NewObjectStoreFromConfig(context.Background(), &config.Config{
|
||||
Pipeline: &config.PipelineConfig{
|
||||
Storage: config.StorageConfig{Backend: "s3"},
|
||||
},
|
||||
})
|
||||
if err == nil || !strings.Contains(err.Error(), "pipeline.storage.s3 is required") {
|
||||
t.Fatalf("NewObjectStoreFromConfig() error = %v, want missing storage.s3 error", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestNewObjectStoreFromConfigNoRemoteBackendConfigured(t *testing.T) {
|
||||
_, err := NewObjectStoreFromConfig(context.Background(), &config.Config{
|
||||
Pipeline: &config.PipelineConfig{
|
||||
Storage: config.StorageConfig{Backend: "local"},
|
||||
},
|
||||
})
|
||||
if err == nil || !strings.Contains(err.Error(), "no remote object store backend is configured") {
|
||||
t.Fatalf("NewObjectStoreFromConfig() error = %v, want no-backend error", err)
|
||||
}
|
||||
}
|
||||
@@ -1,6 +1,14 @@
|
||||
package storage
|
||||
|
||||
import "context"
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"sort"
|
||||
"strings"
|
||||
"time"
|
||||
)
|
||||
|
||||
// NoopBackend is a deterministic no-op archive/storage adapter.
|
||||
type NoopBackend struct{}
|
||||
@@ -18,6 +26,21 @@ type FakeBackend struct {
|
||||
Requests []ArchiveRequest
|
||||
Err error
|
||||
Result ArchiveResult
|
||||
|
||||
Objects map[string]FakeObject
|
||||
Uploads []FakeUploadCall
|
||||
|
||||
ListErr error
|
||||
DownloadErr error
|
||||
UploadErr error
|
||||
ExistsErr error
|
||||
}
|
||||
|
||||
// FakeUploadCall captures one upload invocation in call order.
|
||||
type FakeUploadCall struct {
|
||||
LocalPath string
|
||||
Key string
|
||||
Options UploadOptions
|
||||
}
|
||||
|
||||
// Archive records request and returns configured response.
|
||||
@@ -38,3 +61,148 @@ func (f *FakeBackend) Archive(ctx context.Context, req ArchiveRequest) (ArchiveR
|
||||
}
|
||||
return res, nil
|
||||
}
|
||||
|
||||
// FakeObject is a deterministic fake object-store record.
|
||||
type FakeObject struct {
|
||||
Key string
|
||||
Data []byte
|
||||
Metadata map[string]string
|
||||
ETag string
|
||||
LastModified *time.Time
|
||||
}
|
||||
|
||||
// SeedObject inserts or replaces an object in the fake object store.
|
||||
func (f *FakeBackend) SeedObject(obj FakeObject) {
|
||||
if f.Objects == nil {
|
||||
f.Objects = map[string]FakeObject{}
|
||||
}
|
||||
key := normalizeObjectKey(obj.Key)
|
||||
obj.Key = key
|
||||
obj.Data = append([]byte(nil), obj.Data...)
|
||||
obj.Metadata = copyMetadata(obj.Metadata)
|
||||
f.Objects[key] = obj
|
||||
}
|
||||
|
||||
// List returns deterministic prefix-filtered objects.
|
||||
func (f *FakeBackend) List(ctx context.Context, prefix string) ([]ObjectInfo, error) {
|
||||
if err := ctx.Err(); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if f.ListErr != nil {
|
||||
return nil, f.ListErr
|
||||
}
|
||||
|
||||
normalizedPrefix := normalizeObjectKey(prefix)
|
||||
keys := make([]string, 0, len(f.Objects))
|
||||
for key := range f.Objects {
|
||||
if strings.HasPrefix(key, normalizedPrefix) {
|
||||
keys = append(keys, key)
|
||||
}
|
||||
}
|
||||
sort.Strings(keys)
|
||||
|
||||
out := make([]ObjectInfo, 0, len(keys))
|
||||
for _, key := range keys {
|
||||
obj := f.Objects[key]
|
||||
out = append(out, ObjectInfo{
|
||||
Key: obj.Key,
|
||||
Size: int64(len(obj.Data)),
|
||||
ETag: obj.ETag,
|
||||
LastModified: obj.LastModified,
|
||||
})
|
||||
}
|
||||
return out, nil
|
||||
}
|
||||
|
||||
// Download writes one object to a local path.
|
||||
func (f *FakeBackend) Download(ctx context.Context, key, localPath string) error {
|
||||
if err := ctx.Err(); err != nil {
|
||||
return err
|
||||
}
|
||||
if f.DownloadErr != nil {
|
||||
return f.DownloadErr
|
||||
}
|
||||
if strings.TrimSpace(localPath) == "" {
|
||||
return fmt.Errorf("download object: local path is required")
|
||||
}
|
||||
|
||||
obj, ok := f.Objects[normalizeObjectKey(key)]
|
||||
if !ok {
|
||||
return fmt.Errorf("download object %q: %w", key, os.ErrNotExist)
|
||||
}
|
||||
|
||||
if err := os.MkdirAll(filepath.Dir(localPath), 0o755); err != nil {
|
||||
return fmt.Errorf("download object %q: create parent directory: %w", key, err)
|
||||
}
|
||||
if err := os.WriteFile(localPath, obj.Data, 0o644); err != nil {
|
||||
return fmt.Errorf("download object %q: write local file: %w", key, err)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// Upload reads a local file and stores it under key.
|
||||
func (f *FakeBackend) Upload(ctx context.Context, localPath, key string, opts UploadOptions) (ObjectInfo, error) {
|
||||
if err := ctx.Err(); err != nil {
|
||||
return ObjectInfo{}, err
|
||||
}
|
||||
if f.UploadErr != nil {
|
||||
return ObjectInfo{}, f.UploadErr
|
||||
}
|
||||
if strings.TrimSpace(localPath) == "" {
|
||||
return ObjectInfo{}, fmt.Errorf("upload object: local path is required")
|
||||
}
|
||||
if strings.TrimSpace(key) == "" {
|
||||
return ObjectInfo{}, fmt.Errorf("upload object: key is required")
|
||||
}
|
||||
|
||||
data, err := os.ReadFile(localPath)
|
||||
if err != nil {
|
||||
return ObjectInfo{}, fmt.Errorf("upload object %q from %q: %w", key, localPath, err)
|
||||
}
|
||||
|
||||
normalizedKey := normalizeObjectKey(key)
|
||||
f.Uploads = append(f.Uploads, FakeUploadCall{
|
||||
LocalPath: localPath,
|
||||
Key: normalizedKey,
|
||||
Options: UploadOptions{
|
||||
Metadata: copyMetadata(opts.Metadata),
|
||||
ContentType: opts.ContentType,
|
||||
},
|
||||
})
|
||||
now := time.Now().UTC()
|
||||
obj := FakeObject{
|
||||
Key: normalizedKey,
|
||||
Data: data,
|
||||
Metadata: copyMetadata(opts.Metadata),
|
||||
LastModified: &now,
|
||||
}
|
||||
f.SeedObject(obj)
|
||||
return ObjectInfo{
|
||||
Key: normalizedKey,
|
||||
Size: int64(len(data)),
|
||||
LastModified: &now,
|
||||
}, nil
|
||||
}
|
||||
|
||||
// Exists checks object presence.
|
||||
func (f *FakeBackend) Exists(ctx context.Context, key string) (bool, error) {
|
||||
if err := ctx.Err(); err != nil {
|
||||
return false, err
|
||||
}
|
||||
if f.ExistsErr != nil {
|
||||
return false, f.ExistsErr
|
||||
}
|
||||
_, ok := f.Objects[normalizeObjectKey(key)]
|
||||
return ok, nil
|
||||
}
|
||||
|
||||
func copyMetadata(in map[string]string) map[string]string {
|
||||
if len(in) == 0 {
|
||||
return nil
|
||||
}
|
||||
out := make(map[string]string, len(in))
|
||||
for k, v := range in {
|
||||
out[k] = v
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
@@ -3,6 +3,9 @@ package storage
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
@@ -29,3 +32,84 @@ func TestFakeBackendError(t *testing.T) {
|
||||
t.Fatal("expected error, got nil")
|
||||
}
|
||||
}
|
||||
|
||||
func TestFakeBackendListPrefixFiltering(t *testing.T) {
|
||||
fake := &FakeBackend{}
|
||||
fake.SeedObject(FakeObject{Key: "dnd/campaigns/forsaken/audio/a.flac", Data: []byte("a")})
|
||||
fake.SeedObject(FakeObject{Key: "dnd/campaigns/forsaken/audio/b.flac", Data: []byte("b")})
|
||||
fake.SeedObject(FakeObject{Key: "dnd/campaigns/other/audio/c.flac", Data: []byte("c")})
|
||||
|
||||
items, err := fake.List(context.Background(), "dnd/campaigns/forsaken/audio/")
|
||||
if err != nil {
|
||||
t.Fatalf("List() error = %v", err)
|
||||
}
|
||||
if len(items) != 2 {
|
||||
t.Fatalf("List() len = %d, want 2", len(items))
|
||||
}
|
||||
if items[0].Key != "dnd/campaigns/forsaken/audio/a.flac" || items[1].Key != "dnd/campaigns/forsaken/audio/b.flac" {
|
||||
t.Fatalf("List() keys = %#v", items)
|
||||
}
|
||||
}
|
||||
|
||||
func TestFakeBackendDownload(t *testing.T) {
|
||||
fake := &FakeBackend{}
|
||||
fake.SeedObject(FakeObject{Key: "audio/a.flac", Data: []byte("audio-a")})
|
||||
|
||||
dst := filepath.Join(t.TempDir(), "nested", "a.flac")
|
||||
if err := fake.Download(context.Background(), "audio/a.flac", dst); err != nil {
|
||||
t.Fatalf("Download() error = %v", err)
|
||||
}
|
||||
data, err := os.ReadFile(dst)
|
||||
if err != nil {
|
||||
t.Fatalf("ReadFile() error = %v", err)
|
||||
}
|
||||
if string(data) != "audio-a" {
|
||||
t.Fatalf("downloaded content = %q, want %q", string(data), "audio-a")
|
||||
}
|
||||
}
|
||||
|
||||
func TestFakeBackendUploadAndExists(t *testing.T) {
|
||||
fake := &FakeBackend{}
|
||||
|
||||
local := filepath.Join(t.TempDir(), "upload.txt")
|
||||
if err := os.WriteFile(local, []byte("payload"), 0o644); err != nil {
|
||||
t.Fatalf("WriteFile() error = %v", err)
|
||||
}
|
||||
|
||||
info, err := fake.Upload(context.Background(), local, `runs\id\artifact.txt`, UploadOptions{
|
||||
Metadata: map[string]string{"kind": "artifact"},
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatalf("Upload() error = %v", err)
|
||||
}
|
||||
if info.Key != "runs/id/artifact.txt" {
|
||||
t.Fatalf("Upload() key = %q, want normalized key", info.Key)
|
||||
}
|
||||
|
||||
ok, err := fake.Exists(context.Background(), "runs/id/artifact.txt")
|
||||
if err != nil {
|
||||
t.Fatalf("Exists() error = %v", err)
|
||||
}
|
||||
if !ok {
|
||||
t.Fatal("Exists() = false, want true")
|
||||
}
|
||||
}
|
||||
|
||||
func TestFakeBackendObjectErrors(t *testing.T) {
|
||||
fake := &FakeBackend{DownloadErr: errors.New("download fail"), UploadErr: errors.New("upload fail"), ListErr: errors.New("list fail"), ExistsErr: errors.New("exists fail")}
|
||||
|
||||
if _, err := fake.List(context.Background(), "x"); err == nil || !strings.Contains(err.Error(), "list fail") {
|
||||
t.Fatalf("List() error = %v, want list fail", err)
|
||||
}
|
||||
if err := fake.Download(context.Background(), "x", filepath.Join(t.TempDir(), "x")); err == nil || !strings.Contains(err.Error(), "download fail") {
|
||||
t.Fatalf("Download() error = %v, want download fail", err)
|
||||
}
|
||||
local := filepath.Join(t.TempDir(), "x.txt")
|
||||
_ = os.WriteFile(local, []byte("x"), 0o644)
|
||||
if _, err := fake.Upload(context.Background(), local, "x", UploadOptions{}); err == nil || !strings.Contains(err.Error(), "upload fail") {
|
||||
t.Fatalf("Upload() error = %v, want upload fail", err)
|
||||
}
|
||||
if _, err := fake.Exists(context.Background(), "x"); err == nil || !strings.Contains(err.Error(), "exists fail") {
|
||||
t.Fatalf("Exists() error = %v, want exists fail", err)
|
||||
}
|
||||
}
|
||||
|
||||
8
internal/adapters/storage/keys.go
Normal file
8
internal/adapters/storage/keys.go
Normal file
@@ -0,0 +1,8 @@
|
||||
package storage
|
||||
|
||||
import "strings"
|
||||
|
||||
func normalizeObjectKey(key string) string {
|
||||
normalized := strings.ReplaceAll(strings.TrimSpace(key), "\\", "/")
|
||||
return strings.TrimLeft(normalized, "/")
|
||||
}
|
||||
21
internal/adapters/storage/keys_test.go
Normal file
21
internal/adapters/storage/keys_test.go
Normal file
@@ -0,0 +1,21 @@
|
||||
package storage
|
||||
|
||||
import "testing"
|
||||
|
||||
func TestNormalizeObjectKey(t *testing.T) {
|
||||
tests := []struct {
|
||||
in string
|
||||
want string
|
||||
}{
|
||||
{in: `dnd\campaigns\forsaken\a.flac`, want: "dnd/campaigns/forsaken/a.flac"},
|
||||
{in: " /dnd/campaigns/forsaken/a.flac ", want: "dnd/campaigns/forsaken/a.flac"},
|
||||
{in: "//dnd/campaigns/forsaken/a.flac", want: "dnd/campaigns/forsaken/a.flac"},
|
||||
{in: "", want: ""},
|
||||
}
|
||||
|
||||
for _, tt := range tests {
|
||||
if got := normalizeObjectKey(tt.in); got != tt.want {
|
||||
t.Fatalf("normalizeObjectKey(%q) = %q, want %q", tt.in, got, tt.want)
|
||||
}
|
||||
}
|
||||
}
|
||||
32
internal/adapters/storage/object_store.go
Normal file
32
internal/adapters/storage/object_store.go
Normal file
@@ -0,0 +1,32 @@
|
||||
package storage
|
||||
|
||||
import (
|
||||
"context"
|
||||
"time"
|
||||
)
|
||||
|
||||
// ObjectStore is a remote object storage boundary used by future prepare/archive work.
|
||||
//
|
||||
// Key invariant:
|
||||
// callers pass full bucket-relative object keys. Backend implementations do not
|
||||
// infer Narratio session semantics and do not prepend root prefixes.
|
||||
type ObjectStore interface {
|
||||
List(ctx context.Context, prefix string) ([]ObjectInfo, error)
|
||||
Download(ctx context.Context, key, localPath string) error
|
||||
Upload(ctx context.Context, localPath, key string, opts UploadOptions) (ObjectInfo, error)
|
||||
Exists(ctx context.Context, key string) (bool, error)
|
||||
}
|
||||
|
||||
// ObjectInfo describes one object in remote storage.
|
||||
type ObjectInfo struct {
|
||||
Key string
|
||||
Size int64
|
||||
ETag string
|
||||
LastModified *time.Time
|
||||
}
|
||||
|
||||
// UploadOptions configures optional object upload metadata.
|
||||
type UploadOptions struct {
|
||||
Metadata map[string]string
|
||||
ContentType string
|
||||
}
|
||||
241
internal/adapters/storage/s3_backend.go
Normal file
241
internal/adapters/storage/s3_backend.go
Normal file
@@ -0,0 +1,241 @@
|
||||
package storage
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"fmt"
|
||||
"io"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
awsconfig "github.com/aws/aws-sdk-go-v2/config"
|
||||
"github.com/aws/aws-sdk-go-v2/service/s3"
|
||||
"github.com/aws/aws-sdk-go-v2/service/s3/types"
|
||||
"github.com/aws/smithy-go"
|
||||
|
||||
"gitea.maximumdirect.net/eric/narratio/internal/config"
|
||||
)
|
||||
|
||||
type s3API interface {
|
||||
ListObjectsV2(ctx context.Context, params *s3.ListObjectsV2Input, optFns ...func(*s3.Options)) (*s3.ListObjectsV2Output, error)
|
||||
GetObject(ctx context.Context, params *s3.GetObjectInput, optFns ...func(*s3.Options)) (*s3.GetObjectOutput, error)
|
||||
PutObject(ctx context.Context, params *s3.PutObjectInput, optFns ...func(*s3.Options)) (*s3.PutObjectOutput, error)
|
||||
HeadObject(ctx context.Context, params *s3.HeadObjectInput, optFns ...func(*s3.Options)) (*s3.HeadObjectOutput, error)
|
||||
}
|
||||
|
||||
// S3Backend is an ObjectStore implementation backed by S3-compatible APIs.
|
||||
type S3Backend struct {
|
||||
bucket string
|
||||
client s3API
|
||||
}
|
||||
|
||||
type s3ClientOptions struct {
|
||||
Region string
|
||||
Endpoint string
|
||||
ForcePathStyle bool
|
||||
}
|
||||
|
||||
var newS3Client = func(ctx context.Context, opts s3ClientOptions) (s3API, error) {
|
||||
loadOpts := make([]func(*awsconfig.LoadOptions) error, 0, 1)
|
||||
if strings.TrimSpace(opts.Region) != "" {
|
||||
loadOpts = append(loadOpts, awsconfig.WithRegion(strings.TrimSpace(opts.Region)))
|
||||
}
|
||||
awsCfg, err := awsconfig.LoadDefaultConfig(ctx, loadOpts...)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("load aws config: %w", err)
|
||||
}
|
||||
|
||||
return s3.NewFromConfig(awsCfg, func(o *s3.Options) {
|
||||
if strings.TrimSpace(opts.Endpoint) != "" {
|
||||
endpoint := strings.TrimSpace(opts.Endpoint)
|
||||
o.BaseEndpoint = &endpoint
|
||||
}
|
||||
o.UsePathStyle = opts.ForcePathStyle
|
||||
}), nil
|
||||
}
|
||||
|
||||
// NewS3BackendFromConfig builds an S3 backend from resolved config.
|
||||
func NewS3BackendFromConfig(ctx context.Context, cfg config.StorageS3Config) (*S3Backend, error) {
|
||||
bucket := strings.TrimSpace(cfg.Bucket)
|
||||
if bucket == "" {
|
||||
return nil, fmt.Errorf("storage.s3.bucket is required")
|
||||
}
|
||||
|
||||
client, err := newS3Client(ctx, s3ClientOptions{
|
||||
Region: cfg.Region,
|
||||
Endpoint: cfg.Endpoint,
|
||||
ForcePathStyle: cfg.ForcePathStyle,
|
||||
})
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("build s3 client: %w", err)
|
||||
}
|
||||
|
||||
return &S3Backend{
|
||||
bucket: bucket,
|
||||
client: client,
|
||||
}, nil
|
||||
}
|
||||
|
||||
// List returns objects under prefix.
|
||||
func (b *S3Backend) List(ctx context.Context, prefix string) ([]ObjectInfo, error) {
|
||||
normalizedPrefix := normalizeObjectKey(prefix)
|
||||
out := make([]ObjectInfo, 0)
|
||||
var token *string
|
||||
|
||||
for {
|
||||
resp, err := b.client.ListObjectsV2(ctx, &s3.ListObjectsV2Input{
|
||||
Bucket: &b.bucket,
|
||||
Prefix: &normalizedPrefix,
|
||||
ContinuationToken: token,
|
||||
})
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("list objects under %q: %w", normalizedPrefix, err)
|
||||
}
|
||||
|
||||
for _, item := range resp.Contents {
|
||||
var lastModified *time.Time
|
||||
if item.LastModified != nil {
|
||||
t := *item.LastModified
|
||||
lastModified = &t
|
||||
}
|
||||
out = append(out, ObjectInfo{
|
||||
Key: normalizeObjectKey(valueOrEmpty(item.Key)),
|
||||
Size: valueOrZeroInt64(item.Size),
|
||||
ETag: strings.Trim(valueOrEmpty(item.ETag), "\""),
|
||||
LastModified: lastModified,
|
||||
})
|
||||
}
|
||||
|
||||
if !valueOrFalseBool(resp.IsTruncated) || resp.NextContinuationToken == nil {
|
||||
break
|
||||
}
|
||||
token = resp.NextContinuationToken
|
||||
}
|
||||
return out, nil
|
||||
}
|
||||
|
||||
// Download retrieves one object to localPath, creating parent directories as needed.
|
||||
func (b *S3Backend) Download(ctx context.Context, key, localPath string) error {
|
||||
normalizedKey := normalizeObjectKey(key)
|
||||
if strings.TrimSpace(localPath) == "" {
|
||||
return fmt.Errorf("download object: local path is required")
|
||||
}
|
||||
|
||||
resp, err := b.client.GetObject(ctx, &s3.GetObjectInput{
|
||||
Bucket: &b.bucket,
|
||||
Key: &normalizedKey,
|
||||
})
|
||||
if err != nil {
|
||||
return fmt.Errorf("download object %q: %w", normalizedKey, err)
|
||||
}
|
||||
defer resp.Body.Close()
|
||||
|
||||
if err := os.MkdirAll(filepath.Dir(localPath), 0o755); err != nil {
|
||||
return fmt.Errorf("download object %q: create parent directory: %w", normalizedKey, err)
|
||||
}
|
||||
dst, err := os.Create(localPath)
|
||||
if err != nil {
|
||||
return fmt.Errorf("download object %q: create local file: %w", normalizedKey, err)
|
||||
}
|
||||
defer dst.Close()
|
||||
|
||||
if _, err := io.Copy(dst, resp.Body); err != nil {
|
||||
return fmt.Errorf("download object %q: copy body: %w", normalizedKey, err)
|
||||
}
|
||||
if err := dst.Sync(); err != nil {
|
||||
return fmt.Errorf("download object %q: sync local file: %w", normalizedKey, err)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// Upload sends a local file to key.
|
||||
func (b *S3Backend) Upload(ctx context.Context, localPath, key string, opts UploadOptions) (ObjectInfo, error) {
|
||||
normalizedKey := normalizeObjectKey(key)
|
||||
if strings.TrimSpace(localPath) == "" {
|
||||
return ObjectInfo{}, fmt.Errorf("upload object: local path is required")
|
||||
}
|
||||
if normalizedKey == "" {
|
||||
return ObjectInfo{}, fmt.Errorf("upload object: key is required")
|
||||
}
|
||||
|
||||
file, err := os.Open(localPath)
|
||||
if err != nil {
|
||||
return ObjectInfo{}, fmt.Errorf("upload object %q from %q: %w", normalizedKey, localPath, err)
|
||||
}
|
||||
defer file.Close()
|
||||
|
||||
stat, err := file.Stat()
|
||||
if err != nil {
|
||||
return ObjectInfo{}, fmt.Errorf("upload object %q from %q: stat local file: %w", normalizedKey, localPath, err)
|
||||
}
|
||||
|
||||
input := &s3.PutObjectInput{
|
||||
Bucket: &b.bucket,
|
||||
Key: &normalizedKey,
|
||||
Body: file,
|
||||
Metadata: copyMetadata(opts.Metadata),
|
||||
}
|
||||
if strings.TrimSpace(opts.ContentType) != "" {
|
||||
ct := strings.TrimSpace(opts.ContentType)
|
||||
input.ContentType = &ct
|
||||
}
|
||||
|
||||
resp, err := b.client.PutObject(ctx, input)
|
||||
if err != nil {
|
||||
return ObjectInfo{}, fmt.Errorf("upload object %q from %q: %w", normalizedKey, localPath, err)
|
||||
}
|
||||
|
||||
return ObjectInfo{
|
||||
Key: normalizedKey,
|
||||
Size: stat.Size(),
|
||||
ETag: strings.Trim(valueOrEmpty(resp.ETag), "\""),
|
||||
}, nil
|
||||
}
|
||||
|
||||
// Exists checks whether one object key exists.
|
||||
func (b *S3Backend) Exists(ctx context.Context, key string) (bool, error) {
|
||||
normalizedKey := normalizeObjectKey(key)
|
||||
_, err := b.client.HeadObject(ctx, &s3.HeadObjectInput{
|
||||
Bucket: &b.bucket,
|
||||
Key: &normalizedKey,
|
||||
})
|
||||
if err == nil {
|
||||
return true, nil
|
||||
}
|
||||
|
||||
var notFound *types.NotFound
|
||||
if errors.As(err, ¬Found) {
|
||||
return false, nil
|
||||
}
|
||||
var apiErr smithy.APIError
|
||||
if errors.As(err, &apiErr) {
|
||||
switch apiErr.ErrorCode() {
|
||||
case "NotFound", "NoSuchKey", "404":
|
||||
return false, nil
|
||||
}
|
||||
}
|
||||
return false, fmt.Errorf("head object %q: %w", normalizedKey, err)
|
||||
}
|
||||
|
||||
func valueOrEmpty(v *string) string {
|
||||
if v == nil {
|
||||
return ""
|
||||
}
|
||||
return *v
|
||||
}
|
||||
|
||||
func valueOrZeroInt64(v *int64) int64 {
|
||||
if v == nil {
|
||||
return 0
|
||||
}
|
||||
return *v
|
||||
}
|
||||
|
||||
func valueOrFalseBool(v *bool) bool {
|
||||
if v == nil {
|
||||
return false
|
||||
}
|
||||
return *v
|
||||
}
|
||||
224
internal/adapters/storage/s3_backend_test.go
Normal file
224
internal/adapters/storage/s3_backend_test.go
Normal file
@@ -0,0 +1,224 @@
|
||||
package storage
|
||||
|
||||
import (
|
||||
"context"
|
||||
"io"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"github.com/aws/aws-sdk-go-v2/service/s3"
|
||||
"github.com/aws/aws-sdk-go-v2/service/s3/types"
|
||||
"github.com/aws/smithy-go"
|
||||
|
||||
"gitea.maximumdirect.net/eric/narratio/internal/config"
|
||||
)
|
||||
|
||||
type fakeS3API struct {
|
||||
listOut *s3.ListObjectsV2Output
|
||||
listErr error
|
||||
|
||||
getBody io.ReadCloser
|
||||
getErr error
|
||||
|
||||
putOut *s3.PutObjectOutput
|
||||
putErr error
|
||||
|
||||
headErr error
|
||||
|
||||
lastList *s3.ListObjectsV2Input
|
||||
lastGet *s3.GetObjectInput
|
||||
lastPut *s3.PutObjectInput
|
||||
lastHead *s3.HeadObjectInput
|
||||
}
|
||||
|
||||
func (f *fakeS3API) ListObjectsV2(_ context.Context, params *s3.ListObjectsV2Input, _ ...func(*s3.Options)) (*s3.ListObjectsV2Output, error) {
|
||||
f.lastList = params
|
||||
if f.listErr != nil {
|
||||
return nil, f.listErr
|
||||
}
|
||||
if f.listOut == nil {
|
||||
return &s3.ListObjectsV2Output{}, nil
|
||||
}
|
||||
return f.listOut, nil
|
||||
}
|
||||
|
||||
func (f *fakeS3API) GetObject(_ context.Context, params *s3.GetObjectInput, _ ...func(*s3.Options)) (*s3.GetObjectOutput, error) {
|
||||
f.lastGet = params
|
||||
if f.getErr != nil {
|
||||
return nil, f.getErr
|
||||
}
|
||||
body := f.getBody
|
||||
if body == nil {
|
||||
body = io.NopCloser(strings.NewReader(""))
|
||||
}
|
||||
return &s3.GetObjectOutput{Body: body}, nil
|
||||
}
|
||||
|
||||
func (f *fakeS3API) PutObject(_ context.Context, params *s3.PutObjectInput, _ ...func(*s3.Options)) (*s3.PutObjectOutput, error) {
|
||||
f.lastPut = params
|
||||
if f.putErr != nil {
|
||||
return nil, f.putErr
|
||||
}
|
||||
if f.putOut == nil {
|
||||
return &s3.PutObjectOutput{}, nil
|
||||
}
|
||||
return f.putOut, nil
|
||||
}
|
||||
|
||||
func (f *fakeS3API) HeadObject(_ context.Context, params *s3.HeadObjectInput, _ ...func(*s3.Options)) (*s3.HeadObjectOutput, error) {
|
||||
f.lastHead = params
|
||||
if f.headErr != nil {
|
||||
return nil, f.headErr
|
||||
}
|
||||
return &s3.HeadObjectOutput{}, nil
|
||||
}
|
||||
|
||||
func TestS3BackendListAndKeyNormalization(t *testing.T) {
|
||||
lastModified := time.Date(2026, 5, 16, 12, 0, 0, 0, time.UTC)
|
||||
client := &fakeS3API{
|
||||
listOut: &s3.ListObjectsV2Output{
|
||||
Contents: []types.Object{
|
||||
{Key: strPtr(`dnd\campaigns\forsaken\a.flac`), Size: int64Ptr(7), ETag: strPtr(`"abc"`), LastModified: &lastModified},
|
||||
},
|
||||
},
|
||||
}
|
||||
backend := &S3Backend{bucket: "bucket-1", client: client}
|
||||
|
||||
items, err := backend.List(context.Background(), `dnd\campaigns\`)
|
||||
if err != nil {
|
||||
t.Fatalf("List() error = %v", err)
|
||||
}
|
||||
if len(items) != 1 {
|
||||
t.Fatalf("List() len = %d, want 1", len(items))
|
||||
}
|
||||
if items[0].Key != "dnd/campaigns/forsaken/a.flac" {
|
||||
t.Fatalf("List() key = %q, want normalized slash key", items[0].Key)
|
||||
}
|
||||
if items[0].ETag != "abc" {
|
||||
t.Fatalf("List() ETag = %q, want %q", items[0].ETag, "abc")
|
||||
}
|
||||
if client.lastList == nil || *client.lastList.Prefix != "dnd/campaigns/" {
|
||||
t.Fatalf("List() prefix = %#v, want normalized prefix", client.lastList)
|
||||
}
|
||||
}
|
||||
|
||||
func TestS3BackendDownloadCreatesParentDirectory(t *testing.T) {
|
||||
client := &fakeS3API{getBody: io.NopCloser(strings.NewReader("audio"))}
|
||||
backend := &S3Backend{bucket: "bucket-1", client: client}
|
||||
|
||||
dst := filepath.Join(t.TempDir(), "nested", "clip.flac")
|
||||
if err := backend.Download(context.Background(), `audio\clip.flac`, dst); err != nil {
|
||||
t.Fatalf("Download() error = %v", err)
|
||||
}
|
||||
|
||||
data, err := os.ReadFile(dst)
|
||||
if err != nil {
|
||||
t.Fatalf("ReadFile() error = %v", err)
|
||||
}
|
||||
if string(data) != "audio" {
|
||||
t.Fatalf("downloaded content = %q, want %q", string(data), "audio")
|
||||
}
|
||||
if client.lastGet == nil || *client.lastGet.Key != "audio/clip.flac" {
|
||||
t.Fatalf("GetObject key = %#v, want normalized key", client.lastGet)
|
||||
}
|
||||
}
|
||||
|
||||
func TestS3BackendUploadAndExists(t *testing.T) {
|
||||
client := &fakeS3API{putOut: &s3.PutObjectOutput{ETag: strPtr(`"etag123"`)}}
|
||||
backend := &S3Backend{bucket: "bucket-1", client: client}
|
||||
|
||||
local := filepath.Join(t.TempDir(), "artifact.txt")
|
||||
if err := os.WriteFile(local, []byte("artifact"), 0o644); err != nil {
|
||||
t.Fatalf("WriteFile() error = %v", err)
|
||||
}
|
||||
|
||||
info, err := backend.Upload(context.Background(), local, `runs\id\artifact.txt`, UploadOptions{
|
||||
Metadata: map[string]string{"kind": "artifact"},
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatalf("Upload() error = %v", err)
|
||||
}
|
||||
if info.Key != "runs/id/artifact.txt" {
|
||||
t.Fatalf("Upload key = %q, want normalized key", info.Key)
|
||||
}
|
||||
if info.ETag != "etag123" {
|
||||
t.Fatalf("Upload ETag = %q, want %q", info.ETag, "etag123")
|
||||
}
|
||||
if client.lastPut == nil || *client.lastPut.Key != "runs/id/artifact.txt" {
|
||||
t.Fatalf("PutObject key = %#v, want normalized key", client.lastPut)
|
||||
}
|
||||
|
||||
ok, err := backend.Exists(context.Background(), "runs/id/artifact.txt")
|
||||
if err != nil {
|
||||
t.Fatalf("Exists() error = %v", err)
|
||||
}
|
||||
if !ok {
|
||||
t.Fatal("Exists() = false, want true")
|
||||
}
|
||||
}
|
||||
|
||||
func TestS3BackendUploadMissingLocalFile(t *testing.T) {
|
||||
backend := &S3Backend{bucket: "bucket-1", client: &fakeS3API{}}
|
||||
_, err := backend.Upload(context.Background(), filepath.Join(t.TempDir(), "missing.txt"), "key.txt", UploadOptions{})
|
||||
if err == nil || !strings.Contains(err.Error(), "no such file") {
|
||||
t.Fatalf("Upload() error = %v, want missing local file error", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestS3BackendExistsNotFound(t *testing.T) {
|
||||
backend := &S3Backend{
|
||||
bucket: "bucket-1",
|
||||
client: &fakeS3API{
|
||||
headErr: &smithy.GenericAPIError{Code: "NotFound", Message: "missing"},
|
||||
},
|
||||
}
|
||||
ok, err := backend.Exists(context.Background(), "missing-key")
|
||||
if err != nil {
|
||||
t.Fatalf("Exists() error = %v", err)
|
||||
}
|
||||
if ok {
|
||||
t.Fatal("Exists() = true, want false")
|
||||
}
|
||||
}
|
||||
|
||||
func TestNewS3BackendFromConfigUsesClientOptions(t *testing.T) {
|
||||
original := newS3Client
|
||||
t.Cleanup(func() { newS3Client = original })
|
||||
|
||||
var got s3ClientOptions
|
||||
newS3Client = func(_ context.Context, opts s3ClientOptions) (s3API, error) {
|
||||
got = opts
|
||||
return &fakeS3API{}, nil
|
||||
}
|
||||
|
||||
backend, err := NewS3BackendFromConfig(context.Background(), config.StorageS3Config{
|
||||
Bucket: "my-archive",
|
||||
Region: "us-east-1",
|
||||
Endpoint: "http://localhost:9000",
|
||||
ForcePathStyle: true,
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatalf("NewS3BackendFromConfig() error = %v", err)
|
||||
}
|
||||
if backend.bucket != "my-archive" {
|
||||
t.Fatalf("backend.bucket = %q, want %q", backend.bucket, "my-archive")
|
||||
}
|
||||
if got.Region != "us-east-1" || got.Endpoint != "http://localhost:9000" || !got.ForcePathStyle {
|
||||
t.Fatalf("client options = %#v, want region/endpoint/path-style values", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestNewS3BackendFromConfigRequiresBucket(t *testing.T) {
|
||||
_, err := NewS3BackendFromConfig(context.Background(), config.StorageS3Config{})
|
||||
if err == nil || !strings.Contains(err.Error(), "bucket is required") {
|
||||
t.Fatalf("NewS3BackendFromConfig() error = %v, want bucket validation", err)
|
||||
}
|
||||
}
|
||||
|
||||
func strPtr(v string) *string { return &v }
|
||||
func int64Ptr(v int64) *int64 { return &v }
|
||||
|
||||
var _ s3API = (*fakeS3API)(nil)
|
||||
@@ -208,6 +208,7 @@ notification:
|
||||
timeout: 10s
|
||||
`
|
||||
sessionYAML := `session_id: ` + sessionID + `
|
||||
campaign: sample-campaign
|
||||
inputs:
|
||||
audio_dir: ./audio
|
||||
speakers_file: ./speakers.yml
|
||||
@@ -271,6 +272,7 @@ notification:
|
||||
timeout: 10s
|
||||
`
|
||||
sessionYAML := `session_id: 2026-05-03
|
||||
campaign: sample-campaign
|
||||
inputs:
|
||||
audio_dir: ./audio
|
||||
speakers_file: ./speakers.yml
|
||||
@@ -377,6 +379,11 @@ func writeValidConfigFiles(t *testing.T, workspaceRoot string, transcribeURL ...
|
||||
root: ` + workspaceRoot + `
|
||||
storage:
|
||||
backend: s3
|
||||
s3:
|
||||
bucket: test-bucket
|
||||
archive:
|
||||
enabled: true
|
||||
upload_run: false
|
||||
whisperx:
|
||||
transcribe_url: ` + url + `
|
||||
timeout: 2s
|
||||
@@ -400,6 +407,7 @@ notification:
|
||||
`
|
||||
|
||||
sessionYAML := `session_id: 2026-05-03
|
||||
campaign: sample-campaign
|
||||
inputs:
|
||||
audio_dir: ./audio
|
||||
speakers_file: ./speakers.yml
|
||||
|
||||
@@ -113,6 +113,7 @@ notification:
|
||||
timeout: 10s
|
||||
`
|
||||
sessionYAML := `session_id: 2026-05-03
|
||||
campaign: sample-campaign
|
||||
inputs:
|
||||
audio_dir: ./audio
|
||||
speakers_file: ./speakers.yml
|
||||
|
||||
@@ -81,6 +81,13 @@ func executeStages(ctx context.Context, cfg *config.Config, stages []stage.Stage
|
||||
if env.Storage == nil {
|
||||
env.Storage = &storage.NoopBackend{}
|
||||
}
|
||||
if env.ObjectStore == nil && needsObjectStoreForRun(env.Config, stages) {
|
||||
objectStore, err := storage.NewObjectStoreFromConfig(ctx, env.Config)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("initialize object store backend: %w", err)
|
||||
}
|
||||
env.ObjectStore = objectStore
|
||||
}
|
||||
if env.Notifier == nil {
|
||||
env.Notifier = ¬ify.NoopSender{}
|
||||
}
|
||||
@@ -104,6 +111,15 @@ func executeStages(ctx context.Context, cfg *config.Config, stages []stage.Stage
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
identityChanged, err := ensureManifestIdentity(cfg, m)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("initialize manifest identity: %w", err)
|
||||
}
|
||||
if identityChanged {
|
||||
if err := env.ManifestStore.Save(ctx, manifestPath, m); err != nil {
|
||||
return nil, fmt.Errorf("save manifest identity %q: %w", manifestPath, err)
|
||||
}
|
||||
}
|
||||
|
||||
stageEnv := env
|
||||
|
||||
@@ -230,7 +246,7 @@ func buildDefaultAuditaRunner(cfg *config.Config) (audita.Runner, error) {
|
||||
}
|
||||
|
||||
a := cfg.Pipeline.Audita
|
||||
if strings.TrimSpace(a.Binary) == "" || strings.TrimSpace(a.Timeout) == "" || len(a.Modules) == 0 || strings.TrimSpace(a.BaseURL) == "" || strings.TrimSpace(a.Model) == "" {
|
||||
if strings.TrimSpace(a.Binary) == "" || strings.TrimSpace(a.Timeout) == "" || strings.TrimSpace(a.BaseURL) == "" || strings.TrimSpace(a.Model) == "" {
|
||||
// Compatibility fallback for tests or internal call paths that bypass config validation/defaults.
|
||||
return &audita.NoopRunner{}, nil
|
||||
}
|
||||
@@ -247,7 +263,12 @@ func buildDefaultAuditaRunner(cfg *config.Config) (audita.Runner, error) {
|
||||
append([]string(nil), a.Modules...),
|
||||
a.BaseURL,
|
||||
a.Model,
|
||||
a.LLMConcurrency,
|
||||
a.TranscriptDescription,
|
||||
a.ConfigPath,
|
||||
a.OutputSchema,
|
||||
a.WorkDirRetention,
|
||||
a.TotalLLMConcurrency,
|
||||
a.ProposalLLMConcurrency,
|
||||
a.ValidationModel,
|
||||
a.ValidationLLMConcurrency,
|
||||
report,
|
||||
@@ -333,6 +354,89 @@ func applyStageResultToManifest(m *manifest.Manifest, stageName string, result *
|
||||
}
|
||||
}
|
||||
|
||||
func ensureManifestIdentity(cfg *config.Config, m *manifest.Manifest) (bool, error) {
|
||||
if cfg == nil || cfg.Pipeline == nil || cfg.Session == nil || m == nil {
|
||||
return false, nil
|
||||
}
|
||||
|
||||
changed := false
|
||||
campaign := strings.TrimSpace(cfg.Session.Campaign)
|
||||
sessionID := strings.TrimSpace(cfg.Session.SessionID)
|
||||
if sessionID == "" {
|
||||
sessionID = strings.TrimSpace(m.SessionID)
|
||||
}
|
||||
|
||||
if m.Campaign == "" && campaign != "" {
|
||||
m.Campaign = campaign
|
||||
changed = true
|
||||
}
|
||||
if m.RunID == "" {
|
||||
runID, err := artifacts.NewRunID()
|
||||
if err != nil {
|
||||
return false, err
|
||||
}
|
||||
m.RunID = runID
|
||||
changed = true
|
||||
}
|
||||
if m.LocalWorkDir == "" && campaign != "" && sessionID != "" && m.RunID != "" {
|
||||
m.LocalWorkDir = artifacts.SessionRunWorkDir(cfg.Pipeline.Workspace.Root, campaign, sessionID, m.RunID)
|
||||
changed = true
|
||||
}
|
||||
if m.LocalSpoolDir == "" && campaign != "" && sessionID != "" && m.RunID != "" && strings.TrimSpace(cfg.Pipeline.Spool.Root) != "" {
|
||||
m.LocalSpoolDir = artifacts.SessionSpoolAudioDir(cfg.Pipeline.Spool.Root, campaign, sessionID, m.RunID)
|
||||
changed = true
|
||||
}
|
||||
if cfg.Pipeline.Storage.S3 != nil {
|
||||
if m.S3Bucket == "" && strings.TrimSpace(cfg.Pipeline.Storage.S3.Bucket) != "" {
|
||||
m.S3Bucket = strings.TrimSpace(cfg.Pipeline.Storage.S3.Bucket)
|
||||
changed = true
|
||||
}
|
||||
sessionPrefix := artifacts.S3SessionPrefix(cfg.Pipeline.Storage.S3.RootPrefix, campaign, sessionID)
|
||||
if m.S3SessionPrefix == "" && sessionPrefix != "" {
|
||||
m.S3SessionPrefix = sessionPrefix
|
||||
changed = true
|
||||
}
|
||||
runPrefix := artifacts.S3RunPrefix(sessionPrefix, m.RunID)
|
||||
if m.S3RunPrefix == "" && runPrefix != "" {
|
||||
m.S3RunPrefix = runPrefix
|
||||
changed = true
|
||||
}
|
||||
}
|
||||
|
||||
return changed, nil
|
||||
}
|
||||
|
||||
func manifestPathFor(cfg *config.Config) string {
|
||||
return filepath.Join(cfg.Pipeline.Workspace.Root, "work", cfg.Session.SessionID, "manifest.json")
|
||||
}
|
||||
|
||||
func needsObjectStoreForRun(cfg *config.Config, stages []stage.Stage) bool {
|
||||
if cfg == nil || cfg.Pipeline == nil || cfg.Session == nil {
|
||||
return false
|
||||
}
|
||||
stageRequested := func(name string) bool {
|
||||
for _, s := range stages {
|
||||
if s != nil && s.Name() == name {
|
||||
return true
|
||||
}
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
if cfg.Session.Inputs.AudioS3 != nil && stageRequested("prepare") {
|
||||
return true
|
||||
}
|
||||
if !stageRequested("archive") {
|
||||
return false
|
||||
}
|
||||
if cfg.Pipeline.Archive == nil {
|
||||
return false
|
||||
}
|
||||
if cfg.Pipeline.Archive.Enabled != nil && !*cfg.Pipeline.Archive.Enabled {
|
||||
return false
|
||||
}
|
||||
if cfg.Pipeline.Archive.UploadRun != nil && !*cfg.Pipeline.Archive.UploadRun {
|
||||
return false
|
||||
}
|
||||
return true
|
||||
}
|
||||
|
||||
@@ -144,6 +144,15 @@ func TestExecuteStagesPlaceholderSuccessUpdatesManifest(t *testing.T) {
|
||||
}
|
||||
continue
|
||||
}
|
||||
if name == "archive" {
|
||||
if sr.Metadata == nil || sr.Metadata["stage"] != "archive" {
|
||||
t.Fatalf("archive metadata missing stage=archive: %#v", sr.Metadata)
|
||||
}
|
||||
if sr.Metadata["skipped"] != true {
|
||||
t.Fatalf("archive metadata missing skipped=true for test config without archive section: %#v", sr.Metadata)
|
||||
}
|
||||
continue
|
||||
}
|
||||
if sr.Metadata == nil || sr.Metadata["placeholder"] != true {
|
||||
t.Fatalf("stage %q missing placeholder metadata", name)
|
||||
}
|
||||
@@ -315,7 +324,7 @@ func TestAdapterBackedStageFailureMarksManifestFailed(t *testing.T) {
|
||||
{name: "merge", env: &Env{Seriatim: &seriatim.FakeRunner{Err: errors.New("merge fail")}}},
|
||||
{name: "polish", env: &Env{Audita: &audita.FakeRunner{Err: errors.New("polish fail")}}},
|
||||
{name: "analyze", env: &Env{Scriptorium: &scriptorium.FakeRunner{RunErr: errors.New("analyze fail")}}},
|
||||
{name: "archive", env: &Env{Storage: &storage.FakeBackend{Err: errors.New("archive fail")}}},
|
||||
{name: "archive", env: &Env{ObjectStore: &storage.FakeBackend{UploadErr: errors.New("archive fail")}}},
|
||||
{name: "notify", env: &Env{Notifier: ¬ify.FakeSender{Err: errors.New("notify fail")}}},
|
||||
}
|
||||
|
||||
@@ -400,6 +409,41 @@ func TestAdapterBackedStageFailureMarksManifestFailed(t *testing.T) {
|
||||
},
|
||||
}
|
||||
}
|
||||
if tc.name == "archive" {
|
||||
cfg.Pipeline.Archive = &config.ArchiveConfig{
|
||||
Enabled: boolPtr(true),
|
||||
UploadRun: boolPtr(true),
|
||||
}
|
||||
cfg.Pipeline.Storage.S3 = &config.StorageS3Config{
|
||||
Bucket: "my-dnd-archive",
|
||||
RootPrefix: "dnd",
|
||||
}
|
||||
runID := "20260516T010203Z-0a1b2c3d"
|
||||
runWorkDir := filepath.Join(cfg.Pipeline.Workspace.Root, "work", cfg.Session.Campaign, cfg.Session.SessionID, runID)
|
||||
if err := os.MkdirAll(filepath.Join(runWorkDir, "inputs"), 0o755); err != nil {
|
||||
t.Fatalf("mkdir archive inputs dir: %v", err)
|
||||
}
|
||||
if err := os.WriteFile(filepath.Join(runWorkDir, "inputs", "session.yml"), []byte("session_id: 2026-05-03\n"), 0o644); err != nil {
|
||||
t.Fatalf("write archive fixture session.yml: %v", err)
|
||||
}
|
||||
if err := os.WriteFile(filepath.Join(runWorkDir, "manifest.json"), []byte("{}\n"), 0o644); err != nil {
|
||||
t.Fatalf("write archive fixture manifest.json: %v", err)
|
||||
}
|
||||
|
||||
seed := manifest.New(cfg.Session.SessionID, time.Now().UTC())
|
||||
seed.Campaign = cfg.Session.Campaign
|
||||
seed.RunID = runID
|
||||
seed.LocalWorkDir = runWorkDir
|
||||
seed.S3Bucket = "my-dnd-archive"
|
||||
seed.S3SessionPrefix = "dnd/campaigns/" + cfg.Session.Campaign + "/sessions/" + cfg.Session.SessionID + "/"
|
||||
seed.S3RunPrefix = seed.S3SessionPrefix + "runs/" + runID + "/"
|
||||
for _, name := range []string{"prepare", "transcribe", "merge", "polish", "normalize", "trim", "analyze"} {
|
||||
seed.MarkStageSucceeded(name, time.Now().UTC(), nil)
|
||||
}
|
||||
if err := tc.env.ManifestStore.Save(context.Background(), manifestPathFor(cfg), seed); err != nil {
|
||||
t.Fatalf("seed archive manifest: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
_, runErr := executeStages(context.Background(), cfg, []stage.Stage{selected}, RunOptions{Env: tc.env})
|
||||
if runErr == nil {
|
||||
@@ -430,7 +474,7 @@ func testConfig(t *testing.T) *config.Config {
|
||||
pipelinePath := filepath.Join(cfgDir, "pipeline.yml")
|
||||
|
||||
mustWriteFile(t, pipelinePath, "workspace:\n root: "+workspace+"\n")
|
||||
mustWriteFile(t, sessionPath, "session_id: 2026-05-03\n")
|
||||
mustWriteFile(t, sessionPath, "session_id: 2026-05-03\ncampaign: sample-campaign\n")
|
||||
mustWriteFile(t, filepath.Join(cfgDir, "speakers.yml"), "alice: alice.flac\n")
|
||||
mustWriteFile(t, filepath.Join(cfgDir, "autocorrect.yml"), "[]\n")
|
||||
mustWriteFile(t, filepath.Join(cfgDir, "glossary.yml"), "[]\n")
|
||||
@@ -442,6 +486,7 @@ func testConfig(t *testing.T) *config.Config {
|
||||
SessionPath: sessionPath,
|
||||
Session: &config.SessionConfig{
|
||||
SessionID: "2026-05-03",
|
||||
Campaign: "sample-campaign",
|
||||
Inputs: config.SessionInputsConfig{
|
||||
AudioDir: "./audio",
|
||||
SpeakersFile: "./speakers.yml",
|
||||
@@ -461,3 +506,8 @@ func mustWriteFile(t *testing.T, path, contents string) {
|
||||
t.Fatalf("WriteFile(%q): %v", path, err)
|
||||
}
|
||||
}
|
||||
|
||||
func boolPtr(v bool) *bool {
|
||||
p := v
|
||||
return &p
|
||||
}
|
||||
|
||||
@@ -1,6 +1,8 @@
|
||||
package artifacts
|
||||
|
||||
import "path/filepath"
|
||||
import (
|
||||
"path/filepath"
|
||||
)
|
||||
|
||||
// SessionPaths contains canonical local paths for one session work directory.
|
||||
type SessionPaths struct {
|
||||
@@ -23,6 +25,16 @@ func SessionWorkDir(rootDir, sessionID string) string {
|
||||
return filepath.Join(rootDir, "work", sessionID)
|
||||
}
|
||||
|
||||
// SessionRunWorkDir returns the campaign/session/run scoped local work directory.
|
||||
func SessionRunWorkDir(rootDir, campaign, sessionID, runID string) string {
|
||||
return filepath.Join(rootDir, "work", campaign, sessionID, runID)
|
||||
}
|
||||
|
||||
// SessionSpoolAudioDir returns the campaign/session/run scoped local spool audio path.
|
||||
func SessionSpoolAudioDir(spoolRoot, campaign, sessionID, runID string) string {
|
||||
return filepath.Join(spoolRoot, campaign, sessionID, runID, "audio")
|
||||
}
|
||||
|
||||
func buildSessionPaths(workspaceRoot, sessionID string) SessionPaths {
|
||||
root := SessionWorkDir(workspaceRoot, sessionID)
|
||||
transcripts := filepath.Join(root, "transcripts")
|
||||
|
||||
24
internal/artifacts/paths_model_test.go
Normal file
24
internal/artifacts/paths_model_test.go
Normal file
@@ -0,0 +1,24 @@
|
||||
package artifacts
|
||||
|
||||
import (
|
||||
"path/filepath"
|
||||
"testing"
|
||||
)
|
||||
|
||||
func TestSessionRunWorkDir(t *testing.T) {
|
||||
root := "/tmp/workspace"
|
||||
got := SessionRunWorkDir(root, "forsaken", "2026-04-19", "20260515T031522Z-a1b2c3d4")
|
||||
want := filepath.Join(root, "work", "forsaken", "2026-04-19", "20260515T031522Z-a1b2c3d4")
|
||||
if got != want {
|
||||
t.Fatalf("SessionRunWorkDir() = %q, want %q", got, want)
|
||||
}
|
||||
}
|
||||
|
||||
func TestSessionSpoolAudioDir(t *testing.T) {
|
||||
root := "/var/spool/narratio"
|
||||
got := SessionSpoolAudioDir(root, "forsaken", "2026-04-19", "20260515T031522Z-a1b2c3d4")
|
||||
want := filepath.Join(root, "forsaken", "2026-04-19", "20260515T031522Z-a1b2c3d4", "audio")
|
||||
if got != want {
|
||||
t.Fatalf("SessionSpoolAudioDir() = %q, want %q", got, want)
|
||||
}
|
||||
}
|
||||
30
internal/artifacts/run_id.go
Normal file
30
internal/artifacts/run_id.go
Normal file
@@ -0,0 +1,30 @@
|
||||
package artifacts
|
||||
|
||||
import (
|
||||
"crypto/rand"
|
||||
"encoding/hex"
|
||||
"fmt"
|
||||
"io"
|
||||
"time"
|
||||
)
|
||||
|
||||
// NewRunID returns a run ID in format: YYYYMMDDTHHMMSSZ-xxxxxxxx.
|
||||
func NewRunID() (string, error) {
|
||||
return NewRunIDWith(time.Now().UTC(), rand.Reader)
|
||||
}
|
||||
|
||||
// NewRunIDWith returns a run ID in format: YYYYMMDDTHHMMSSZ-xxxxxxxx
|
||||
// using an injected timestamp and randomness source.
|
||||
func NewRunIDWith(now time.Time, random io.Reader) (string, error) {
|
||||
if random == nil {
|
||||
random = rand.Reader
|
||||
}
|
||||
|
||||
var suffix [4]byte
|
||||
if _, err := io.ReadFull(random, suffix[:]); err != nil {
|
||||
return "", fmt.Errorf("generate run id random suffix: %w", err)
|
||||
}
|
||||
|
||||
ts := now.UTC().Format("20060102T150405Z")
|
||||
return ts + "-" + hex.EncodeToString(suffix[:]), nil
|
||||
}
|
||||
37
internal/artifacts/run_id_test.go
Normal file
37
internal/artifacts/run_id_test.go
Normal file
@@ -0,0 +1,37 @@
|
||||
package artifacts
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"regexp"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
)
|
||||
|
||||
func TestNewRunIDWithFormat(t *testing.T) {
|
||||
now := time.Date(2026, 5, 15, 3, 15, 22, 0, time.UTC)
|
||||
random := bytes.NewReader([]byte{0xa1, 0xb2, 0xc3, 0xd4})
|
||||
|
||||
runID, err := NewRunIDWith(now, random)
|
||||
if err != nil {
|
||||
t.Fatalf("NewRunIDWith() error = %v", err)
|
||||
}
|
||||
if runID != "20260515T031522Z-a1b2c3d4" {
|
||||
t.Fatalf("runID = %q, want %q", runID, "20260515T031522Z-a1b2c3d4")
|
||||
}
|
||||
}
|
||||
|
||||
func TestNewRunIDWithShape(t *testing.T) {
|
||||
runID, err := NewRunIDWith(time.Now().UTC(), bytes.NewReader([]byte{0x01, 0x02, 0x03, 0x04}))
|
||||
if err != nil {
|
||||
t.Fatalf("NewRunIDWith() error = %v", err)
|
||||
}
|
||||
pattern := regexp.MustCompile(`^\d{8}T\d{6}Z-[0-9a-f]{8}$`)
|
||||
if !pattern.MatchString(runID) {
|
||||
t.Fatalf("runID = %q, want pattern %q", runID, pattern.String())
|
||||
}
|
||||
suffix := runID[len(runID)-8:]
|
||||
if strings.ToLower(suffix) != suffix {
|
||||
t.Fatalf("runID suffix = %q, want lowercase", suffix)
|
||||
}
|
||||
}
|
||||
73
internal/artifacts/s3_keys.go
Normal file
73
internal/artifacts/s3_keys.go
Normal file
@@ -0,0 +1,73 @@
|
||||
package artifacts
|
||||
|
||||
import (
|
||||
"path"
|
||||
"strings"
|
||||
)
|
||||
|
||||
// S3SessionPrefix builds the canonical S3 session prefix.
|
||||
// Format: {root_prefix}/campaigns/{campaign}/sessions/{session_id}/
|
||||
func S3SessionPrefix(rootPrefix, campaign, sessionID string) string {
|
||||
prefix := path.Join(
|
||||
cleanS3PathPart(rootPrefix),
|
||||
"campaigns",
|
||||
cleanS3PathPart(campaign),
|
||||
"sessions",
|
||||
cleanS3PathPart(sessionID),
|
||||
)
|
||||
return ensureS3TrailingSlash(prefix)
|
||||
}
|
||||
|
||||
// S3RunPrefix builds the canonical S3 run prefix.
|
||||
// Format: {session_prefix}/runs/{run_id}/
|
||||
func S3RunPrefix(sessionPrefix, runID string) string {
|
||||
prefix := path.Join(strings.TrimSuffix(cleanS3Key(sessionPrefix), "/"), "runs", cleanS3PathPart(runID))
|
||||
return ensureS3TrailingSlash(prefix)
|
||||
}
|
||||
|
||||
// S3AudioPrefix builds the session audio prefix from configured audio_s3.prefix.
|
||||
// Format: {session_prefix}/{audio_s3.prefix}
|
||||
func S3AudioPrefix(sessionPrefix, audioPrefix string) string {
|
||||
key := path.Join(strings.TrimSuffix(cleanS3Key(sessionPrefix), "/"), cleanS3Key(audioPrefix))
|
||||
return ensureS3TrailingSlash(key)
|
||||
}
|
||||
|
||||
// S3CurrentManifestKey returns the current manifest pointer key.
|
||||
// Format: {session_prefix}/current/manifest.json
|
||||
func S3CurrentManifestKey(sessionPrefix string) string {
|
||||
return path.Join(strings.TrimSuffix(cleanS3Key(sessionPrefix), "/"), "current", "manifest.json")
|
||||
}
|
||||
|
||||
// S3CurrentRunPointerKey returns the current run pointer key.
|
||||
// Format: {session_prefix}/current/run_id.txt
|
||||
func S3CurrentRunPointerKey(sessionPrefix string) string {
|
||||
return path.Join(strings.TrimSuffix(cleanS3Key(sessionPrefix), "/"), "current", "run_id.txt")
|
||||
}
|
||||
|
||||
// S3PromotedArtifactKey returns the destination key for one promoted artifact.
|
||||
// Format: {session_prefix}/{promotion.to}
|
||||
func S3PromotedArtifactKey(sessionPrefix, to string) string {
|
||||
return path.Join(strings.TrimSuffix(cleanS3Key(sessionPrefix), "/"), cleanS3Key(to))
|
||||
}
|
||||
|
||||
// S3RunRelativeDestinationKey returns a run-scoped key for a workdir-relative path.
|
||||
// Format: {run_prefix}/{relative_workdir_path}
|
||||
func S3RunRelativeDestinationKey(runPrefix, relativeWorkdirPath string) string {
|
||||
return path.Join(strings.TrimSuffix(cleanS3Key(runPrefix), "/"), cleanS3Key(relativeWorkdirPath))
|
||||
}
|
||||
|
||||
func ensureS3TrailingSlash(v string) string {
|
||||
key := cleanS3Key(v)
|
||||
if key == "" {
|
||||
return ""
|
||||
}
|
||||
return strings.TrimSuffix(key, "/") + "/"
|
||||
}
|
||||
|
||||
func cleanS3PathPart(v string) string {
|
||||
return strings.Trim(strings.ReplaceAll(strings.TrimSpace(v), "\\", "/"), "/")
|
||||
}
|
||||
|
||||
func cleanS3Key(v string) string {
|
||||
return strings.ReplaceAll(strings.TrimSpace(v), "\\", "/")
|
||||
}
|
||||
45
internal/artifacts/s3_keys_test.go
Normal file
45
internal/artifacts/s3_keys_test.go
Normal file
@@ -0,0 +1,45 @@
|
||||
package artifacts
|
||||
|
||||
import (
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
func TestS3KeyConstruction(t *testing.T) {
|
||||
runID := "20260515T031522Z-a1b2c3d4"
|
||||
sessionPrefix := S3SessionPrefix("dnd", "forsaken", "2026-04-19")
|
||||
if sessionPrefix != "dnd/campaigns/forsaken/sessions/2026-04-19/" {
|
||||
t.Fatalf("sessionPrefix = %q", sessionPrefix)
|
||||
}
|
||||
|
||||
audioPrefix := S3AudioPrefix(sessionPrefix, "audio/")
|
||||
if audioPrefix != "dnd/campaigns/forsaken/sessions/2026-04-19/audio/" {
|
||||
t.Fatalf("audioPrefix = %q", audioPrefix)
|
||||
}
|
||||
|
||||
runPrefix := S3RunPrefix(sessionPrefix, runID)
|
||||
wantRunPrefix := "dnd/campaigns/forsaken/sessions/2026-04-19/runs/" + runID + "/"
|
||||
if runPrefix != wantRunPrefix {
|
||||
t.Fatalf("runPrefix = %q, want %q", runPrefix, wantRunPrefix)
|
||||
}
|
||||
|
||||
runPointer := S3CurrentRunPointerKey(sessionPrefix)
|
||||
if runPointer != "dnd/campaigns/forsaken/sessions/2026-04-19/current/run_id.txt" {
|
||||
t.Fatalf("run pointer key = %q", runPointer)
|
||||
}
|
||||
|
||||
manifestKey := S3CurrentManifestKey(sessionPrefix)
|
||||
if manifestKey != "dnd/campaigns/forsaken/sessions/2026-04-19/current/manifest.json" {
|
||||
t.Fatalf("manifest key = %q", manifestKey)
|
||||
}
|
||||
|
||||
promoted := S3PromotedArtifactKey(sessionPrefix, "transcripts/trimmed.json")
|
||||
if promoted != "dnd/campaigns/forsaken/sessions/2026-04-19/transcripts/trimmed.json" {
|
||||
t.Fatalf("promoted key = %q", promoted)
|
||||
}
|
||||
|
||||
runRelative := S3RunRelativeDestinationKey(runPrefix, `logs\whisperx.stdout.log`)
|
||||
if !strings.HasSuffix(runRelative, "/logs/whisperx.stdout.log") {
|
||||
t.Fatalf("runRelative key = %q, want normalized forward slashes", runRelative)
|
||||
}
|
||||
}
|
||||
@@ -12,6 +12,8 @@ type Config struct {
|
||||
type PipelineConfig struct {
|
||||
Workspace WorkspaceConfig `yaml:"workspace"`
|
||||
Storage StorageConfig `yaml:"storage"`
|
||||
Spool SpoolConfig `yaml:"spool"`
|
||||
Archive *ArchiveConfig `yaml:"archive"`
|
||||
Secrets *SecretsConfig `yaml:"secrets"`
|
||||
WhisperX WhisperXConfig `yaml:"whisperx"`
|
||||
Seriatim SeriatimConfig `yaml:"seriatim"`
|
||||
@@ -44,9 +46,39 @@ type SecretsConfig struct {
|
||||
|
||||
// StorageConfig configures storage backends and related parameters.
|
||||
type StorageConfig struct {
|
||||
Backend string `yaml:"backend"`
|
||||
Bucket string `yaml:"bucket"`
|
||||
Prefix string `yaml:"prefix"`
|
||||
Backend string `yaml:"backend"`
|
||||
Bucket string `yaml:"bucket"`
|
||||
Prefix string `yaml:"prefix"`
|
||||
S3 *StorageS3Config `yaml:"s3"`
|
||||
}
|
||||
|
||||
// StorageS3Config configures S3 storage coordinates.
|
||||
type StorageS3Config struct {
|
||||
Bucket string `yaml:"bucket"`
|
||||
RootPrefix string `yaml:"root_prefix"`
|
||||
Region string `yaml:"region"`
|
||||
Endpoint string `yaml:"endpoint"`
|
||||
ForcePathStyle bool `yaml:"force_path_style"`
|
||||
}
|
||||
|
||||
// SpoolConfig configures local spool storage for staged data.
|
||||
type SpoolConfig struct {
|
||||
Root string `yaml:"root"`
|
||||
DeleteAudioAfterArchive bool `yaml:"delete_audio_after_archive"`
|
||||
}
|
||||
|
||||
// ArchiveConfig configures archive behavior and artifact promotions.
|
||||
type ArchiveConfig struct {
|
||||
Enabled *bool `yaml:"enabled"`
|
||||
UploadRun *bool `yaml:"upload_run"`
|
||||
PromoteArtifacts []ArchivePromotionRule `yaml:"promote_artifacts"`
|
||||
}
|
||||
|
||||
// ArchivePromotionRule configures one artifact promotion mapping.
|
||||
type ArchivePromotionRule struct {
|
||||
From string `yaml:"from"`
|
||||
To string `yaml:"to"`
|
||||
Required *bool `yaml:"required"`
|
||||
}
|
||||
|
||||
// WhisperXConfig configures WhisperX adapter settings.
|
||||
@@ -85,9 +117,14 @@ type AuditaConfig struct {
|
||||
Modules []string `yaml:"modules"`
|
||||
BaseURL string `yaml:"base_url"`
|
||||
Model string `yaml:"model"`
|
||||
LLMConcurrency *int `yaml:"llm_concurrency"`
|
||||
TotalLLMConcurrency *int `yaml:"total_llm_concurrency"`
|
||||
ProposalLLMConcurrency *int `yaml:"proposal_llm_concurrency"`
|
||||
ValidationModel string `yaml:"validation_model"`
|
||||
ValidationLLMConcurrency *int `yaml:"validation_llm_concurrency"`
|
||||
TranscriptDescription string `yaml:"transcript_description"`
|
||||
ConfigPath string `yaml:"config_path"`
|
||||
OutputSchema string `yaml:"output_schema"`
|
||||
WorkDirRetention string `yaml:"work_dir_retention"`
|
||||
Report *bool `yaml:"report"`
|
||||
}
|
||||
|
||||
@@ -175,9 +212,15 @@ type ArtifactSettings struct {
|
||||
|
||||
// SessionInputsConfig contains per-session input references.
|
||||
type SessionInputsConfig struct {
|
||||
AudioDir string `yaml:"audio_dir"`
|
||||
AudioFiles []string `yaml:"audio_files"`
|
||||
SpeakersFile string `yaml:"speakers_file"`
|
||||
AutocorrectFile string `yaml:"autocorrect_file"`
|
||||
GlossaryFile string `yaml:"glossary_file"`
|
||||
AudioDir string `yaml:"audio_dir"`
|
||||
AudioFiles []string `yaml:"audio_files"`
|
||||
AudioS3 *SessionAudioS3Input `yaml:"audio_s3"`
|
||||
SpeakersFile string `yaml:"speakers_file"`
|
||||
AutocorrectFile string `yaml:"autocorrect_file"`
|
||||
GlossaryFile string `yaml:"glossary_file"`
|
||||
}
|
||||
|
||||
// SessionAudioS3Input configures S3 session-audio input discovery.
|
||||
type SessionAudioS3Input struct {
|
||||
Prefix string `yaml:"prefix"`
|
||||
}
|
||||
|
||||
@@ -81,6 +81,9 @@ func applyPipelineDefaults(cfg *PipelineConfig) {
|
||||
if cfg == nil {
|
||||
return
|
||||
}
|
||||
applyStorageDefaults(&cfg.Storage)
|
||||
applySpoolDefaults(&cfg.Spool)
|
||||
applyArchiveDefaults(&cfg.Archive)
|
||||
applyWhisperXDefaults(&cfg.WhisperX)
|
||||
applySeriatimDefaults(&cfg.Seriatim)
|
||||
applyAuditaDefaults(&cfg.Audita)
|
||||
@@ -92,6 +95,54 @@ func applyPipelineDefaults(cfg *PipelineConfig) {
|
||||
applyScriptoriumDefaults(cfg.Scriptorium)
|
||||
}
|
||||
|
||||
func applyStorageDefaults(cfg *StorageConfig) {
|
||||
if cfg == nil {
|
||||
return
|
||||
}
|
||||
if cfg.S3 == nil {
|
||||
cfg.S3 = &StorageS3Config{}
|
||||
}
|
||||
if cfg.S3.RootPrefix == "" {
|
||||
cfg.S3.RootPrefix = "dnd"
|
||||
}
|
||||
}
|
||||
|
||||
func applySpoolDefaults(cfg *SpoolConfig) {
|
||||
if cfg == nil {
|
||||
return
|
||||
}
|
||||
if cfg.Root == "" {
|
||||
cfg.Root = "/var/spool/narratio"
|
||||
}
|
||||
}
|
||||
|
||||
func applyArchiveDefaults(cfg **ArchiveConfig) {
|
||||
if cfg == nil {
|
||||
return
|
||||
}
|
||||
if *cfg == nil {
|
||||
*cfg = &ArchiveConfig{}
|
||||
}
|
||||
|
||||
if (*cfg).Enabled == nil {
|
||||
(*cfg).Enabled = boolPtr(true)
|
||||
}
|
||||
if (*cfg).UploadRun == nil {
|
||||
(*cfg).UploadRun = boolPtr(true)
|
||||
}
|
||||
if len((*cfg).PromoteArtifacts) == 0 {
|
||||
(*cfg).PromoteArtifacts = []ArchivePromotionRule{
|
||||
{From: "transcripts/trimmed.json", To: "transcripts/trimmed.json", Required: boolPtr(true)},
|
||||
{From: "artifacts/session_recap.md", To: "artifacts/session_recap.md", Required: boolPtr(true)},
|
||||
}
|
||||
}
|
||||
for i := range (*cfg).PromoteArtifacts {
|
||||
if (*cfg).PromoteArtifacts[i].Required == nil {
|
||||
(*cfg).PromoteArtifacts[i].Required = boolPtr(true)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func applyWhisperXDefaults(cfg *WhisperXConfig) {
|
||||
if cfg == nil {
|
||||
return
|
||||
@@ -143,29 +194,12 @@ func applyAuditaDefaults(cfg *AuditaConfig) {
|
||||
if cfg.Timeout == "" {
|
||||
cfg.Timeout = "3h"
|
||||
}
|
||||
if cfg.Modules == nil {
|
||||
cfg.Modules = []string{
|
||||
"glossary",
|
||||
"homophones",
|
||||
"glossary",
|
||||
"spoken_word",
|
||||
"grammar",
|
||||
"homophones",
|
||||
"glossary",
|
||||
}
|
||||
}
|
||||
if cfg.BaseURL == "" {
|
||||
cfg.BaseURL = "https://openrouter.ai/api/v1"
|
||||
}
|
||||
if cfg.Model == "" {
|
||||
cfg.Model = "openrouter/google/gemma-4-31b-it"
|
||||
}
|
||||
if cfg.LLMConcurrency == nil {
|
||||
cfg.LLMConcurrency = intPtr(1)
|
||||
}
|
||||
if cfg.ValidationLLMConcurrency == nil {
|
||||
cfg.ValidationLLMConcurrency = intPtr(1)
|
||||
}
|
||||
if cfg.Report == nil {
|
||||
cfg.Report = boolPtr(true)
|
||||
}
|
||||
|
||||
@@ -453,7 +453,7 @@ inputs:
|
||||
wantValidate: "pipeline config \"pipeline.yml\" invalid: pipeline.audita.timeout must be a valid duration",
|
||||
},
|
||||
{
|
||||
name: "empty audita modules fails",
|
||||
name: "empty audita modules is valid override",
|
||||
pipelineYAML: `workspace:
|
||||
root: /tmp/narratio
|
||||
whisperx:
|
||||
@@ -471,7 +471,6 @@ inputs:
|
||||
autocorrect_file: ./autocorrect.yml
|
||||
glossary_file: ./glossary.yml
|
||||
`,
|
||||
wantValidate: "pipeline config \"pipeline.yml\" invalid: pipeline.audita.modules must include at least one module",
|
||||
},
|
||||
{
|
||||
name: "empty audita module item fails",
|
||||
@@ -541,7 +540,7 @@ inputs:
|
||||
wantValidate: "pipeline config \"pipeline.yml\" invalid: pipeline.audita.base_url must be a valid URL",
|
||||
},
|
||||
{
|
||||
name: "invalid audita llm_concurrency fails",
|
||||
name: "legacy audita llm_concurrency field fails strict decode",
|
||||
pipelineYAML: `workspace:
|
||||
root: /tmp/narratio
|
||||
whisperx:
|
||||
@@ -550,7 +549,7 @@ seriatim:
|
||||
binary: seriatim
|
||||
audita:
|
||||
binary: audita
|
||||
llm_concurrency: 0
|
||||
llm_concurrency: 1
|
||||
`,
|
||||
sessionYAML: `session_id: 2026-05-03
|
||||
inputs:
|
||||
@@ -559,7 +558,49 @@ inputs:
|
||||
autocorrect_file: ./autocorrect.yml
|
||||
glossary_file: ./glossary.yml
|
||||
`,
|
||||
wantValidate: "pipeline config \"pipeline.yml\" invalid: pipeline.audita.llm_concurrency must be > 0",
|
||||
wantLoadErr: "strict decode failed",
|
||||
},
|
||||
{
|
||||
name: "invalid audita total_llm_concurrency fails",
|
||||
pipelineYAML: `workspace:
|
||||
root: /tmp/narratio
|
||||
whisperx:
|
||||
transcribe_url: https://transcription.ai.rakestrawhome.com/transcribe
|
||||
seriatim:
|
||||
binary: seriatim
|
||||
audita:
|
||||
binary: audita
|
||||
total_llm_concurrency: 0
|
||||
`,
|
||||
sessionYAML: `session_id: 2026-05-03
|
||||
inputs:
|
||||
audio_dir: ./audio
|
||||
speakers_file: ./speakers.yml
|
||||
autocorrect_file: ./autocorrect.yml
|
||||
glossary_file: ./glossary.yml
|
||||
`,
|
||||
wantValidate: "pipeline config \"pipeline.yml\" invalid: pipeline.audita.total_llm_concurrency must be > 0",
|
||||
},
|
||||
{
|
||||
name: "invalid audita proposal_llm_concurrency fails",
|
||||
pipelineYAML: `workspace:
|
||||
root: /tmp/narratio
|
||||
whisperx:
|
||||
transcribe_url: https://transcription.ai.rakestrawhome.com/transcribe
|
||||
seriatim:
|
||||
binary: seriatim
|
||||
audita:
|
||||
binary: audita
|
||||
proposal_llm_concurrency: 0
|
||||
`,
|
||||
sessionYAML: `session_id: 2026-05-03
|
||||
inputs:
|
||||
audio_dir: ./audio
|
||||
speakers_file: ./speakers.yml
|
||||
autocorrect_file: ./autocorrect.yml
|
||||
glossary_file: ./glossary.yml
|
||||
`,
|
||||
wantValidate: "pipeline config \"pipeline.yml\" invalid: pipeline.audita.proposal_llm_concurrency must be > 0",
|
||||
},
|
||||
{
|
||||
name: "invalid audita validation_llm_concurrency fails",
|
||||
@@ -582,6 +623,48 @@ inputs:
|
||||
`,
|
||||
wantValidate: "pipeline config \"pipeline.yml\" invalid: pipeline.audita.validation_llm_concurrency must be > 0",
|
||||
},
|
||||
{
|
||||
name: "invalid audita output_schema fails",
|
||||
pipelineYAML: `workspace:
|
||||
root: /tmp/narratio
|
||||
whisperx:
|
||||
transcribe_url: https://transcription.ai.rakestrawhome.com/transcribe
|
||||
seriatim:
|
||||
binary: seriatim
|
||||
audita:
|
||||
binary: audita
|
||||
output_schema: bad
|
||||
`,
|
||||
sessionYAML: `session_id: 2026-05-03
|
||||
inputs:
|
||||
audio_dir: ./audio
|
||||
speakers_file: ./speakers.yml
|
||||
autocorrect_file: ./autocorrect.yml
|
||||
glossary_file: ./glossary.yml
|
||||
`,
|
||||
wantValidate: "pipeline config \"pipeline.yml\" invalid: pipeline.audita.output_schema must be one of: bare-segments, audita-v1",
|
||||
},
|
||||
{
|
||||
name: "invalid audita work_dir_retention fails",
|
||||
pipelineYAML: `workspace:
|
||||
root: /tmp/narratio
|
||||
whisperx:
|
||||
transcribe_url: https://transcription.ai.rakestrawhome.com/transcribe
|
||||
seriatim:
|
||||
binary: seriatim
|
||||
audita:
|
||||
binary: audita
|
||||
work_dir_retention: sometimes
|
||||
`,
|
||||
sessionYAML: `session_id: 2026-05-03
|
||||
inputs:
|
||||
audio_dir: ./audio
|
||||
speakers_file: ./speakers.yml
|
||||
autocorrect_file: ./autocorrect.yml
|
||||
glossary_file: ./glossary.yml
|
||||
`,
|
||||
wantValidate: "pipeline config \"pipeline.yml\" invalid: pipeline.audita.work_dir_retention must be one of: always, auto, never",
|
||||
},
|
||||
}
|
||||
|
||||
for _, tt := range tests {
|
||||
@@ -645,8 +728,8 @@ inputs:
|
||||
if cfg.Pipeline.Audita.LLMAPIKeyEnv != "" {
|
||||
t.Fatalf("audita.llm_api_key_env = %q, want empty by default", cfg.Pipeline.Audita.LLMAPIKeyEnv)
|
||||
}
|
||||
if got := strings.Join(cfg.Pipeline.Audita.Modules, ","); got != "glossary,homophones,glossary,spoken_word,grammar,homophones,glossary" {
|
||||
t.Fatalf("audita.modules = %q, want default sequence", got)
|
||||
if cfg.Pipeline.Audita.Modules != nil {
|
||||
t.Fatalf("audita.modules = %#v, want nil default (optional override)", cfg.Pipeline.Audita.Modules)
|
||||
}
|
||||
if cfg.Pipeline.Audita.BaseURL != "https://openrouter.ai/api/v1" {
|
||||
t.Fatalf("audita.base_url = %q, want %q", cfg.Pipeline.Audita.BaseURL, "https://openrouter.ai/api/v1")
|
||||
@@ -654,14 +737,17 @@ inputs:
|
||||
if cfg.Pipeline.Audita.Model != "openrouter/google/gemma-4-31b-it" {
|
||||
t.Fatalf("audita.model = %q, want %q", cfg.Pipeline.Audita.Model, "openrouter/google/gemma-4-31b-it")
|
||||
}
|
||||
if cfg.Pipeline.Audita.LLMConcurrency == nil || *cfg.Pipeline.Audita.LLMConcurrency != 1 {
|
||||
t.Fatalf("audita.llm_concurrency = %v, want 1", cfg.Pipeline.Audita.LLMConcurrency)
|
||||
}
|
||||
if cfg.Pipeline.Audita.ValidationModel != "" {
|
||||
t.Fatalf("audita.validation_model = %q, want empty default", cfg.Pipeline.Audita.ValidationModel)
|
||||
}
|
||||
if cfg.Pipeline.Audita.ValidationLLMConcurrency == nil || *cfg.Pipeline.Audita.ValidationLLMConcurrency != 1 {
|
||||
t.Fatalf("audita.validation_llm_concurrency = %v, want 1", cfg.Pipeline.Audita.ValidationLLMConcurrency)
|
||||
if cfg.Pipeline.Audita.TotalLLMConcurrency != nil {
|
||||
t.Fatalf("audita.total_llm_concurrency = %v, want nil default", cfg.Pipeline.Audita.TotalLLMConcurrency)
|
||||
}
|
||||
if cfg.Pipeline.Audita.ProposalLLMConcurrency != nil {
|
||||
t.Fatalf("audita.proposal_llm_concurrency = %v, want nil default", cfg.Pipeline.Audita.ProposalLLMConcurrency)
|
||||
}
|
||||
if cfg.Pipeline.Audita.ValidationLLMConcurrency != nil {
|
||||
t.Fatalf("audita.validation_llm_concurrency = %v, want nil default", cfg.Pipeline.Audita.ValidationLLMConcurrency)
|
||||
}
|
||||
if cfg.Pipeline.Audita.Report == nil || *cfg.Pipeline.Audita.Report != true {
|
||||
t.Fatalf("audita.report = %v, want true", cfg.Pipeline.Audita.Report)
|
||||
@@ -725,7 +811,8 @@ func TestValidateMissingAudioSource(t *testing.T) {
|
||||
Modules: []string{"glossary", "homophones"},
|
||||
BaseURL: "https://openrouter.ai/api/v1",
|
||||
Model: "openrouter/google/gemma-4-31b-it",
|
||||
LLMConcurrency: intPtr(1),
|
||||
TotalLLMConcurrency: intPtr(1),
|
||||
ProposalLLMConcurrency: intPtr(1),
|
||||
ValidationModel: "",
|
||||
ValidationLLMConcurrency: intPtr(1),
|
||||
Report: boolPtr(true),
|
||||
@@ -733,6 +820,7 @@ func TestValidateMissingAudioSource(t *testing.T) {
|
||||
},
|
||||
Session: &SessionConfig{
|
||||
SessionID: "2026-05-03",
|
||||
Campaign: "sample-campaign",
|
||||
Inputs: SessionInputsConfig{
|
||||
SpeakersFile: "speakers.yml",
|
||||
AutocorrectFile: "autocorrect.yml",
|
||||
@@ -745,7 +833,7 @@ func TestValidateMissingAudioSource(t *testing.T) {
|
||||
if err == nil {
|
||||
t.Fatal("expected validation error, got nil")
|
||||
}
|
||||
if !strings.Contains(err.Error(), "audio_dir or at least one audio_files") {
|
||||
if !strings.Contains(err.Error(), "audio_dir, at least one audio_files entry, or audio_s3") {
|
||||
t.Fatalf("error = %q, want audio source guidance", err.Error())
|
||||
}
|
||||
if !strings.Contains(err.Error(), "session config") {
|
||||
@@ -774,6 +862,12 @@ func writeConfigFiles(t *testing.T, pipelineYAML, sessionYAML string) (string, s
|
||||
}
|
||||
pipelineYAML += "audita:\n binary: audita\n"
|
||||
}
|
||||
if !strings.Contains(sessionYAML, "\ncampaign:") && !strings.HasPrefix(sessionYAML, "campaign:") {
|
||||
if !strings.HasSuffix(sessionYAML, "\n") {
|
||||
sessionYAML += "\n"
|
||||
}
|
||||
sessionYAML += "campaign: sample-campaign\n"
|
||||
}
|
||||
|
||||
dir := t.TempDir()
|
||||
pipelinePath := filepath.Join(dir, "pipeline.yml")
|
||||
|
||||
235
internal/config/storage_archive_test.go
Normal file
235
internal/config/storage_archive_test.go
Normal file
@@ -0,0 +1,235 @@
|
||||
package config
|
||||
|
||||
import (
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
func TestStorageS3DefaultsAndValidation(t *testing.T) {
|
||||
pipelineYAML := testPipelineBaseYAML + `
|
||||
storage:
|
||||
backend: s3
|
||||
s3:
|
||||
bucket: my-dnd-archive
|
||||
`
|
||||
pipelinePath, sessionPath := writeConfigFiles(t, pipelineYAML, testSessionBaseYAML)
|
||||
|
||||
cfg, err := Load(pipelinePath, sessionPath)
|
||||
if err != nil {
|
||||
t.Fatalf("Load() error = %v", err)
|
||||
}
|
||||
if cfg.Pipeline.Storage.S3 == nil {
|
||||
t.Fatal("storage.s3 should be initialized")
|
||||
}
|
||||
if cfg.Pipeline.Storage.S3.RootPrefix != "dnd" {
|
||||
t.Fatalf("storage.s3.root_prefix = %q, want dnd", cfg.Pipeline.Storage.S3.RootPrefix)
|
||||
}
|
||||
if cfg.Pipeline.Storage.S3.ForcePathStyle {
|
||||
t.Fatalf("storage.s3.force_path_style = true, want false default")
|
||||
}
|
||||
|
||||
if err := Validate(cfg); err != nil {
|
||||
t.Fatalf("Validate() error = %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestSpoolAndArchiveDefaults(t *testing.T) {
|
||||
pipelinePath, sessionPath := writeConfigFiles(t, testPipelineBaseYAML, testSessionBaseYAML)
|
||||
|
||||
cfg, err := Load(pipelinePath, sessionPath)
|
||||
if err != nil {
|
||||
t.Fatalf("Load() error = %v", err)
|
||||
}
|
||||
|
||||
if cfg.Pipeline.Spool.Root != "/var/spool/narratio" {
|
||||
t.Fatalf("spool.root = %q, want /var/spool/narratio", cfg.Pipeline.Spool.Root)
|
||||
}
|
||||
if cfg.Pipeline.Spool.DeleteAudioAfterArchive {
|
||||
t.Fatalf("spool.delete_audio_after_archive = true, want false")
|
||||
}
|
||||
if cfg.Pipeline.Archive == nil {
|
||||
t.Fatal("archive should be initialized by defaults")
|
||||
}
|
||||
if cfg.Pipeline.Archive.Enabled == nil || !*cfg.Pipeline.Archive.Enabled {
|
||||
t.Fatalf("archive.enabled = %#v, want true", cfg.Pipeline.Archive.Enabled)
|
||||
}
|
||||
if cfg.Pipeline.Archive.UploadRun == nil || !*cfg.Pipeline.Archive.UploadRun {
|
||||
t.Fatalf("archive.upload_run = %#v, want true", cfg.Pipeline.Archive.UploadRun)
|
||||
}
|
||||
if len(cfg.Pipeline.Archive.PromoteArtifacts) != 2 {
|
||||
t.Fatalf("archive.promote_artifacts len = %d, want 2 defaults", len(cfg.Pipeline.Archive.PromoteArtifacts))
|
||||
}
|
||||
for i, item := range cfg.Pipeline.Archive.PromoteArtifacts {
|
||||
if item.Required == nil || !*item.Required {
|
||||
t.Fatalf("archive.promote_artifacts[%d].required = %#v, want true", i, item.Required)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestArchivePromotionPathValidation(t *testing.T) {
|
||||
tests := []struct {
|
||||
name string
|
||||
ruleYML string
|
||||
wantErr string
|
||||
}{
|
||||
{
|
||||
name: "absolute from path rejected",
|
||||
ruleYML: `archive:
|
||||
promote_artifacts:
|
||||
- from: "/transcripts/trimmed.json"
|
||||
to: "transcripts/trimmed.json"
|
||||
`,
|
||||
wantErr: "must be a relative path",
|
||||
},
|
||||
{
|
||||
name: "traversal to path rejected",
|
||||
ruleYML: `archive:
|
||||
promote_artifacts:
|
||||
- from: "transcripts/trimmed.json"
|
||||
to: "../trimmed.json"
|
||||
`,
|
||||
wantErr: "must not contain path traversal",
|
||||
},
|
||||
}
|
||||
|
||||
for _, tt := range tests {
|
||||
t.Run(tt.name, func(t *testing.T) {
|
||||
pipelineYAML := testPipelineBaseYAML + "\n" + tt.ruleYML
|
||||
pipelinePath, sessionPath := writeConfigFiles(t, pipelineYAML, testSessionBaseYAML)
|
||||
|
||||
cfg, err := Load(pipelinePath, sessionPath)
|
||||
if err != nil {
|
||||
t.Fatalf("Load() error = %v", err)
|
||||
}
|
||||
err = Validate(cfg)
|
||||
if err == nil || !strings.Contains(err.Error(), tt.wantErr) {
|
||||
t.Fatalf("Validate() error = %v, want to contain %q", err, tt.wantErr)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestSessionAudioS3Validation(t *testing.T) {
|
||||
tests := []struct {
|
||||
name string
|
||||
sessionYAML string
|
||||
wantErr string
|
||||
}{
|
||||
{
|
||||
name: "valid audio_s3 prefix",
|
||||
sessionYAML: `session_id: 2026-05-03
|
||||
campaign: forsaken
|
||||
inputs:
|
||||
audio_s3:
|
||||
prefix: audio/
|
||||
speakers_file: ./speakers.yml
|
||||
autocorrect_file: ./autocorrect.yml
|
||||
glossary_file: ./glossary.yml
|
||||
`,
|
||||
},
|
||||
{
|
||||
name: "invalid audio_s3 absolute prefix",
|
||||
sessionYAML: `session_id: 2026-05-03
|
||||
campaign: forsaken
|
||||
inputs:
|
||||
audio_s3:
|
||||
prefix: /audio/
|
||||
speakers_file: ./speakers.yml
|
||||
autocorrect_file: ./autocorrect.yml
|
||||
glossary_file: ./glossary.yml
|
||||
`,
|
||||
wantErr: "session.inputs.audio_s3.prefix must be a relative path",
|
||||
},
|
||||
{
|
||||
name: "invalid audio_s3 traversal prefix",
|
||||
sessionYAML: `session_id: 2026-05-03
|
||||
campaign: forsaken
|
||||
inputs:
|
||||
audio_s3:
|
||||
prefix: ../audio/
|
||||
speakers_file: ./speakers.yml
|
||||
autocorrect_file: ./autocorrect.yml
|
||||
glossary_file: ./glossary.yml
|
||||
`,
|
||||
wantErr: "session.inputs.audio_s3.prefix must not contain path traversal",
|
||||
},
|
||||
{
|
||||
name: "local and s3 audio conflict",
|
||||
sessionYAML: `session_id: 2026-05-03
|
||||
campaign: forsaken
|
||||
inputs:
|
||||
audio_dir: ./audio
|
||||
audio_s3:
|
||||
prefix: audio/
|
||||
speakers_file: ./speakers.yml
|
||||
autocorrect_file: ./autocorrect.yml
|
||||
glossary_file: ./glossary.yml
|
||||
`,
|
||||
wantErr: "mutually exclusive",
|
||||
},
|
||||
}
|
||||
|
||||
for _, tt := range tests {
|
||||
t.Run(tt.name, func(t *testing.T) {
|
||||
pipelineYAML := testPipelineBaseYAML + `
|
||||
storage:
|
||||
backend: s3
|
||||
s3:
|
||||
bucket: my-dnd-archive
|
||||
`
|
||||
pipelinePath, sessionPath := writeConfigFiles(t, pipelineYAML, tt.sessionYAML)
|
||||
|
||||
cfg, err := Load(pipelinePath, sessionPath)
|
||||
if err != nil {
|
||||
t.Fatalf("Load() error = %v", err)
|
||||
}
|
||||
err = Validate(cfg)
|
||||
if tt.wantErr != "" {
|
||||
if err == nil || !strings.Contains(err.Error(), tt.wantErr) {
|
||||
t.Fatalf("Validate() error = %v, want to contain %q", err, tt.wantErr)
|
||||
}
|
||||
return
|
||||
}
|
||||
if err != nil {
|
||||
t.Fatalf("Validate() error = %v", err)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestStorageS3BucketRequiredWhenS3DependentFeatureEnabled(t *testing.T) {
|
||||
pipelineYAML := testPipelineBaseYAML + `
|
||||
storage:
|
||||
backend: s3
|
||||
`
|
||||
sessionYAML := `session_id: 2026-05-03
|
||||
campaign: forsaken
|
||||
inputs:
|
||||
audio_s3:
|
||||
prefix: audio/
|
||||
speakers_file: ./speakers.yml
|
||||
autocorrect_file: ./autocorrect.yml
|
||||
glossary_file: ./glossary.yml
|
||||
`
|
||||
pipelinePath, sessionPath := writeConfigFiles(t, pipelineYAML, sessionYAML)
|
||||
cfg, err := Load(pipelinePath, sessionPath)
|
||||
if err != nil {
|
||||
t.Fatalf("Load() error = %v", err)
|
||||
}
|
||||
|
||||
err = Validate(cfg)
|
||||
if err == nil || !strings.Contains(err.Error(), "pipeline.storage.s3.bucket is required") {
|
||||
t.Fatalf("Validate() error = %v, want bucket requirement", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestLocalAudioConfigStillValid(t *testing.T) {
|
||||
pipelinePath, sessionPath := writeConfigFiles(t, testPipelineBaseYAML, testSessionBaseYAML)
|
||||
cfg, err := Load(pipelinePath, sessionPath)
|
||||
if err != nil {
|
||||
t.Fatalf("Load() error = %v", err)
|
||||
}
|
||||
if err := Validate(cfg); err != nil {
|
||||
t.Fatalf("Validate() error = %v", err)
|
||||
}
|
||||
}
|
||||
@@ -3,6 +3,8 @@ package config
|
||||
import (
|
||||
"fmt"
|
||||
"net/url"
|
||||
"path/filepath"
|
||||
"regexp"
|
||||
"strings"
|
||||
"time"
|
||||
)
|
||||
@@ -25,6 +27,9 @@ func Validate(cfg *Config) error {
|
||||
if err := validateSession(cfg.Session); err != nil {
|
||||
return fmt.Errorf("session config %q invalid: %w", shortName(cfg.SessionPath, "session.yml"), err)
|
||||
}
|
||||
if err := validateCrossConfig(cfg.Pipeline, cfg.Session); err != nil {
|
||||
return fmt.Errorf("pipeline/session config invalid: %w", err)
|
||||
}
|
||||
|
||||
return nil
|
||||
}
|
||||
@@ -36,6 +41,15 @@ func validatePipeline(cfg *PipelineConfig) error {
|
||||
if err := validateSecrets(cfg.Secrets); err != nil {
|
||||
return err
|
||||
}
|
||||
if err := validateStorage(cfg.Storage); err != nil {
|
||||
return err
|
||||
}
|
||||
if err := validateSpool(cfg.Spool); err != nil {
|
||||
return err
|
||||
}
|
||||
if err := validateArchive(cfg.Archive); err != nil {
|
||||
return err
|
||||
}
|
||||
if err := validateWhisperX(cfg.WhisperX); err != nil {
|
||||
return err
|
||||
}
|
||||
@@ -64,6 +78,48 @@ func validatePipeline(cfg *PipelineConfig) error {
|
||||
return nil
|
||||
}
|
||||
|
||||
func validateStorage(cfg StorageConfig) error {
|
||||
if cfg.S3 == nil {
|
||||
return nil
|
||||
}
|
||||
if strings.TrimSpace(cfg.S3.RootPrefix) == "" {
|
||||
return fmt.Errorf("pipeline.storage.s3.root_prefix must be non-empty")
|
||||
}
|
||||
if err := validateRelativeSafePath("pipeline.storage.s3.root_prefix", cfg.S3.RootPrefix); err != nil {
|
||||
return err
|
||||
}
|
||||
if cfg.S3.Endpoint != "" && strings.TrimSpace(cfg.S3.Endpoint) == "" {
|
||||
return fmt.Errorf("pipeline.storage.s3.endpoint must be non-empty when provided")
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func validateSpool(cfg SpoolConfig) error {
|
||||
return nil
|
||||
}
|
||||
|
||||
func validateArchive(cfg *ArchiveConfig) error {
|
||||
if cfg == nil {
|
||||
return nil
|
||||
}
|
||||
for i, item := range cfg.PromoteArtifacts {
|
||||
prefix := fmt.Sprintf("pipeline.archive.promote_artifacts[%d]", i)
|
||||
if strings.TrimSpace(item.From) == "" {
|
||||
return fmt.Errorf("%s.from is required", prefix)
|
||||
}
|
||||
if strings.TrimSpace(item.To) == "" {
|
||||
return fmt.Errorf("%s.to is required", prefix)
|
||||
}
|
||||
if err := validateRelativeSafePath(prefix+".from", item.From); err != nil {
|
||||
return err
|
||||
}
|
||||
if err := validateRelativeSafePath(prefix+".to", item.To); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func validateSecrets(cfg *SecretsConfig) error {
|
||||
if cfg == nil {
|
||||
return nil
|
||||
@@ -200,15 +256,12 @@ func validateAudita(cfg AuditaConfig) error {
|
||||
if err := validateDuration("pipeline.audita.timeout", cfg.Timeout); err != nil {
|
||||
return err
|
||||
}
|
||||
if len(cfg.Modules) == 0 {
|
||||
return fmt.Errorf("pipeline.audita.modules must include at least one module")
|
||||
}
|
||||
for i, mod := range cfg.Modules {
|
||||
m := strings.TrimSpace(mod)
|
||||
if m == "" {
|
||||
for i, m := range cfg.Modules {
|
||||
module := strings.TrimSpace(m)
|
||||
if module == "" {
|
||||
return fmt.Errorf("pipeline.audita.modules[%d] must be non-empty", i)
|
||||
}
|
||||
switch m {
|
||||
switch module {
|
||||
case "glossary", "homophones", "spoken_word", "grammar":
|
||||
default:
|
||||
return fmt.Errorf("pipeline.audita.modules[%d] must be one of: glossary, homophones, spoken_word, grammar", i)
|
||||
@@ -226,18 +279,31 @@ func validateAudita(cfg AuditaConfig) error {
|
||||
if strings.TrimSpace(cfg.Model) == "" {
|
||||
return fmt.Errorf("pipeline.audita.model is required")
|
||||
}
|
||||
if cfg.LLMConcurrency == nil {
|
||||
return fmt.Errorf("pipeline.audita.llm_concurrency must be set (defaults should populate this)")
|
||||
if cfg.TotalLLMConcurrency != nil && *cfg.TotalLLMConcurrency <= 0 {
|
||||
return fmt.Errorf("pipeline.audita.total_llm_concurrency must be > 0")
|
||||
}
|
||||
if *cfg.LLMConcurrency <= 0 {
|
||||
return fmt.Errorf("pipeline.audita.llm_concurrency must be > 0")
|
||||
if cfg.ProposalLLMConcurrency != nil && *cfg.ProposalLLMConcurrency <= 0 {
|
||||
return fmt.Errorf("pipeline.audita.proposal_llm_concurrency must be > 0")
|
||||
}
|
||||
if cfg.ValidationLLMConcurrency == nil {
|
||||
return fmt.Errorf("pipeline.audita.validation_llm_concurrency must be set (defaults should populate this)")
|
||||
}
|
||||
if *cfg.ValidationLLMConcurrency <= 0 {
|
||||
if cfg.ValidationLLMConcurrency != nil && *cfg.ValidationLLMConcurrency <= 0 {
|
||||
return fmt.Errorf("pipeline.audita.validation_llm_concurrency must be > 0")
|
||||
}
|
||||
if strings.TrimSpace(cfg.TranscriptDescription) == "" && cfg.TranscriptDescription != "" {
|
||||
return fmt.Errorf("pipeline.audita.transcript_description must be non-empty when provided")
|
||||
}
|
||||
if strings.TrimSpace(cfg.ConfigPath) == "" && cfg.ConfigPath != "" {
|
||||
return fmt.Errorf("pipeline.audita.config_path must be non-empty when provided")
|
||||
}
|
||||
switch strings.TrimSpace(cfg.OutputSchema) {
|
||||
case "", "bare-segments", "audita-v1":
|
||||
default:
|
||||
return fmt.Errorf("pipeline.audita.output_schema must be one of: bare-segments, audita-v1")
|
||||
}
|
||||
switch strings.TrimSpace(cfg.WorkDirRetention) {
|
||||
case "", "always", "auto", "never":
|
||||
default:
|
||||
return fmt.Errorf("pipeline.audita.work_dir_retention must be one of: always, auto, never")
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
@@ -297,6 +363,9 @@ func validateSession(cfg *SessionConfig) error {
|
||||
if strings.TrimSpace(cfg.SessionID) == "" {
|
||||
return fmt.Errorf("session.session_id is required")
|
||||
}
|
||||
if strings.TrimSpace(cfg.Campaign) == "" {
|
||||
return fmt.Errorf("session.campaign is required")
|
||||
}
|
||||
|
||||
if strings.TrimSpace(cfg.Inputs.SpeakersFile) == "" {
|
||||
return fmt.Errorf("session.inputs.speakers_file is required")
|
||||
@@ -310,13 +379,79 @@ func validateSession(cfg *SessionConfig) error {
|
||||
|
||||
hasAudioDir := strings.TrimSpace(cfg.Inputs.AudioDir) != ""
|
||||
hasAudioFiles := len(cfg.Inputs.AudioFiles) > 0
|
||||
if !hasAudioDir && !hasAudioFiles {
|
||||
return fmt.Errorf("session.inputs requires audio_dir or at least one audio_files entry")
|
||||
hasAudioS3 := cfg.Inputs.AudioS3 != nil
|
||||
if hasAudioS3 {
|
||||
if strings.TrimSpace(cfg.Inputs.AudioS3.Prefix) == "" {
|
||||
return fmt.Errorf("session.inputs.audio_s3.prefix is required when session.inputs.audio_s3 is configured")
|
||||
}
|
||||
if err := validateRelativeSafePath("session.inputs.audio_s3.prefix", cfg.Inputs.AudioS3.Prefix); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
if hasAudioS3 && (hasAudioDir || hasAudioFiles) {
|
||||
return fmt.Errorf("session.inputs.audio_dir/audio_files and session.inputs.audio_s3 are mutually exclusive")
|
||||
}
|
||||
if !hasAudioDir && !hasAudioFiles && !hasAudioS3 {
|
||||
return fmt.Errorf("session.inputs requires audio_dir, at least one audio_files entry, or audio_s3")
|
||||
}
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
func validateCrossConfig(pipeline *PipelineConfig, session *SessionConfig) error {
|
||||
if pipeline == nil || session == nil {
|
||||
return nil
|
||||
}
|
||||
if pipeline.Storage.S3 == nil {
|
||||
return nil
|
||||
}
|
||||
|
||||
audioS3Enabled := session.Inputs.AudioS3 != nil
|
||||
archiveUploadEnabled := archiveUploadConfiguredForS3(pipeline)
|
||||
if (audioS3Enabled || archiveUploadEnabled) && strings.TrimSpace(pipeline.Storage.S3.Bucket) == "" {
|
||||
return fmt.Errorf("pipeline.storage.s3.bucket is required when S3 session audio or archive upload is enabled")
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func archiveUploadConfiguredForS3(pipeline *PipelineConfig) bool {
|
||||
if pipeline == nil || pipeline.Archive == nil {
|
||||
return false
|
||||
}
|
||||
if !strings.EqualFold(strings.TrimSpace(pipeline.Storage.Backend), "s3") {
|
||||
return false
|
||||
}
|
||||
enabled := true
|
||||
if pipeline.Archive.Enabled != nil {
|
||||
enabled = *pipeline.Archive.Enabled
|
||||
}
|
||||
upload := true
|
||||
if pipeline.Archive.UploadRun != nil {
|
||||
upload = *pipeline.Archive.UploadRun
|
||||
}
|
||||
return enabled && upload
|
||||
}
|
||||
|
||||
var windowsAbsPathRE = regexp.MustCompile(`^[A-Za-z]:[\\/].*`)
|
||||
|
||||
func validateRelativeSafePath(fieldName, value string) error {
|
||||
trimmed := strings.TrimSpace(value)
|
||||
if trimmed == "" {
|
||||
return fmt.Errorf("%s must be non-empty", fieldName)
|
||||
}
|
||||
if filepath.IsAbs(trimmed) || strings.HasPrefix(trimmed, "/") || strings.HasPrefix(trimmed, "\\") || windowsAbsPathRE.MatchString(trimmed) {
|
||||
return fmt.Errorf("%s must be a relative path", fieldName)
|
||||
}
|
||||
|
||||
normalized := strings.ReplaceAll(trimmed, "\\", "/")
|
||||
for _, segment := range strings.Split(normalized, "/") {
|
||||
if segment == ".." {
|
||||
return fmt.Errorf("%s must not contain path traversal", fieldName)
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func validateDuration(fieldName, value string) error {
|
||||
trimmed := strings.TrimSpace(value)
|
||||
if trimmed == "" {
|
||||
|
||||
@@ -14,9 +14,15 @@ type ErrorRecord struct {
|
||||
|
||||
// InputRecord captures one resolved input and optional checksum.
|
||||
type InputRecord struct {
|
||||
Kind string `json:"kind"`
|
||||
Path string `json:"path"`
|
||||
Checksum string `json:"checksum,omitempty"`
|
||||
Kind string `json:"kind"`
|
||||
Path string `json:"path"`
|
||||
Checksum string `json:"checksum,omitempty"`
|
||||
Source string `json:"source,omitempty"`
|
||||
S3Bucket string `json:"s3_bucket,omitempty"`
|
||||
S3Key string `json:"s3_key,omitempty"`
|
||||
S3Size int64 `json:"s3_size,omitempty"`
|
||||
S3ETag string `json:"s3_etag,omitempty"`
|
||||
SpoolPath string `json:"spool_path,omitempty"`
|
||||
}
|
||||
|
||||
// ArtifactRecord captures one produced artifact and optional remote metadata.
|
||||
@@ -45,6 +51,13 @@ type StageRecord struct {
|
||||
// Manifest is the durable run-state record for a session execution.
|
||||
type Manifest struct {
|
||||
SessionID string `json:"session_id"`
|
||||
Campaign string `json:"campaign,omitempty"`
|
||||
RunID string `json:"run_id,omitempty"`
|
||||
LocalWorkDir string `json:"local_workdir,omitempty"`
|
||||
LocalSpoolDir string `json:"local_spool_dir,omitempty"`
|
||||
S3Bucket string `json:"s3_bucket,omitempty"`
|
||||
S3SessionPrefix string `json:"s3_session_prefix,omitempty"`
|
||||
S3RunPrefix string `json:"s3_run_prefix,omitempty"`
|
||||
PipelineVersion string `json:"pipeline_version,omitempty"`
|
||||
CreatedAt time.Time `json:"created_at"`
|
||||
UpdatedAt time.Time `json:"updated_at"`
|
||||
|
||||
@@ -22,6 +22,13 @@ func TestLocalStoreCreateSaveLoadRoundTrip(t *testing.T) {
|
||||
now := time.Date(2026, 5, 3, 12, 0, 0, 0, time.UTC)
|
||||
m.MarkStageRunning("prepare", now)
|
||||
m.MarkStageSucceeded("prepare", now.Add(2*time.Second), []ArtifactRecord{{Kind: "transcript", LocalPath: "transcripts/merged.json"}})
|
||||
m.Campaign = "forsaken"
|
||||
m.RunID = "20260515T031522Z-a1b2c3d4"
|
||||
m.LocalWorkDir = "/var/lib/narratio/work/forsaken/2026-05-03/20260515T031522Z-a1b2c3d4"
|
||||
m.LocalSpoolDir = "/var/spool/narratio/forsaken/2026-05-03/20260515T031522Z-a1b2c3d4/audio"
|
||||
m.S3Bucket = "my-dnd-archive"
|
||||
m.S3SessionPrefix = "dnd/campaigns/forsaken/sessions/2026-05-03/"
|
||||
m.S3RunPrefix = "dnd/campaigns/forsaken/sessions/2026-05-03/runs/20260515T031522Z-a1b2c3d4/"
|
||||
|
||||
path := filepath.Join(t.TempDir(), "manifest.json")
|
||||
if err := store.Save(ctx, path, m); err != nil {
|
||||
@@ -36,6 +43,12 @@ func TestLocalStoreCreateSaveLoadRoundTrip(t *testing.T) {
|
||||
if loaded.SessionID != "2026-05-03" {
|
||||
t.Fatalf("SessionID = %q, want %q", loaded.SessionID, "2026-05-03")
|
||||
}
|
||||
if loaded.Campaign != "forsaken" {
|
||||
t.Fatalf("Campaign = %q, want %q", loaded.Campaign, "forsaken")
|
||||
}
|
||||
if loaded.RunID != "20260515T031522Z-a1b2c3d4" {
|
||||
t.Fatalf("RunID = %q, want run id", loaded.RunID)
|
||||
}
|
||||
stage, ok := loaded.Stages["prepare"]
|
||||
if !ok {
|
||||
t.Fatalf("stage prepare not found")
|
||||
|
||||
525
internal/stage/archive.go
Normal file
525
internal/stage/archive.go
Normal file
@@ -0,0 +1,525 @@
|
||||
package stage
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"io/fs"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"sort"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"gitea.maximumdirect.net/eric/narratio/internal/adapters/storage"
|
||||
"gitea.maximumdirect.net/eric/narratio/internal/artifacts"
|
||||
"gitea.maximumdirect.net/eric/narratio/internal/config"
|
||||
"gitea.maximumdirect.net/eric/narratio/internal/manifest"
|
||||
)
|
||||
|
||||
type archiveStage struct{}
|
||||
|
||||
var archivePrerequisiteStages = []string{
|
||||
"prepare",
|
||||
"transcribe",
|
||||
"merge",
|
||||
"polish",
|
||||
"normalize",
|
||||
"trim",
|
||||
"analyze",
|
||||
}
|
||||
|
||||
var archiveRunUploadDirs = []string{
|
||||
"inputs",
|
||||
"transcripts",
|
||||
"artifacts",
|
||||
"reports",
|
||||
"config",
|
||||
"logs",
|
||||
}
|
||||
|
||||
func (archiveStage) Name() string { return "archive" }
|
||||
|
||||
func (archiveStage) Declares() IODecl {
|
||||
return IODecl{
|
||||
Inputs: []artifacts.Ref{
|
||||
{Kind: "manifest", Category: "input", RelativePath: "manifest.json"},
|
||||
},
|
||||
}
|
||||
}
|
||||
|
||||
func (archiveStage) Run(ctx context.Context, env *Env, m *manifest.Manifest) (*StageResult, error) {
|
||||
if env == nil || env.Config == nil || env.Config.Pipeline == nil || env.Config.Session == nil {
|
||||
return nil, fmt.Errorf("archive: resolved config must include pipeline and session")
|
||||
}
|
||||
|
||||
if archiveDisabled(env) {
|
||||
return &StageResult{
|
||||
Metadata: map[string]any{
|
||||
"stage": "archive",
|
||||
"skipped": true,
|
||||
"archive_enabled": false,
|
||||
"audio_upload_skipped": true,
|
||||
"current_pointer_written": false,
|
||||
},
|
||||
}, nil
|
||||
}
|
||||
if archiveRunUploadDisabled(env) {
|
||||
return &StageResult{
|
||||
Metadata: map[string]any{
|
||||
"stage": "archive",
|
||||
"skipped": true,
|
||||
"upload_run_enabled": false,
|
||||
"audio_upload_skipped": true,
|
||||
"current_pointer_written": false,
|
||||
},
|
||||
}, nil
|
||||
}
|
||||
|
||||
if err := validateArchivePrerequisites(m); err != nil {
|
||||
return nil, fmt.Errorf("archive: %w", err)
|
||||
}
|
||||
if env.ObjectStore == nil {
|
||||
return nil, fmt.Errorf("archive: remote object store backend is required when archive run upload is enabled")
|
||||
}
|
||||
|
||||
workDir, err := archiveWorkDir(env, m)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("archive: resolve local workdir: %w", err)
|
||||
}
|
||||
workDirInfo, err := os.Stat(workDir)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("archive: local workdir %q: %w", workDir, err)
|
||||
}
|
||||
if !workDirInfo.IsDir() {
|
||||
return nil, fmt.Errorf("archive: local workdir %q is not a directory", workDir)
|
||||
}
|
||||
|
||||
runPrefix, err := archiveRunPrefix(env, m)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("archive: resolve s3 run prefix: %w", err)
|
||||
}
|
||||
sessionPrefix, err := archiveSessionPrefix(env, m)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("archive: resolve s3 session prefix: %w", err)
|
||||
}
|
||||
bucket := archiveBucket(env, m)
|
||||
if bucket == "" {
|
||||
return nil, fmt.Errorf("archive: resolve s3 bucket: bucket is required")
|
||||
}
|
||||
runID := strings.TrimSpace(m.RunID)
|
||||
if runID == "" {
|
||||
return nil, fmt.Errorf("archive: run id is required")
|
||||
}
|
||||
|
||||
runFiles, err := collectArchiveRunFiles(workDir)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("archive: collect run files: %w", err)
|
||||
}
|
||||
promotions, err := resolveArchivePromotions(workDir, env.Config.Pipeline.Archive.PromoteArtifacts)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("archive: resolve promotion rules: %w", err)
|
||||
}
|
||||
currentManifestSource := filepath.Join(workDir, "manifest.json")
|
||||
if info, err := os.Stat(currentManifestSource); err != nil {
|
||||
return nil, fmt.Errorf("archive: current manifest source %q: %w", currentManifestSource, err)
|
||||
} else if info.IsDir() {
|
||||
return nil, fmt.Errorf("archive: current manifest source %q is a directory", currentManifestSource)
|
||||
}
|
||||
|
||||
runUploaded := make([]string, 0, len(runFiles))
|
||||
for _, rel := range runFiles {
|
||||
localPath := filepath.Join(workDir, filepath.FromSlash(rel))
|
||||
key := artifacts.S3RunRelativeDestinationKey(runPrefix, rel)
|
||||
if _, err := env.ObjectStore.Upload(ctx, localPath, key, storage.UploadOptions{}); err != nil {
|
||||
return nil, fmt.Errorf("archive: upload run file %q to %q: %w", rel, key, err)
|
||||
}
|
||||
runUploaded = append(runUploaded, rel)
|
||||
}
|
||||
|
||||
promotedUploaded := make([]string, 0, len(promotions))
|
||||
skippedOptional := make([]string, 0)
|
||||
for _, promotion := range promotions {
|
||||
if !promotion.Exists {
|
||||
if promotion.Required {
|
||||
return nil, fmt.Errorf("archive: required promotion source missing: %q", promotion.From)
|
||||
}
|
||||
skippedOptional = append(skippedOptional, promotion.To)
|
||||
continue
|
||||
}
|
||||
key := artifacts.S3PromotedArtifactKey(sessionPrefix, promotion.To)
|
||||
if _, err := env.ObjectStore.Upload(ctx, promotion.LocalPath, key, storage.UploadOptions{}); err != nil {
|
||||
return nil, fmt.Errorf("archive: upload promoted output %q to %q: %w", promotion.From, key, err)
|
||||
}
|
||||
promotedUploaded = append(promotedUploaded, promotion.To)
|
||||
}
|
||||
|
||||
currentManifestKey := artifacts.S3CurrentManifestKey(sessionPrefix)
|
||||
manifestTempPath, err := writeCurrentManifestSnapshot(m, archiveMetadataPreview(
|
||||
bucket,
|
||||
runPrefix,
|
||||
sessionPrefix,
|
||||
runUploaded,
|
||||
promotedUploaded,
|
||||
skippedOptional,
|
||||
currentManifestKey,
|
||||
))
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("archive: build current manifest snapshot: %w", err)
|
||||
}
|
||||
defer func() { _ = os.Remove(manifestTempPath) }()
|
||||
|
||||
if _, err := env.ObjectStore.Upload(ctx, manifestTempPath, currentManifestKey, storage.UploadOptions{
|
||||
ContentType: "application/json",
|
||||
}); err != nil {
|
||||
return nil, fmt.Errorf("archive: upload current manifest to %q: %w", currentManifestKey, err)
|
||||
}
|
||||
|
||||
currentRunPointerKey := artifacts.S3CurrentRunPointerKey(sessionPrefix)
|
||||
runIDTempPath, err := writeCurrentRunIDPointer(runID)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("archive: build current run id pointer: %w", err)
|
||||
}
|
||||
defer func() { _ = os.Remove(runIDTempPath) }()
|
||||
|
||||
if _, err := env.ObjectStore.Upload(ctx, runIDTempPath, currentRunPointerKey, storage.UploadOptions{
|
||||
ContentType: "text/plain; charset=utf-8",
|
||||
}); err != nil {
|
||||
return nil, fmt.Errorf("archive: upload current run pointer to %q: %w", currentRunPointerKey, err)
|
||||
}
|
||||
|
||||
return &StageResult{
|
||||
Metadata: map[string]any{
|
||||
"stage": "archive",
|
||||
"uploaded": true,
|
||||
"s3_bucket": bucket,
|
||||
"s3_run_prefix": runPrefix,
|
||||
"run_files_uploaded": len(runUploaded),
|
||||
"run_uploaded_paths": runUploaded,
|
||||
"promoted_files_uploaded": len(promotedUploaded),
|
||||
"promoted_paths": promotedUploaded,
|
||||
"skipped_optional_promotions": skippedOptional,
|
||||
"current_manifest_key": currentManifestKey,
|
||||
"current_run_id_key": currentRunPointerKey,
|
||||
"current_pointer_written": true,
|
||||
"audio_upload_skipped": true,
|
||||
},
|
||||
}, nil
|
||||
}
|
||||
|
||||
type archivePromotion struct {
|
||||
From string
|
||||
To string
|
||||
Required bool
|
||||
LocalPath string
|
||||
Exists bool
|
||||
}
|
||||
|
||||
func archiveDisabled(env *Env) bool {
|
||||
cfg := env.Config.Pipeline.Archive
|
||||
if cfg == nil {
|
||||
return true
|
||||
}
|
||||
return cfg.Enabled != nil && !*cfg.Enabled
|
||||
}
|
||||
|
||||
func archiveRunUploadDisabled(env *Env) bool {
|
||||
cfg := env.Config.Pipeline.Archive
|
||||
if cfg == nil {
|
||||
return true
|
||||
}
|
||||
return cfg.UploadRun != nil && !*cfg.UploadRun
|
||||
}
|
||||
|
||||
func validateArchivePrerequisites(m *manifest.Manifest) error {
|
||||
if m == nil {
|
||||
return fmt.Errorf("manifest is required")
|
||||
}
|
||||
for _, stageName := range archivePrerequisiteStages {
|
||||
sr := m.Stages[stageName]
|
||||
if sr == nil {
|
||||
return fmt.Errorf("prerequisite stage %q has not succeeded", stageName)
|
||||
}
|
||||
if sr.Status != manifest.StatusSucceeded {
|
||||
return fmt.Errorf("prerequisite stage %q status is %q (want %q)", stageName, sr.Status, manifest.StatusSucceeded)
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func archiveWorkDir(env *Env, m *manifest.Manifest) (string, error) {
|
||||
workDir := strings.TrimSpace(m.LocalWorkDir)
|
||||
if workDir != "" {
|
||||
cleaned := filepath.Clean(workDir)
|
||||
if info, err := os.Stat(cleaned); err == nil && info.IsDir() {
|
||||
return cleaned, nil
|
||||
}
|
||||
}
|
||||
|
||||
sessionID := strings.TrimSpace(env.Config.Session.SessionID)
|
||||
if sessionID == "" {
|
||||
sessionID = strings.TrimSpace(m.SessionID)
|
||||
}
|
||||
campaign := strings.TrimSpace(env.Config.Session.Campaign)
|
||||
if campaign == "" {
|
||||
campaign = strings.TrimSpace(m.Campaign)
|
||||
}
|
||||
runID := strings.TrimSpace(m.RunID)
|
||||
if runID == "" {
|
||||
return "", fmt.Errorf("run id is required")
|
||||
}
|
||||
if campaign == "" || sessionID == "" {
|
||||
return "", fmt.Errorf("campaign and session id are required")
|
||||
}
|
||||
runScoped := artifacts.SessionRunWorkDir(env.Config.Pipeline.Workspace.Root, campaign, sessionID, runID)
|
||||
if info, err := os.Stat(runScoped); err == nil && info.IsDir() {
|
||||
return runScoped, nil
|
||||
}
|
||||
legacy := artifacts.SessionWorkDir(env.Config.Pipeline.Workspace.Root, sessionID)
|
||||
return legacy, nil
|
||||
}
|
||||
|
||||
func archiveRunPrefix(env *Env, m *manifest.Manifest) (string, error) {
|
||||
runPrefix := strings.TrimSpace(m.S3RunPrefix)
|
||||
if runPrefix != "" {
|
||||
return runPrefix, nil
|
||||
}
|
||||
|
||||
sessionPrefix, err := archiveSessionPrefix(env, m)
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
runID := strings.TrimSpace(m.RunID)
|
||||
if runID == "" {
|
||||
return "", fmt.Errorf("run id is required")
|
||||
}
|
||||
return artifacts.S3RunPrefix(sessionPrefix, runID), nil
|
||||
}
|
||||
|
||||
func archiveSessionPrefix(env *Env, m *manifest.Manifest) (string, error) {
|
||||
if m != nil && strings.TrimSpace(m.S3SessionPrefix) != "" {
|
||||
return strings.TrimSpace(m.S3SessionPrefix), nil
|
||||
}
|
||||
|
||||
sessionID := strings.TrimSpace(env.Config.Session.SessionID)
|
||||
if sessionID == "" {
|
||||
sessionID = strings.TrimSpace(m.SessionID)
|
||||
}
|
||||
campaign := strings.TrimSpace(env.Config.Session.Campaign)
|
||||
if campaign == "" {
|
||||
campaign = strings.TrimSpace(m.Campaign)
|
||||
}
|
||||
if env.Config.Pipeline.Storage.S3 == nil {
|
||||
return "", fmt.Errorf("pipeline.storage.s3 configuration is required")
|
||||
}
|
||||
sessionPrefix := artifacts.S3SessionPrefix(env.Config.Pipeline.Storage.S3.RootPrefix, campaign, sessionID)
|
||||
if strings.TrimSpace(sessionPrefix) == "" {
|
||||
return "", fmt.Errorf("session prefix is required")
|
||||
}
|
||||
return sessionPrefix, nil
|
||||
}
|
||||
|
||||
func archiveBucket(env *Env, m *manifest.Manifest) string {
|
||||
if m != nil && strings.TrimSpace(m.S3Bucket) != "" {
|
||||
return strings.TrimSpace(m.S3Bucket)
|
||||
}
|
||||
if env == nil || env.Config == nil || env.Config.Pipeline == nil || env.Config.Pipeline.Storage.S3 == nil {
|
||||
return ""
|
||||
}
|
||||
return strings.TrimSpace(env.Config.Pipeline.Storage.S3.Bucket)
|
||||
}
|
||||
|
||||
func resolveArchivePromotions(workDir string, rules []config.ArchivePromotionRule) ([]archivePromotion, error) {
|
||||
out := make([]archivePromotion, 0, len(rules))
|
||||
for _, rule := range rules {
|
||||
from := strings.TrimSpace(rule.From)
|
||||
to := strings.TrimSpace(rule.To)
|
||||
required := rule.Required == nil || *rule.Required
|
||||
|
||||
localPath, err := resolveWorkDirRelativePath(workDir, from)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("promotion from %q: %w", from, err)
|
||||
}
|
||||
info, err := os.Stat(localPath)
|
||||
exists := err == nil && !info.IsDir()
|
||||
if err != nil && !os.IsNotExist(err) {
|
||||
return nil, fmt.Errorf("promotion source %q: %w", from, err)
|
||||
}
|
||||
|
||||
out = append(out, archivePromotion{
|
||||
From: from,
|
||||
To: to,
|
||||
Required: required,
|
||||
LocalPath: localPath,
|
||||
Exists: exists,
|
||||
})
|
||||
}
|
||||
return out, nil
|
||||
}
|
||||
|
||||
func resolveWorkDirRelativePath(workDir, rel string) (string, error) {
|
||||
rel = filepath.Clean(filepath.FromSlash(strings.TrimSpace(rel)))
|
||||
if rel == "." || rel == "" {
|
||||
return "", fmt.Errorf("relative path is required")
|
||||
}
|
||||
full := filepath.Join(workDir, rel)
|
||||
cleanedWork := filepath.Clean(workDir)
|
||||
cleanedFull := filepath.Clean(full)
|
||||
relative, err := filepath.Rel(cleanedWork, cleanedFull)
|
||||
if err != nil {
|
||||
return "", fmt.Errorf("compute relative path: %w", err)
|
||||
}
|
||||
if relative == ".." || strings.HasPrefix(relative, ".."+string(filepath.Separator)) {
|
||||
return "", fmt.Errorf("path escapes workdir")
|
||||
}
|
||||
return cleanedFull, nil
|
||||
}
|
||||
|
||||
func collectArchiveRunFiles(workDir string) ([]string, error) {
|
||||
files := make([]string, 0, 64)
|
||||
|
||||
for _, dirName := range archiveRunUploadDirs {
|
||||
fullDir := filepath.Join(workDir, dirName)
|
||||
info, err := os.Stat(fullDir)
|
||||
if err != nil {
|
||||
if os.IsNotExist(err) {
|
||||
continue
|
||||
}
|
||||
return nil, fmt.Errorf("stat %q: %w", fullDir, err)
|
||||
}
|
||||
if !info.IsDir() {
|
||||
continue
|
||||
}
|
||||
|
||||
if err := filepath.WalkDir(fullDir, func(path string, d fs.DirEntry, walkErr error) error {
|
||||
if walkErr != nil {
|
||||
return walkErr
|
||||
}
|
||||
if d.IsDir() {
|
||||
return nil
|
||||
}
|
||||
rel, err := filepath.Rel(workDir, path)
|
||||
if err != nil {
|
||||
return fmt.Errorf("relative path from %q to %q: %w", workDir, path, err)
|
||||
}
|
||||
rel = filepath.ToSlash(rel)
|
||||
files = append(files, rel)
|
||||
return nil
|
||||
}); err != nil {
|
||||
return nil, fmt.Errorf("walk %q: %w", fullDir, err)
|
||||
}
|
||||
}
|
||||
|
||||
manifestPath := filepath.Join(workDir, "manifest.json")
|
||||
manifestInfo, err := os.Stat(manifestPath)
|
||||
if err != nil {
|
||||
if os.IsNotExist(err) {
|
||||
return nil, fmt.Errorf("manifest.json not found in workdir %q", workDir)
|
||||
}
|
||||
return nil, fmt.Errorf("stat %q: %w", manifestPath, err)
|
||||
}
|
||||
if manifestInfo.IsDir() {
|
||||
return nil, fmt.Errorf("manifest path %q is a directory", manifestPath)
|
||||
}
|
||||
files = append(files, "manifest.json")
|
||||
|
||||
sort.Strings(files)
|
||||
return files, nil
|
||||
}
|
||||
|
||||
func writeCurrentManifestSnapshot(m *manifest.Manifest, archiveMetadata map[string]any) (string, error) {
|
||||
if m == nil {
|
||||
return "", fmt.Errorf("manifest is required")
|
||||
}
|
||||
clone := *m
|
||||
clone.Stages = make(map[string]*manifest.StageRecord, len(m.Stages))
|
||||
for name, sr := range m.Stages {
|
||||
if sr == nil {
|
||||
continue
|
||||
}
|
||||
stageCopy := *sr
|
||||
if sr.Outputs != nil {
|
||||
stageCopy.Outputs = append([]manifest.ArtifactRecord(nil), sr.Outputs...)
|
||||
}
|
||||
if sr.Logs != nil {
|
||||
stageCopy.Logs = append([]string(nil), sr.Logs...)
|
||||
}
|
||||
if sr.GeneratedConfigs != nil {
|
||||
stageCopy.GeneratedConfigs = append([]string(nil), sr.GeneratedConfigs...)
|
||||
}
|
||||
if sr.Metadata != nil {
|
||||
metaCopy := make(map[string]any, len(sr.Metadata))
|
||||
for k, v := range sr.Metadata {
|
||||
metaCopy[k] = v
|
||||
}
|
||||
stageCopy.Metadata = metaCopy
|
||||
}
|
||||
clone.Stages[name] = &stageCopy
|
||||
}
|
||||
|
||||
now := time.Now().UTC()
|
||||
clone.MarkStageSucceeded("archive", now, nil)
|
||||
if sr := clone.Stages["archive"]; sr != nil {
|
||||
sr.Metadata = archiveMetadata
|
||||
}
|
||||
|
||||
data, err := json.MarshalIndent(&clone, "", " ")
|
||||
if err != nil {
|
||||
return "", fmt.Errorf("marshal manifest: %w", err)
|
||||
}
|
||||
data = append(data, '\n')
|
||||
|
||||
tmp, err := os.CreateTemp("", "narratio-current-manifest-*.json")
|
||||
if err != nil {
|
||||
return "", fmt.Errorf("create temp file: %w", err)
|
||||
}
|
||||
path := tmp.Name()
|
||||
if _, err := tmp.Write(data); err != nil {
|
||||
_ = tmp.Close()
|
||||
return "", fmt.Errorf("write temp manifest: %w", err)
|
||||
}
|
||||
if err := tmp.Close(); err != nil {
|
||||
return "", fmt.Errorf("close temp manifest: %w", err)
|
||||
}
|
||||
return path, nil
|
||||
}
|
||||
|
||||
func writeCurrentRunIDPointer(runID string) (string, error) {
|
||||
tmp, err := os.CreateTemp("", "narratio-current-run-id-*.txt")
|
||||
if err != nil {
|
||||
return "", fmt.Errorf("create temp file: %w", err)
|
||||
}
|
||||
path := tmp.Name()
|
||||
if _, err := tmp.WriteString(runID + "\n"); err != nil {
|
||||
_ = tmp.Close()
|
||||
return "", fmt.Errorf("write temp run id pointer: %w", err)
|
||||
}
|
||||
if err := tmp.Close(); err != nil {
|
||||
return "", fmt.Errorf("close temp run id pointer: %w", err)
|
||||
}
|
||||
return path, nil
|
||||
}
|
||||
|
||||
func archiveMetadataPreview(
|
||||
bucket, runPrefix, sessionPrefix string,
|
||||
runUploaded []string,
|
||||
promotedUploaded []string,
|
||||
skippedOptional []string,
|
||||
currentManifestKey string,
|
||||
) map[string]any {
|
||||
return map[string]any{
|
||||
"stage": "archive",
|
||||
"uploaded": true,
|
||||
"s3_bucket": bucket,
|
||||
"s3_run_prefix": runPrefix,
|
||||
"run_files_uploaded": len(runUploaded),
|
||||
"run_uploaded_paths": append([]string(nil), runUploaded...),
|
||||
"promoted_files_uploaded": len(promotedUploaded),
|
||||
"promoted_paths": append([]string(nil), promotedUploaded...),
|
||||
"skipped_optional_promotions": append([]string(nil), skippedOptional...),
|
||||
"current_manifest_key": currentManifestKey,
|
||||
"current_run_id_key": artifacts.S3CurrentRunPointerKey(sessionPrefix),
|
||||
"current_pointer_written": false,
|
||||
"audio_upload_skipped": true,
|
||||
}
|
||||
}
|
||||
322
internal/stage/archive_test.go
Normal file
322
internal/stage/archive_test.go
Normal file
@@ -0,0 +1,322 @@
|
||||
package stage
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"path/filepath"
|
||||
"reflect"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"gitea.maximumdirect.net/eric/narratio/internal/adapters/storage"
|
||||
"gitea.maximumdirect.net/eric/narratio/internal/artifacts"
|
||||
"gitea.maximumdirect.net/eric/narratio/internal/config"
|
||||
"gitea.maximumdirect.net/eric/narratio/internal/manifest"
|
||||
)
|
||||
|
||||
func TestArchiveSkipsWhenDisabled(t *testing.T) {
|
||||
env, m, _ := archiveFixture(t)
|
||||
env.Config.Pipeline.Archive.Enabled = boolPtr(false)
|
||||
|
||||
result, err := archiveStage{}.Run(context.Background(), env, m)
|
||||
if err != nil {
|
||||
t.Fatalf("Run() error = %v", err)
|
||||
}
|
||||
if result.Metadata["skipped"] != true {
|
||||
t.Fatalf("metadata = %#v, want skipped=true", result.Metadata)
|
||||
}
|
||||
if len(env.ObjectStore.(*storage.FakeBackend).Uploads) != 0 {
|
||||
t.Fatalf("unexpected uploads when archive disabled")
|
||||
}
|
||||
}
|
||||
|
||||
func TestArchiveSkipsRunUploadWhenDisabled(t *testing.T) {
|
||||
env, m, _ := archiveFixture(t)
|
||||
env.Config.Pipeline.Archive.UploadRun = boolPtr(false)
|
||||
|
||||
result, err := archiveStage{}.Run(context.Background(), env, m)
|
||||
if err != nil {
|
||||
t.Fatalf("Run() error = %v", err)
|
||||
}
|
||||
if result.Metadata["skipped"] != true {
|
||||
t.Fatalf("metadata = %#v, want skipped=true", result.Metadata)
|
||||
}
|
||||
if len(env.ObjectStore.(*storage.FakeBackend).Uploads) != 0 {
|
||||
t.Fatalf("unexpected uploads when upload_run disabled")
|
||||
}
|
||||
}
|
||||
|
||||
func TestArchiveFailsWhenPrerequisiteNotSucceeded(t *testing.T) {
|
||||
env, m, _ := archiveFixture(t)
|
||||
m.Stages["trim"].Status = manifest.StatusFailed
|
||||
|
||||
_, err := archiveStage{}.Run(context.Background(), env, m)
|
||||
if err == nil || !strings.Contains(err.Error(), `prerequisite stage "trim"`) {
|
||||
t.Fatalf("Run() error = %v, want prerequisite failure", err)
|
||||
}
|
||||
if len(env.ObjectStore.(*storage.FakeBackend).Uploads) != 0 {
|
||||
t.Fatalf("unexpected uploads on prerequisite failure")
|
||||
}
|
||||
}
|
||||
|
||||
func TestArchiveUploadsRunRecordPromotionsAndCurrentPointer(t *testing.T) {
|
||||
env, m, _ := archiveFixture(t)
|
||||
fake := env.ObjectStore.(*storage.FakeBackend)
|
||||
|
||||
result, err := archiveStage{}.Run(context.Background(), env, m)
|
||||
if err != nil {
|
||||
t.Fatalf("Run() error = %v", err)
|
||||
}
|
||||
|
||||
runPrefix := m.S3RunPrefix
|
||||
sessionPrefix := m.S3SessionPrefix
|
||||
wantRunUploads := []string{
|
||||
"artifacts/session_recap.md",
|
||||
"config/audita.generated.yml",
|
||||
"inputs/session.yml",
|
||||
"logs/audita.stderr.log",
|
||||
"manifest.json",
|
||||
"reports/audita.report.json",
|
||||
"transcripts/raw/speaker.json",
|
||||
"transcripts/trimmed.json",
|
||||
}
|
||||
for _, rel := range wantRunUploads {
|
||||
key := runPrefix + rel
|
||||
if _, ok := fake.Objects[key]; !ok {
|
||||
t.Fatalf("missing run upload key %q", key)
|
||||
}
|
||||
}
|
||||
|
||||
trimmedKey := sessionPrefix + "transcripts/trimmed.json"
|
||||
recapKey := sessionPrefix + "artifacts/session_recap.md"
|
||||
if _, ok := fake.Objects[trimmedKey]; !ok {
|
||||
t.Fatalf("missing promoted key %q", trimmedKey)
|
||||
}
|
||||
if _, ok := fake.Objects[recapKey]; !ok {
|
||||
t.Fatalf("missing promoted key %q", recapKey)
|
||||
}
|
||||
|
||||
currentManifestKey := sessionPrefix + "current/manifest.json"
|
||||
currentRunIDKey := sessionPrefix + "current/run_id.txt"
|
||||
if _, ok := fake.Objects[currentManifestKey]; !ok {
|
||||
t.Fatalf("missing current manifest key %q", currentManifestKey)
|
||||
}
|
||||
if _, ok := fake.Objects[currentRunIDKey]; !ok {
|
||||
t.Fatalf("missing current run pointer key %q", currentRunIDKey)
|
||||
}
|
||||
if got := string(fake.Objects[currentRunIDKey].Data); got != m.RunID+"\n" {
|
||||
t.Fatalf("run pointer contents = %q, want %q", got, m.RunID+"\\n")
|
||||
}
|
||||
|
||||
audioKey := runPrefix + "audio/speaker.flac"
|
||||
if _, ok := fake.Objects[audioKey]; ok {
|
||||
t.Fatalf("audio key %q should not be uploaded", audioKey)
|
||||
}
|
||||
|
||||
uploads := fake.Uploads
|
||||
if len(uploads) == 0 {
|
||||
t.Fatal("expected uploads")
|
||||
}
|
||||
if uploads[len(uploads)-1].Key != currentRunIDKey {
|
||||
t.Fatalf("last upload key = %q, want current run pointer key %q", uploads[len(uploads)-1].Key, currentRunIDKey)
|
||||
}
|
||||
|
||||
if result.Metadata["current_pointer_written"] != true {
|
||||
t.Fatalf("metadata = %#v, want current_pointer_written=true", result.Metadata)
|
||||
}
|
||||
if result.Metadata["promoted_files_uploaded"] != 2 {
|
||||
t.Fatalf("metadata promoted_files_uploaded = %#v, want 2", result.Metadata["promoted_files_uploaded"])
|
||||
}
|
||||
}
|
||||
|
||||
func TestArchiveUsesCustomPromotionRules(t *testing.T) {
|
||||
env, m, workDir := archiveFixture(t)
|
||||
env.Config.Pipeline.Archive.PromoteArtifacts = []config.ArchivePromotionRule{
|
||||
{From: "transcripts/trimmed.json", To: "published/trimmed.json", Required: boolPtr(true)},
|
||||
{From: "artifacts/session_recap.md", To: "published/recap.md", Required: boolPtr(true)},
|
||||
}
|
||||
writeStageTestFile(t, filepath.Join(workDir, "published", "ignored.txt"), "ignore\n")
|
||||
|
||||
_, err := archiveStage{}.Run(context.Background(), env, m)
|
||||
if err != nil {
|
||||
t.Fatalf("Run() error = %v", err)
|
||||
}
|
||||
|
||||
fake := env.ObjectStore.(*storage.FakeBackend)
|
||||
if _, ok := fake.Objects[m.S3SessionPrefix+"published/trimmed.json"]; !ok {
|
||||
t.Fatalf("missing custom promoted trimmed key")
|
||||
}
|
||||
if _, ok := fake.Objects[m.S3SessionPrefix+"published/recap.md"]; !ok {
|
||||
t.Fatalf("missing custom promoted recap key")
|
||||
}
|
||||
}
|
||||
|
||||
func TestArchiveSkipsOptionalMissingPromotion(t *testing.T) {
|
||||
env, m, _ := archiveFixture(t)
|
||||
env.Config.Pipeline.Archive.PromoteArtifacts = []config.ArchivePromotionRule{
|
||||
{From: "transcripts/trimmed.json", To: "transcripts/trimmed.json", Required: boolPtr(true)},
|
||||
{From: "artifacts/optional.md", To: "artifacts/optional.md", Required: boolPtr(false)},
|
||||
}
|
||||
|
||||
result, err := archiveStage{}.Run(context.Background(), env, m)
|
||||
if err != nil {
|
||||
t.Fatalf("Run() error = %v", err)
|
||||
}
|
||||
got, _ := result.Metadata["skipped_optional_promotions"].([]string)
|
||||
want := []string{"artifacts/optional.md"}
|
||||
if !reflect.DeepEqual(got, want) {
|
||||
t.Fatalf("skipped_optional_promotions = %#v, want %#v", got, want)
|
||||
}
|
||||
}
|
||||
|
||||
func TestArchiveFailsWhenRequiredPromotionMissing(t *testing.T) {
|
||||
env, m, _ := archiveFixture(t)
|
||||
env.Config.Pipeline.Archive.PromoteArtifacts = []config.ArchivePromotionRule{
|
||||
{From: "artifacts/missing.md", To: "artifacts/missing.md", Required: boolPtr(true)},
|
||||
}
|
||||
|
||||
_, err := archiveStage{}.Run(context.Background(), env, m)
|
||||
if err == nil || !strings.Contains(err.Error(), "required promotion source missing") {
|
||||
t.Fatalf("Run() error = %v, want required promotion missing failure", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestArchiveDoesNotWriteCurrentPointerWhenPromotionUploadFails(t *testing.T) {
|
||||
env, m, _ := archiveFixture(t)
|
||||
fake := env.ObjectStore.(*storage.FakeBackend)
|
||||
trimmedKey := m.S3SessionPrefix + "transcripts/trimmed.json"
|
||||
|
||||
origUploadErr := fake.UploadErr
|
||||
fake.UploadErr = nil
|
||||
failingKey := trimmedKey
|
||||
fake.Uploads = nil
|
||||
|
||||
originalUpload := fake.Upload
|
||||
_ = originalUpload
|
||||
// Use UploadErr toggle by checking call sequence in postcondition.
|
||||
// First failure point is promotion upload; simulate by setting error immediately before promotion key write.
|
||||
// We cannot hook FakeBackend per-key without changing public behavior; use dedicated backend wrapper instead.
|
||||
env.ObjectStore = &promotionFailingStore{delegate: fake, failKey: failingKey}
|
||||
|
||||
_, err := archiveStage{}.Run(context.Background(), env, m)
|
||||
if err == nil || !strings.Contains(err.Error(), "promoted output") {
|
||||
t.Fatalf("Run() error = %v, want promotion upload failure", err)
|
||||
}
|
||||
if _, ok := fake.Objects[m.S3SessionPrefix+"current/run_id.txt"]; ok {
|
||||
t.Fatalf("unexpected current pointer write on promotion failure")
|
||||
}
|
||||
fake.UploadErr = origUploadErr
|
||||
}
|
||||
|
||||
func TestArchiveDoesNotWriteCurrentPointerWhenCurrentManifestUploadFails(t *testing.T) {
|
||||
env, m, _ := archiveFixture(t)
|
||||
fake := env.ObjectStore.(*storage.FakeBackend)
|
||||
env.ObjectStore = &promotionFailingStore{delegate: fake, failKey: m.S3SessionPrefix + "current/manifest.json"}
|
||||
|
||||
_, err := archiveStage{}.Run(context.Background(), env, m)
|
||||
if err == nil || !strings.Contains(err.Error(), "current manifest") {
|
||||
t.Fatalf("Run() error = %v, want current manifest upload failure", err)
|
||||
}
|
||||
if _, ok := fake.Objects[m.S3SessionPrefix+"current/run_id.txt"]; ok {
|
||||
t.Fatalf("unexpected current pointer write when current manifest upload fails")
|
||||
}
|
||||
}
|
||||
|
||||
func TestArchiveFailsWithoutObjectStore(t *testing.T) {
|
||||
env, m, _ := archiveFixture(t)
|
||||
env.ObjectStore = nil
|
||||
|
||||
_, err := archiveStage{}.Run(context.Background(), env, m)
|
||||
if err == nil || !strings.Contains(err.Error(), "object store") {
|
||||
t.Fatalf("Run() error = %v, want object store backend failure", err)
|
||||
}
|
||||
}
|
||||
|
||||
func archiveFixture(t *testing.T) (*Env, *manifest.Manifest, string) {
|
||||
t.Helper()
|
||||
|
||||
root := t.TempDir()
|
||||
runID := "20260516T010203Z-1a2b3c4d"
|
||||
campaign := "forsaken"
|
||||
sessionID := "2026-04-19"
|
||||
workDir := filepath.Join(root, "work", campaign, sessionID, runID)
|
||||
|
||||
writeStageTestFile(t, filepath.Join(workDir, "inputs", "session.yml"), "session_id: 2026-04-19\n")
|
||||
writeStageTestFile(t, filepath.Join(workDir, "transcripts", "raw", "speaker.json"), "{}\n")
|
||||
writeStageTestFile(t, filepath.Join(workDir, "transcripts", "trimmed.json"), "{}\n")
|
||||
writeStageTestFile(t, filepath.Join(workDir, "artifacts", "session_recap.md"), "# recap\n")
|
||||
writeStageTestFile(t, filepath.Join(workDir, "reports", "audita.report.json"), "{}\n")
|
||||
writeStageTestFile(t, filepath.Join(workDir, "config", "audita.generated.yml"), "key: value\n")
|
||||
writeStageTestFile(t, filepath.Join(workDir, "logs", "audita.stderr.log"), "stderr\n")
|
||||
writeStageTestFile(t, filepath.Join(workDir, "audio", "speaker.flac"), "flac")
|
||||
writeStageTestFile(t, filepath.Join(workDir, "manifest.json"), "{}\n")
|
||||
|
||||
m := manifest.New(sessionID, time.Date(2026, 5, 16, 1, 2, 3, 0, time.UTC))
|
||||
m.Campaign = campaign
|
||||
m.RunID = runID
|
||||
m.LocalWorkDir = workDir
|
||||
m.S3Bucket = "my-dnd-archive"
|
||||
m.S3SessionPrefix = artifacts.S3SessionPrefix("dnd", campaign, sessionID)
|
||||
m.S3RunPrefix = artifacts.S3RunPrefix(m.S3SessionPrefix, runID)
|
||||
for _, name := range archivePrerequisiteStages {
|
||||
m.MarkStageSucceeded(name, time.Date(2026, 5, 16, 1, 2, 3, 0, time.UTC), nil)
|
||||
}
|
||||
|
||||
env := &Env{
|
||||
Config: &config.Config{
|
||||
Pipeline: &config.PipelineConfig{
|
||||
Workspace: config.WorkspaceConfig{Root: root},
|
||||
Storage: config.StorageConfig{
|
||||
S3: &config.StorageS3Config{
|
||||
Bucket: "my-dnd-archive",
|
||||
RootPrefix: "dnd",
|
||||
},
|
||||
},
|
||||
Archive: &config.ArchiveConfig{
|
||||
Enabled: boolPtr(true),
|
||||
UploadRun: boolPtr(true),
|
||||
PromoteArtifacts: []config.ArchivePromotionRule{
|
||||
{From: "transcripts/trimmed.json", To: "transcripts/trimmed.json", Required: boolPtr(true)},
|
||||
{From: "artifacts/session_recap.md", To: "artifacts/session_recap.md", Required: boolPtr(true)},
|
||||
},
|
||||
},
|
||||
},
|
||||
Session: &config.SessionConfig{
|
||||
SessionID: sessionID,
|
||||
Campaign: campaign,
|
||||
},
|
||||
},
|
||||
ObjectStore: &storage.FakeBackend{},
|
||||
}
|
||||
return env, m, workDir
|
||||
}
|
||||
|
||||
type promotionFailingStore struct {
|
||||
delegate *storage.FakeBackend
|
||||
failKey string
|
||||
}
|
||||
|
||||
func (s *promotionFailingStore) List(ctx context.Context, prefix string) ([]storage.ObjectInfo, error) {
|
||||
return s.delegate.List(ctx, prefix)
|
||||
}
|
||||
|
||||
func (s *promotionFailingStore) Download(ctx context.Context, key, localPath string) error {
|
||||
return s.delegate.Download(ctx, key, localPath)
|
||||
}
|
||||
|
||||
func (s *promotionFailingStore) Upload(ctx context.Context, localPath, key string, opts storage.UploadOptions) (storage.ObjectInfo, error) {
|
||||
if strings.TrimSpace(key) == strings.TrimSpace(s.failKey) {
|
||||
return storage.ObjectInfo{}, errors.New("forced upload failure")
|
||||
}
|
||||
return s.delegate.Upload(ctx, localPath, key, opts)
|
||||
}
|
||||
|
||||
func (s *promotionFailingStore) Exists(ctx context.Context, key string) (bool, error) {
|
||||
return s.delegate.Exists(ctx, key)
|
||||
}
|
||||
|
||||
func boolPtr(v bool) *bool {
|
||||
p := v
|
||||
return &p
|
||||
}
|
||||
@@ -3,10 +3,8 @@ package stage
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"path/filepath"
|
||||
|
||||
"gitea.maximumdirect.net/eric/narratio/internal/adapters/notify"
|
||||
"gitea.maximumdirect.net/eric/narratio/internal/adapters/storage"
|
||||
"gitea.maximumdirect.net/eric/narratio/internal/artifacts"
|
||||
"gitea.maximumdirect.net/eric/narratio/internal/manifest"
|
||||
)
|
||||
@@ -47,25 +45,7 @@ func (s placeholderStage) Run(ctx context.Context, env *Env, m *manifest.Manifes
|
||||
return result, nil
|
||||
}
|
||||
|
||||
paths := env.ArtifactStore.SessionPaths(sessionID)
|
||||
|
||||
switch s.name {
|
||||
case "archive":
|
||||
if env.Storage != nil {
|
||||
req := storage.ArchiveRequest{
|
||||
SessionID: sessionID,
|
||||
ManifestPath: paths.ManifestPath,
|
||||
Items: []storage.ArchiveItem{{
|
||||
Kind: "artifact",
|
||||
LocalPath: filepath.Join(paths.ArtifactsDir, "session-log.md"),
|
||||
RemoteKey: "sessions/" + sessionID + "/artifacts/session-log.md",
|
||||
}},
|
||||
}
|
||||
_, err := env.Storage.Archive(ctx, req)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("placeholder archive adapter call failed: %w", err)
|
||||
}
|
||||
}
|
||||
case "notify":
|
||||
if env.Notifier != nil {
|
||||
req := notify.SendRequest{
|
||||
@@ -95,7 +75,7 @@ func All() []Stage {
|
||||
normalizeStage{},
|
||||
trimStage{},
|
||||
analyzeStage{},
|
||||
placeholderStage{name: "archive"},
|
||||
archiveStage{},
|
||||
placeholderStage{name: "notify"},
|
||||
}
|
||||
}
|
||||
|
||||
@@ -49,7 +49,19 @@ func TestStagesReturnExpectedMetadata(t *testing.T) {
|
||||
Config: &config.Config{
|
||||
SessionPath: sessionPath,
|
||||
PipelinePath: pipelinePath,
|
||||
Pipeline: &config.PipelineConfig{Workspace: config.WorkspaceConfig{Root: root}},
|
||||
Pipeline: &config.PipelineConfig{
|
||||
Workspace: config.WorkspaceConfig{Root: root},
|
||||
Storage: config.StorageConfig{
|
||||
S3: &config.StorageS3Config{
|
||||
Bucket: "my-dnd-archive",
|
||||
RootPrefix: "dnd",
|
||||
},
|
||||
},
|
||||
Archive: &config.ArchiveConfig{
|
||||
Enabled: boolPtr(true),
|
||||
UploadRun: boolPtr(true),
|
||||
},
|
||||
},
|
||||
Session: &config.SessionConfig{
|
||||
SessionID: "2026-05-03",
|
||||
Inputs: config.SessionInputsConfig{
|
||||
@@ -66,10 +78,24 @@ func TestStagesReturnExpectedMetadata(t *testing.T) {
|
||||
Audita: af,
|
||||
Scriptorium: sc,
|
||||
Storage: st,
|
||||
ObjectStore: st,
|
||||
Notifier: nf,
|
||||
}
|
||||
|
||||
m := manifest.New("2026-05-03", time.Now().UTC())
|
||||
for _, name := range []string{"prepare", "transcribe", "merge", "polish", "normalize", "trim", "analyze"} {
|
||||
m.MarkStageSucceeded(name, time.Now().UTC(), nil)
|
||||
}
|
||||
m.RunID = "20260516T000000Z-abcdef12"
|
||||
m.Campaign = "sample-campaign"
|
||||
m.LocalWorkDir = filepath.Join(root, "work", "sample-campaign", "2026-05-03", m.RunID)
|
||||
m.S3RunPrefix = "dnd/campaigns/sample-campaign/sessions/2026-05-03/runs/" + m.RunID + "/"
|
||||
m.S3Bucket = "my-dnd-archive"
|
||||
if err := os.MkdirAll(filepath.Join(m.LocalWorkDir, "inputs"), 0o755); err != nil {
|
||||
t.Fatalf("mkdir workdir inputs: %v", err)
|
||||
}
|
||||
writeStageTestFile(t, filepath.Join(m.LocalWorkDir, "inputs", "session.yml"), "session_id: 2026-05-03\n")
|
||||
writeStageTestFile(t, filepath.Join(m.LocalWorkDir, "manifest.json"), "{}\n")
|
||||
for _, s := range stages {
|
||||
result, err := s.Run(context.Background(), env, m)
|
||||
if err != nil {
|
||||
@@ -150,6 +176,15 @@ func TestStagesReturnExpectedMetadata(t *testing.T) {
|
||||
}
|
||||
continue
|
||||
}
|
||||
if s.Name() == "archive" {
|
||||
if result.Metadata["stage"] != "archive" {
|
||||
t.Fatalf("archive metadata = %#v, want stage=archive", result.Metadata)
|
||||
}
|
||||
if result.Metadata["uploaded"] != true {
|
||||
t.Fatalf("archive metadata = %#v, want uploaded=true", result.Metadata)
|
||||
}
|
||||
continue
|
||||
}
|
||||
if result.Metadata["placeholder"] != true {
|
||||
t.Fatalf("stage %q missing placeholder metadata", s.Name())
|
||||
}
|
||||
@@ -167,8 +202,11 @@ func TestStagesReturnExpectedMetadata(t *testing.T) {
|
||||
if len(sc.RunRequests) != 0 {
|
||||
t.Fatalf("scriptorium run calls = %d, want 0 when scriptorium config is absent", len(sc.RunRequests))
|
||||
}
|
||||
if len(st.Requests) != 1 {
|
||||
t.Fatalf("storage calls = %d, want 1", len(st.Requests))
|
||||
if len(st.Requests) != 0 {
|
||||
t.Fatalf("storage archive calls = %d, want 0", len(st.Requests))
|
||||
}
|
||||
if _, ok := st.Objects["dnd/campaigns/sample-campaign/sessions/2026-05-03/runs/"+m.RunID+"/manifest.json"]; !ok {
|
||||
t.Fatalf("archive upload missing manifest key in fake object store")
|
||||
}
|
||||
if len(nf.Requests) != 1 {
|
||||
t.Fatalf("notify calls = %d, want 1", len(nf.Requests))
|
||||
@@ -181,7 +219,6 @@ func TestPlaceholderAdapterErrorPropagation(t *testing.T) {
|
||||
env *Env
|
||||
wantErr string
|
||||
}{
|
||||
{stageName: "archive", env: &Env{Storage: &storage.FakeBackend{Err: errors.New("sterr")}}, wantErr: "archive"},
|
||||
{stageName: "notify", env: &Env{Notifier: ¬ify.FakeSender{Err: errors.New("nerr")}}, wantErr: "notify"},
|
||||
}
|
||||
|
||||
|
||||
@@ -90,6 +90,12 @@ func (polishStage) Run(ctx context.Context, env *Env, m *manifest.Manifest) (*St
|
||||
Modules: append([]string(nil), env.Config.Pipeline.Audita.Modules...),
|
||||
BaseURL: env.Config.Pipeline.Audita.BaseURL,
|
||||
Model: env.Config.Pipeline.Audita.Model,
|
||||
TranscriptDescription: env.Config.Pipeline.Audita.TranscriptDescription,
|
||||
ConfigPath: env.Config.Pipeline.Audita.ConfigPath,
|
||||
OutputSchema: env.Config.Pipeline.Audita.OutputSchema,
|
||||
WorkDirRetention: env.Config.Pipeline.Audita.WorkDirRetention,
|
||||
TotalLLMConcurrency: env.Config.Pipeline.Audita.TotalLLMConcurrency,
|
||||
ProposalLLMConcurrency: env.Config.Pipeline.Audita.ProposalLLMConcurrency,
|
||||
ValidationModel: env.Config.Pipeline.Audita.ValidationModel,
|
||||
ValidationLLMConcurrency: env.Config.Pipeline.Audita.ValidationLLMConcurrency,
|
||||
StdoutLogPath: stdoutPath,
|
||||
@@ -141,44 +147,52 @@ func (polishStage) Run(ctx context.Context, env *Env, m *manifest.Manifest) (*St
|
||||
if env.Config.Pipeline.Audita.ValidationLLMConcurrency != nil {
|
||||
validationConcurrency = *env.Config.Pipeline.Audita.ValidationLLMConcurrency
|
||||
}
|
||||
var llmConcurrency any
|
||||
if env.Config.Pipeline.Audita.LLMConcurrency != nil {
|
||||
llmConcurrency = *env.Config.Pipeline.Audita.LLMConcurrency
|
||||
var totalLLMConcurrency any
|
||||
if env.Config.Pipeline.Audita.TotalLLMConcurrency != nil {
|
||||
totalLLMConcurrency = *env.Config.Pipeline.Audita.TotalLLMConcurrency
|
||||
}
|
||||
var proposalLLMConcurrency any
|
||||
if env.Config.Pipeline.Audita.ProposalLLMConcurrency != nil {
|
||||
proposalLLMConcurrency = *env.Config.Pipeline.Audita.ProposalLLMConcurrency
|
||||
}
|
||||
|
||||
meta := map[string]any{
|
||||
"stage": "polish",
|
||||
"merged_transcript_path": mergedPath,
|
||||
"merged_transcript_source": source,
|
||||
"glossary_path": glossaryPath,
|
||||
"output_path": finalProcessedPath,
|
||||
"report_path": finalReportPath,
|
||||
"audita_work_dir": workDir,
|
||||
"report_enabled": reportEnabled,
|
||||
"modules": append([]string(nil), req.Modules...),
|
||||
"base_url": req.BaseURL,
|
||||
"model": req.Model,
|
||||
"validation_model": req.ValidationModel,
|
||||
"llm_concurrency": llmConcurrency,
|
||||
"validation_llm_concurrency": validationConcurrency,
|
||||
"llm_api_key_env": env.Config.Pipeline.Audita.LLMAPIKeyEnv,
|
||||
"timeout": env.Config.Pipeline.Audita.Timeout,
|
||||
"binary": env.Config.Pipeline.Audita.Binary,
|
||||
"generated_config_path": generatedConfigPath,
|
||||
"stdout_log_path": stdoutPath,
|
||||
"stderr_log_path": stderrPath,
|
||||
"adapter_duration_ms": res.Duration.Milliseconds(),
|
||||
"adapter_exit_code": res.ExitCode,
|
||||
"adapter_invoked_binary": res.InvokedBinary,
|
||||
"adapter_processed_output_path": res.ProcessedTranscriptPath,
|
||||
"adapter_report_path": res.ReportPath,
|
||||
"adapter_generated_config_path": res.GeneratedConfigPath,
|
||||
"adapter_work_dir": res.WorkDir,
|
||||
"adapter_stdout_log_path": res.StdoutLogPath,
|
||||
"adapter_stderr_log_path": res.StderrLogPath,
|
||||
"credential_env_var": env.Config.Pipeline.Audita.LLMAPIKeyEnv,
|
||||
"credential_present": false,
|
||||
"primary_llm_concurrency_via_env": false,
|
||||
"stage": "polish",
|
||||
"merged_transcript_path": mergedPath,
|
||||
"merged_transcript_source": source,
|
||||
"glossary_path": glossaryPath,
|
||||
"output_path": finalProcessedPath,
|
||||
"report_path": finalReportPath,
|
||||
"audita_work_dir": workDir,
|
||||
"report_enabled": reportEnabled,
|
||||
"modules": append([]string(nil), req.Modules...),
|
||||
"base_url": req.BaseURL,
|
||||
"model": req.Model,
|
||||
"transcript_description": req.TranscriptDescription,
|
||||
"config_path": req.ConfigPath,
|
||||
"output_schema": req.OutputSchema,
|
||||
"work_dir_retention": req.WorkDirRetention,
|
||||
"validation_model": req.ValidationModel,
|
||||
"total_llm_concurrency": totalLLMConcurrency,
|
||||
"proposal_llm_concurrency": proposalLLMConcurrency,
|
||||
"validation_llm_concurrency": validationConcurrency,
|
||||
"llm_api_key_env": env.Config.Pipeline.Audita.LLMAPIKeyEnv,
|
||||
"timeout": env.Config.Pipeline.Audita.Timeout,
|
||||
"binary": env.Config.Pipeline.Audita.Binary,
|
||||
"generated_config_path": generatedConfigPath,
|
||||
"stdout_log_path": stdoutPath,
|
||||
"stderr_log_path": stderrPath,
|
||||
"adapter_duration_ms": res.Duration.Milliseconds(),
|
||||
"adapter_exit_code": res.ExitCode,
|
||||
"adapter_invoked_binary": res.InvokedBinary,
|
||||
"adapter_processed_output_path": res.ProcessedTranscriptPath,
|
||||
"adapter_report_path": res.ReportPath,
|
||||
"adapter_generated_config_path": res.GeneratedConfigPath,
|
||||
"adapter_work_dir": res.WorkDir,
|
||||
"adapter_stdout_log_path": res.StdoutLogPath,
|
||||
"adapter_stderr_log_path": res.StderrLogPath,
|
||||
"credential_env_var": env.Config.Pipeline.Audita.LLMAPIKeyEnv,
|
||||
"credential_present": false,
|
||||
}
|
||||
if res.Metadata != nil {
|
||||
meta["adapter_metadata"] = res.Metadata
|
||||
@@ -188,9 +202,6 @@ func (polishStage) Run(ctx context.Context, env *Env, m *manifest.Manifest) (*St
|
||||
if value, ok := res.Metadata["credential_env_var"]; ok {
|
||||
meta["credential_env_var"] = value
|
||||
}
|
||||
if value, ok := res.Metadata["primary_llm_concurrency_via_env"]; ok {
|
||||
meta["primary_llm_concurrency_via_env"] = value
|
||||
}
|
||||
}
|
||||
|
||||
return &StageResult{
|
||||
|
||||
@@ -60,6 +60,24 @@ func TestPolishStagePolishesMergedTranscriptAndRecordsMetadata(t *testing.T) {
|
||||
if req.ValidationModel != "openrouter/google/gemma-4-31b-it" {
|
||||
t.Fatalf("validation model = %q", req.ValidationModel)
|
||||
}
|
||||
if req.TranscriptDescription != "Campaign Session 42" {
|
||||
t.Fatalf("transcript description = %q", req.TranscriptDescription)
|
||||
}
|
||||
if req.ConfigPath != "/etc/audita/config.yml" {
|
||||
t.Fatalf("config path = %q", req.ConfigPath)
|
||||
}
|
||||
if req.OutputSchema != "audita-v1" {
|
||||
t.Fatalf("output schema = %q", req.OutputSchema)
|
||||
}
|
||||
if req.WorkDirRetention != "auto" {
|
||||
t.Fatalf("work dir retention = %q", req.WorkDirRetention)
|
||||
}
|
||||
if req.TotalLLMConcurrency == nil || *req.TotalLLMConcurrency != 3 {
|
||||
t.Fatalf("total llm concurrency = %#v, want 3", req.TotalLLMConcurrency)
|
||||
}
|
||||
if req.ProposalLLMConcurrency == nil || *req.ProposalLLMConcurrency != 2 {
|
||||
t.Fatalf("proposal llm concurrency = %#v, want 2", req.ProposalLLMConcurrency)
|
||||
}
|
||||
if req.ValidationLLMConcurrency == nil || *req.ValidationLLMConcurrency != 2 {
|
||||
t.Fatalf("validation llm concurrency = %#v, want 2", req.ValidationLLMConcurrency)
|
||||
}
|
||||
@@ -89,6 +107,15 @@ func TestPolishStagePolishesMergedTranscriptAndRecordsMetadata(t *testing.T) {
|
||||
if result.Metadata["audita_work_dir"] != filepath.Join(paths.ArtifactsDir, "audita-work") {
|
||||
t.Fatalf("metadata audita_work_dir = %#v", result.Metadata["audita_work_dir"])
|
||||
}
|
||||
if result.Metadata["total_llm_concurrency"] != 3 {
|
||||
t.Fatalf("metadata total_llm_concurrency = %#v, want 3", result.Metadata["total_llm_concurrency"])
|
||||
}
|
||||
if result.Metadata["proposal_llm_concurrency"] != 2 {
|
||||
t.Fatalf("metadata proposal_llm_concurrency = %#v, want 2", result.Metadata["proposal_llm_concurrency"])
|
||||
}
|
||||
if result.Metadata["output_schema"] != "audita-v1" {
|
||||
t.Fatalf("metadata output_schema = %#v, want audita-v1", result.Metadata["output_schema"])
|
||||
}
|
||||
}
|
||||
|
||||
func TestPolishStageFallsBackToMergedTranscriptPath(t *testing.T) {
|
||||
@@ -221,7 +248,8 @@ func setupPolishEnv(t *testing.T) (*Env, *manifest.Manifest) {
|
||||
writeFile(t, pipelinePath, "workspace:\n root: "+workspace+"\n")
|
||||
|
||||
report := true
|
||||
llmConcurrency := 1
|
||||
totalLLMConcurrency := 3
|
||||
proposalLLMConcurrency := 2
|
||||
validationLLMConcurrency := 2
|
||||
|
||||
cfg := &config.Config{
|
||||
@@ -236,8 +264,13 @@ func setupPolishEnv(t *testing.T) (*Env, *manifest.Manifest) {
|
||||
Modules: []string{"glossary", "homophones", "grammar"},
|
||||
BaseURL: "https://openrouter.ai/api/v1",
|
||||
Model: "openrouter/google/gemma-4-31b-it",
|
||||
TranscriptDescription: "Campaign Session 42",
|
||||
ConfigPath: "/etc/audita/config.yml",
|
||||
OutputSchema: "audita-v1",
|
||||
WorkDirRetention: "auto",
|
||||
ValidationModel: "openrouter/google/gemma-4-31b-it",
|
||||
LLMConcurrency: &llmConcurrency,
|
||||
TotalLLMConcurrency: &totalLLMConcurrency,
|
||||
ProposalLLMConcurrency: &proposalLLMConcurrency,
|
||||
ValidationLLMConcurrency: &validationLLMConcurrency,
|
||||
Report: &report,
|
||||
},
|
||||
|
||||
@@ -6,10 +6,12 @@ import (
|
||||
"encoding/hex"
|
||||
"fmt"
|
||||
"os"
|
||||
"path"
|
||||
"path/filepath"
|
||||
"sort"
|
||||
"strings"
|
||||
|
||||
"gitea.maximumdirect.net/eric/narratio/internal/adapters/storage"
|
||||
"gitea.maximumdirect.net/eric/narratio/internal/artifacts"
|
||||
"gitea.maximumdirect.net/eric/narratio/internal/config"
|
||||
"gitea.maximumdirect.net/eric/narratio/internal/manifest"
|
||||
@@ -33,7 +35,7 @@ func (prepareStage) Declares() IODecl {
|
||||
}
|
||||
}
|
||||
|
||||
func (prepareStage) Run(_ context.Context, env *Env, m *manifest.Manifest) (*StageResult, error) {
|
||||
func (prepareStage) Run(ctx context.Context, env *Env, m *manifest.Manifest) (*StageResult, error) {
|
||||
if env == nil || env.Config == nil {
|
||||
return nil, fmt.Errorf("prepare: stage environment config is required")
|
||||
}
|
||||
@@ -89,13 +91,12 @@ func (prepareStage) Run(_ context.Context, env *Env, m *manifest.Manifest) (*Sta
|
||||
}
|
||||
}
|
||||
|
||||
resolvedAudio, err := resolveAudioFiles(sessionDir, env.Config.Session.Inputs)
|
||||
resolvedLocalAudio, useS3Audio, err := resolveAudioInputs(sessionDir, env.Config.Session.Inputs)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("prepare: resolve audio inputs: %w", err)
|
||||
}
|
||||
|
||||
copiedByDest := map[string]string{}
|
||||
inputs := make([]manifest.InputRecord, 0, 5+len(resolvedAudio))
|
||||
inputs := make([]manifest.InputRecord, 0, 5+len(resolvedLocalAudio))
|
||||
registerInput := func(kind, path, checksum string) {
|
||||
inputs = append(inputs, manifest.InputRecord{Kind: kind, Path: path, Checksum: checksum})
|
||||
}
|
||||
@@ -134,19 +135,14 @@ func (prepareStage) Run(_ context.Context, env *Env, m *manifest.Manifest) (*Sta
|
||||
registerInput(cfgFile.kind, cfgFile.dst, checksum)
|
||||
}
|
||||
|
||||
for _, src := range resolvedAudio {
|
||||
base := filepath.Base(src)
|
||||
if prev, exists := copiedByDest[base]; exists && prev != src {
|
||||
return nil, fmt.Errorf("prepare: duplicate audio basename %q from %q and %q", base, prev, src)
|
||||
if useS3Audio {
|
||||
if err := materializeS3AudioInputs(ctx, env, m, sessionID, &inputs); err != nil {
|
||||
return nil, fmt.Errorf("prepare: materialize s3 audio: %w", err)
|
||||
}
|
||||
copiedByDest[base] = src
|
||||
|
||||
dst := filepath.Join(paths.AudioDir, base)
|
||||
checksum, err := copyFileIfChanged(env.ArtifactStore, src, dst)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("prepare: materialize audio %q: %w", base, err)
|
||||
} else {
|
||||
if err := materializeLocalAudioInputs(env, paths, resolvedLocalAudio, registerInput); err != nil {
|
||||
return nil, fmt.Errorf("prepare: %w", err)
|
||||
}
|
||||
registerInput("audio", dst, checksum)
|
||||
}
|
||||
|
||||
sort.Slice(inputs, func(i, j int) bool {
|
||||
@@ -162,7 +158,7 @@ func (prepareStage) Run(_ context.Context, env *Env, m *manifest.Manifest) (*Sta
|
||||
"prepared": true,
|
||||
"stage": "prepare",
|
||||
"inputs_count": len(inputs),
|
||||
"audio_files_resolved": len(resolvedAudio),
|
||||
"audio_files_resolved": countAudioInputs(inputs),
|
||||
},
|
||||
}, nil
|
||||
}
|
||||
@@ -171,7 +167,23 @@ func renderResolvedPipeline(cfg *config.PipelineConfig) ([]byte, error) {
|
||||
return yaml.Marshal(cfg)
|
||||
}
|
||||
|
||||
func resolveAudioFiles(sessionDir string, inputs config.SessionInputsConfig) ([]string, error) {
|
||||
func resolveAudioInputs(sessionDir string, inputs config.SessionInputsConfig) ([]string, bool, error) {
|
||||
hasLocal := strings.TrimSpace(inputs.AudioDir) != "" || len(inputs.AudioFiles) > 0
|
||||
if inputs.AudioS3 != nil {
|
||||
if hasLocal {
|
||||
return nil, false, fmt.Errorf("audio_dir/audio_files and audio_s3 are mutually exclusive")
|
||||
}
|
||||
return nil, true, nil
|
||||
}
|
||||
|
||||
local, err := resolveLocalAudioFiles(sessionDir, inputs)
|
||||
if err != nil {
|
||||
return nil, false, err
|
||||
}
|
||||
return local, false, nil
|
||||
}
|
||||
|
||||
func resolveLocalAudioFiles(sessionDir string, inputs config.SessionInputsConfig) ([]string, error) {
|
||||
if len(inputs.AudioFiles) > 0 {
|
||||
out := make([]string, 0, len(inputs.AudioFiles))
|
||||
for _, p := range inputs.AudioFiles {
|
||||
@@ -223,6 +235,142 @@ func resolveAudioFiles(sessionDir string, inputs config.SessionInputsConfig) ([]
|
||||
return out, nil
|
||||
}
|
||||
|
||||
func materializeLocalAudioInputs(env *Env, paths artifacts.SessionPaths, resolvedAudio []string, registerInput func(kind, path, checksum string)) error {
|
||||
copiedByDest := map[string]string{}
|
||||
for _, src := range resolvedAudio {
|
||||
base := filepath.Base(src)
|
||||
if prev, exists := copiedByDest[base]; exists && prev != src {
|
||||
return fmt.Errorf("duplicate audio basename %q from %q and %q", base, prev, src)
|
||||
}
|
||||
copiedByDest[base] = src
|
||||
|
||||
dst := filepath.Join(paths.AudioDir, base)
|
||||
checksum, err := copyFileIfChanged(env.ArtifactStore, src, dst)
|
||||
if err != nil {
|
||||
return fmt.Errorf("materialize audio %q: %w", base, err)
|
||||
}
|
||||
registerInput("audio", dst, checksum)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func materializeS3AudioInputs(ctx context.Context, env *Env, m *manifest.Manifest, sessionID string, inputs *[]manifest.InputRecord) error {
|
||||
if env.ObjectStore == nil {
|
||||
return fmt.Errorf("s3 audio input requires object store backend")
|
||||
}
|
||||
if env.Config == nil || env.Config.Pipeline == nil || env.Config.Session == nil || env.Config.Pipeline.Storage.S3 == nil || env.Config.Session.Inputs.AudioS3 == nil {
|
||||
return fmt.Errorf("s3 audio input requires pipeline.storage.s3 and session.inputs.audio_s3 configuration")
|
||||
}
|
||||
|
||||
campaign := strings.TrimSpace(env.Config.Session.Campaign)
|
||||
if campaign == "" {
|
||||
return fmt.Errorf("session campaign is required for s3 audio input")
|
||||
}
|
||||
runID := strings.TrimSpace(m.RunID)
|
||||
if runID == "" {
|
||||
return fmt.Errorf("run id is required for s3 audio input")
|
||||
}
|
||||
|
||||
sessionPrefix := artifacts.S3SessionPrefix(env.Config.Pipeline.Storage.S3.RootPrefix, campaign, sessionID)
|
||||
audioPrefix := artifacts.S3AudioPrefix(sessionPrefix, env.Config.Session.Inputs.AudioS3.Prefix)
|
||||
objects, err := env.ObjectStore.List(ctx, audioPrefix)
|
||||
if err != nil {
|
||||
return fmt.Errorf("list s3 audio objects under %q: %w", audioPrefix, err)
|
||||
}
|
||||
|
||||
audioObjects := make([]storage.ObjectInfo, 0, len(objects))
|
||||
for _, obj := range objects {
|
||||
key := strings.TrimSpace(obj.Key)
|
||||
if key == "" || strings.HasSuffix(key, "/") {
|
||||
continue
|
||||
}
|
||||
if !isFlac(key) {
|
||||
continue
|
||||
}
|
||||
audioObjects = append(audioObjects, obj)
|
||||
}
|
||||
sort.Slice(audioObjects, func(i, j int) bool {
|
||||
return audioObjects[i].Key < audioObjects[j].Key
|
||||
})
|
||||
if len(audioObjects) == 0 {
|
||||
return fmt.Errorf("no .flac files found under s3 audio prefix %q", audioPrefix)
|
||||
}
|
||||
|
||||
spoolAudioDir := strings.TrimSpace(m.LocalSpoolDir)
|
||||
if spoolAudioDir == "" {
|
||||
spoolAudioDir = artifacts.SessionSpoolAudioDir(env.Config.Pipeline.Spool.Root, campaign, sessionID, runID)
|
||||
}
|
||||
workAudioDir := filepath.Join(pathsWorkDirForManifest(env, m, sessionID), "audio")
|
||||
|
||||
if err := os.MkdirAll(spoolAudioDir, 0o755); err != nil {
|
||||
return fmt.Errorf("create spool audio directory %q: %w", spoolAudioDir, err)
|
||||
}
|
||||
if err := os.MkdirAll(workAudioDir, 0o755); err != nil {
|
||||
return fmt.Errorf("create work audio directory %q: %w", workAudioDir, err)
|
||||
}
|
||||
|
||||
seenBase := map[string]string{}
|
||||
for _, obj := range audioObjects {
|
||||
base := path.Base(obj.Key)
|
||||
if prev, exists := seenBase[base]; exists && prev != obj.Key {
|
||||
return fmt.Errorf("duplicate s3 audio basename %q from %q and %q", base, prev, obj.Key)
|
||||
}
|
||||
seenBase[base] = obj.Key
|
||||
|
||||
spoolPath := filepath.Join(spoolAudioDir, base)
|
||||
if err := env.ObjectStore.Download(ctx, obj.Key, spoolPath); err != nil {
|
||||
return fmt.Errorf("download s3 audio object %q: %w", obj.Key, err)
|
||||
}
|
||||
|
||||
workPath := filepath.Join(workAudioDir, base)
|
||||
checksum, err := copyFileIfChanged(env.ArtifactStore, spoolPath, workPath)
|
||||
if err != nil {
|
||||
return fmt.Errorf("materialize downloaded audio %q: %w", base, err)
|
||||
}
|
||||
|
||||
*inputs = append(*inputs, manifest.InputRecord{
|
||||
Kind: "audio",
|
||||
Path: workPath,
|
||||
Checksum: checksum,
|
||||
Source: "s3",
|
||||
S3Bucket: strings.TrimSpace(env.Config.Pipeline.Storage.S3.Bucket),
|
||||
S3Key: obj.Key,
|
||||
S3Size: obj.Size,
|
||||
S3ETag: obj.ETag,
|
||||
SpoolPath: spoolPath,
|
||||
})
|
||||
}
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
func countAudioInputs(inputs []manifest.InputRecord) int {
|
||||
count := 0
|
||||
for _, in := range inputs {
|
||||
if in.Kind == "audio" {
|
||||
count++
|
||||
}
|
||||
}
|
||||
return count
|
||||
}
|
||||
|
||||
func pathsWorkDirForManifest(env *Env, m *manifest.Manifest, sessionID string) string {
|
||||
if env == nil || env.Config == nil || env.Config.Pipeline == nil || env.Config.Session == nil {
|
||||
return ""
|
||||
}
|
||||
if m != nil && strings.TrimSpace(m.LocalWorkDir) != "" {
|
||||
return strings.TrimSpace(m.LocalWorkDir)
|
||||
}
|
||||
runID := ""
|
||||
if m != nil {
|
||||
runID = strings.TrimSpace(m.RunID)
|
||||
}
|
||||
if runID != "" {
|
||||
return artifacts.SessionRunWorkDir(env.Config.Pipeline.Workspace.Root, env.Config.Session.Campaign, sessionID, runID)
|
||||
}
|
||||
return artifacts.SessionWorkDir(env.Config.Pipeline.Workspace.Root, sessionID)
|
||||
}
|
||||
|
||||
func resolvePath(baseDir, p string) (string, error) {
|
||||
trimmed := strings.TrimSpace(p)
|
||||
if trimmed == "" {
|
||||
|
||||
@@ -8,6 +8,7 @@ import (
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"gitea.maximumdirect.net/eric/narratio/internal/adapters/storage"
|
||||
"gitea.maximumdirect.net/eric/narratio/internal/artifacts"
|
||||
"gitea.maximumdirect.net/eric/narratio/internal/config"
|
||||
"gitea.maximumdirect.net/eric/narratio/internal/manifest"
|
||||
@@ -150,6 +151,139 @@ func TestPrepareStageIdempotent(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
func TestPrepareStageS3AudioDownloadAndMaterialization(t *testing.T) {
|
||||
env, m := setupPrepareEnv(t)
|
||||
env.Config.Session.Campaign = "forsaken"
|
||||
env.Config.Session.Inputs.AudioDir = ""
|
||||
env.Config.Session.Inputs.AudioFiles = nil
|
||||
env.Config.Session.Inputs.AudioS3 = &config.SessionAudioS3Input{Prefix: "audio/"}
|
||||
env.Config.Pipeline.Spool = config.SpoolConfig{Root: filepath.Join(t.TempDir(), "spool")}
|
||||
env.Config.Pipeline.Storage.S3 = &config.StorageS3Config{Bucket: "my-dnd-archive", RootPrefix: "dnd"}
|
||||
m.RunID = "20260515T031522Z-a1b2c3d4"
|
||||
m.LocalWorkDir = artifacts.SessionRunWorkDir(env.Config.Pipeline.Workspace.Root, "forsaken", m.SessionID, m.RunID)
|
||||
m.LocalSpoolDir = artifacts.SessionSpoolAudioDir(env.Config.Pipeline.Spool.Root, "forsaken", m.SessionID, m.RunID)
|
||||
|
||||
fake := &storage.FakeBackend{}
|
||||
fake.SeedObject(storage.FakeObject{Key: "dnd/campaigns/forsaken/sessions/2026-05-03/audio/alice.flac", Data: []byte("alice")})
|
||||
fake.SeedObject(storage.FakeObject{Key: "dnd/campaigns/forsaken/sessions/2026-05-03/audio/bob.FLAC", Data: []byte("bob")})
|
||||
fake.SeedObject(storage.FakeObject{Key: "dnd/campaigns/forsaken/sessions/2026-05-03/audio/ignore.txt", Data: []byte("x")})
|
||||
env.ObjectStore = fake
|
||||
|
||||
result, err := (prepareStage{}).Run(context.Background(), env, m)
|
||||
if err != nil {
|
||||
t.Fatalf("prepare.Run() error = %v", err)
|
||||
}
|
||||
if result == nil || result.Metadata["audio_files_resolved"] != 2 {
|
||||
t.Fatalf("result metadata = %#v, want audio_files_resolved=2", result)
|
||||
}
|
||||
|
||||
aliceWork := filepath.Join(m.LocalWorkDir, "audio", "alice.flac")
|
||||
bobWork := filepath.Join(m.LocalWorkDir, "audio", "bob.FLAC")
|
||||
aliceSpool := filepath.Join(m.LocalSpoolDir, "alice.flac")
|
||||
bobSpool := filepath.Join(m.LocalSpoolDir, "bob.FLAC")
|
||||
for _, p := range []string{aliceWork, bobWork, aliceSpool, bobSpool} {
|
||||
if _, err := os.Stat(p); err != nil {
|
||||
t.Fatalf("expected file %q: %v", p, err)
|
||||
}
|
||||
}
|
||||
|
||||
audioInputs := 0
|
||||
for _, in := range m.Inputs {
|
||||
if in.Kind != "audio" {
|
||||
continue
|
||||
}
|
||||
audioInputs++
|
||||
if in.Source != "s3" {
|
||||
t.Fatalf("audio input source = %q, want s3", in.Source)
|
||||
}
|
||||
if in.S3Bucket != "my-dnd-archive" {
|
||||
t.Fatalf("audio input bucket = %q", in.S3Bucket)
|
||||
}
|
||||
if in.S3Key == "" || in.SpoolPath == "" || in.Checksum == "" {
|
||||
t.Fatalf("audio input missing provenance: %#v", in)
|
||||
}
|
||||
}
|
||||
if audioInputs != 2 {
|
||||
t.Fatalf("audio input count = %d, want 2", audioInputs)
|
||||
}
|
||||
}
|
||||
|
||||
func TestPrepareStageS3AudioFailures(t *testing.T) {
|
||||
tests := []struct {
|
||||
name string
|
||||
setup func(env *Env, m *manifest.Manifest, fake *storage.FakeBackend)
|
||||
wantError string
|
||||
}{
|
||||
{
|
||||
name: "no flac files",
|
||||
setup: func(_ *Env, _ *manifest.Manifest, fake *storage.FakeBackend) {
|
||||
fake.SeedObject(storage.FakeObject{Key: "dnd/campaigns/forsaken/sessions/2026-05-03/audio/readme.txt", Data: []byte("x")})
|
||||
},
|
||||
wantError: "no .flac files found",
|
||||
},
|
||||
{
|
||||
name: "list error",
|
||||
setup: func(_ *Env, _ *manifest.Manifest, fake *storage.FakeBackend) {
|
||||
fake.ListErr = os.ErrPermission
|
||||
},
|
||||
wantError: "list s3 audio objects",
|
||||
},
|
||||
{
|
||||
name: "download error",
|
||||
setup: func(_ *Env, _ *manifest.Manifest, fake *storage.FakeBackend) {
|
||||
fake.SeedObject(storage.FakeObject{Key: "dnd/campaigns/forsaken/sessions/2026-05-03/audio/alice.flac", Data: []byte("alice")})
|
||||
fake.DownloadErr = os.ErrPermission
|
||||
},
|
||||
wantError: "download s3 audio object",
|
||||
},
|
||||
{
|
||||
name: "basename collision",
|
||||
setup: func(_ *Env, _ *manifest.Manifest, fake *storage.FakeBackend) {
|
||||
fake.SeedObject(storage.FakeObject{Key: "dnd/campaigns/forsaken/sessions/2026-05-03/audio/a/alice.flac", Data: []byte("a")})
|
||||
fake.SeedObject(storage.FakeObject{Key: "dnd/campaigns/forsaken/sessions/2026-05-03/audio/b/alice.flac", Data: []byte("b")})
|
||||
},
|
||||
wantError: "duplicate s3 audio basename",
|
||||
},
|
||||
}
|
||||
|
||||
for _, tt := range tests {
|
||||
t.Run(tt.name, func(t *testing.T) {
|
||||
env, m := setupPrepareEnv(t)
|
||||
env.Config.Session.Campaign = "forsaken"
|
||||
env.Config.Session.Inputs.AudioDir = ""
|
||||
env.Config.Session.Inputs.AudioFiles = nil
|
||||
env.Config.Session.Inputs.AudioS3 = &config.SessionAudioS3Input{Prefix: "audio/"}
|
||||
env.Config.Pipeline.Spool = config.SpoolConfig{Root: filepath.Join(t.TempDir(), "spool")}
|
||||
env.Config.Pipeline.Storage.S3 = &config.StorageS3Config{Bucket: "my-dnd-archive", RootPrefix: "dnd"}
|
||||
m.RunID = "20260515T031522Z-a1b2c3d4"
|
||||
m.LocalWorkDir = artifacts.SessionRunWorkDir(env.Config.Pipeline.Workspace.Root, "forsaken", m.SessionID, m.RunID)
|
||||
m.LocalSpoolDir = artifacts.SessionSpoolAudioDir(env.Config.Pipeline.Spool.Root, "forsaken", m.SessionID, m.RunID)
|
||||
|
||||
fake := &storage.FakeBackend{}
|
||||
tt.setup(env, m, fake)
|
||||
env.ObjectStore = fake
|
||||
|
||||
_, err := (prepareStage{}).Run(context.Background(), env, m)
|
||||
if err == nil || !strings.Contains(err.Error(), tt.wantError) {
|
||||
t.Fatalf("error = %v, want %q", err, tt.wantError)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestPrepareStageAudioSourceConflictFails(t *testing.T) {
|
||||
env, m := setupPrepareEnv(t)
|
||||
root := filepath.Dir(env.Config.SessionPath)
|
||||
writeFile(t, filepath.Join(root, "audio", "a.flac"), "a")
|
||||
env.Config.Session.Inputs.AudioDir = "./audio"
|
||||
env.Config.Session.Inputs.AudioS3 = &config.SessionAudioS3Input{Prefix: "audio/"}
|
||||
|
||||
_, err := (prepareStage{}).Run(context.Background(), env, m)
|
||||
if err == nil || !strings.Contains(err.Error(), "mutually exclusive") {
|
||||
t.Fatalf("error = %v, want mutually exclusive error", err)
|
||||
}
|
||||
}
|
||||
|
||||
func setupPrepareEnv(t *testing.T) (*Env, *manifest.Manifest) {
|
||||
t.Helper()
|
||||
workspace := t.TempDir()
|
||||
@@ -170,6 +304,7 @@ func setupPrepareEnv(t *testing.T) (*Env, *manifest.Manifest) {
|
||||
PipelinePath: pipelinePath,
|
||||
Session: &config.SessionConfig{
|
||||
SessionID: "2026-05-03",
|
||||
Campaign: "sample-campaign",
|
||||
Inputs: config.SessionInputsConfig{
|
||||
AudioDir: "./audio",
|
||||
SpeakersFile: "./speakers.yml",
|
||||
|
||||
@@ -29,6 +29,7 @@ type Env struct {
|
||||
Scriptorium scriptorium.Runner
|
||||
Analyzer analyzer.Runner
|
||||
Storage storage.Backend
|
||||
ObjectStore storage.ObjectStore
|
||||
Notifier notify.Sender
|
||||
}
|
||||
|
||||
|
||||
31
report-gemma4.md
Normal file
31
report-gemma4.md
Normal file
@@ -0,0 +1,31 @@
|
||||
# Narratio UX Evaluation Report
|
||||
|
||||
## 1. Executive Summary
|
||||
Narratio has a functional core pipeline with robust S3 integration for input and output, but it currently falls short of the intended "minimalist" operator UX. The primary gaps are the lack of session configuration discovery, the absence of session template support (and the `--session-id` flag), and the missing local cleanup logic. While the pipeline runs successfully, the operator must currently provide explicit session file paths for every run.
|
||||
|
||||
## 2. Feature Matrix
|
||||
|
||||
| Feature | Status | Evidence | Tests | Documentation | Notes |
|
||||
| :--- | :--- | :--- | :--- | :--- | :--- |
|
||||
| **Pipeline Config Discovery** | Implemented | `internal/app/pipeline_config_path_test.go` | Yes | Accurate | Checks `/usr/local/etc` and `/etc`. |
|
||||
| **Session Config Discovery** | Missing | `internal/app/run.go:30` | N/A | Stale | `--session` is mandatory. |
|
||||
| **Session Templates** | Missing | `internal/config/load.go` | N/A | Missing | No variable interpolation in `session.yml`. |
|
||||
| **`--session-id` CLI Flag** | Missing | `cmd/narratio` | N/A | Missing | Not implemented in CLI. |
|
||||
| **Minimal Seriatim Config** | Implemented | `internal/config/load.go` | Yes | Accurate | Defaults for timeout/schema provided. |
|
||||
| **Minimal Audita Config** | Implemented | `internal/config/load.go` | Yes | Accurate | Defaults for base_url/model provided. |
|
||||
| **S3 Audio Input** | Implemented | `internal/stage/prepare.go` | Yes | Accurate | Supports `.flac` downloads from S3. |
|
||||
| **S3 Archive & Promotion** | Implemented | `internal/stage/archive.go` | Yes | Accurate | Correct paths and commit markers. |
|
||||
| **Local Cleanup** | Missing | `architecture.md:136` | No | Stale | Config exists, logic is not implemented. |
|
||||
|
||||
## 3. Current Happy Path
|
||||
The shortest command that works today is:
|
||||
`narratio run --session <path_to_session.yml>`
|
||||
*(Assuming `pipeline.yml` is present in `/etc/narratio/` or `/usr/local/etc/narratio/`)*.
|
||||
|
||||
## 4. Gaps to Intended UX
|
||||
1. **Session Discovery & Templates (High):** The requirement to pass `--session` and the inability to use `--session-id` with a template is the largest friction point for operators.
|
||||
2. **Local Cleanup (Medium):** Spool and work directories are not cleaned up after successful archival, leading to local disk growth.
|
||||
3. **Local Pipeline Config (Low):** Narratio does not check `./pipeline.yml`, requiring users to use `--config` or move files to system directories.
|
||||
|
||||
## 5. Recommended Next Implementation Prompt
|
||||
"Implement session configuration discovery and template support. Specifically: 1) Add a search order for `session.yml` (e.g., `./session.yml`, `/etc/narratio/session.yml`) if `--session` is omitted. 2) Implement the `--session-id` CLI flag. 3) Add variable interpolation to `session.yml` so that `{{session_id}}` can be replaced by the value from the flag or the discovered session config before YAML decoding."
|
||||
54
report.md
Normal file
54
report.md
Normal file
@@ -0,0 +1,54 @@
|
||||
## 1. Executive Summary
|
||||
Narratio is close on S3 input/archive mechanics but not yet close on the intended minimal operator UX.
|
||||
Core S3 workflow is implemented (prepare S3 audio download, archive run upload, promotions, current pointers), but key UX items are missing: no `--session-id` flag, no session auto-discovery, and no session template variable injection. Cleanup/retention for spool/workdirs after archive is also still future work.
|
||||
|
||||
## 2. Feature Matrix
|
||||
|
||||
| Feature | Status | Evidence | Tests | Documentation status | Notes |
|
||||
|---|---|---|---|---|---|
|
||||
| Pipeline config auto-discovery when `--config` omitted | Implemented | `internal/app/pipeline_config_path.go`, `internal/config/defaults.go` | `internal/app/pipeline_config_path_test.go`, `internal/app/commands_test.go` | Accurate in `README.md`, `architecture.md` | Order: `/usr/local/etc/narratio/pipeline.yml`, then `/etc/narratio/pipeline.yml`; no `./pipeline.yml` default |
|
||||
| Session config auto-discovery when `--session` omitted | Missing | `--session` required in `internal/app/run.go`, `plan.go`, `resume.go`, `run_stage.go` | Covered by missing-flag tests in `internal/app/commands_test.go` | Accurate (docs do not claim auto-discovery) | No precedence order exists for session file search |
|
||||
| Session template variables in `session.yml` | Missing | Strict decode path in `internal/config/load.go` + strict YAML behavior | No template tests found | Not documented as implemented | No render-before-decode templating mechanism found |
|
||||
| `--session-id` CLI injection | Missing | No `--session-id` flag in command parsers (`run/plan/resume/run-stage`) | No tests for `--session-id` | Not documented as implemented | Intended minimal UX command not currently supported |
|
||||
| Campaign/run-aware work+spool paths | Implemented | `internal/artifacts/paths.go`, usage in prepare/archive | Path/helper tests in `internal/artifacts` + stage tests | Documented in README/architecture/roadmap | Layout includes `{campaign}/{session_id}/{run_id}` |
|
||||
| Run ID generation format | Implemented | `internal/artifacts/run_id.go` | Run ID tests in `internal/artifacts` | Documented | UTC timestamp + random suffix format present |
|
||||
| Storage backend abstraction | Implemented | `internal/adapters/storage/object_store.go` | Storage backend tests in `internal/adapters/storage` | Documented in README/architecture | Narrow interface (`List/Download/Upload/Exists`) |
|
||||
| S3 backend + fake backend | Implemented | `internal/adapters/storage/s3_backend.go`, `fake.go` | Adapter tests pass without live S3 | Documented | No AWS creds in config schema/examples |
|
||||
| Prepare S3 audio input (`inputs.audio_s3`) | Implemented | `internal/stage/prepare.go` | `internal/stage/prepare_test.go` | Documented in `docs/s3-audio-input.md`, README, architecture | Lists prefix, filters `.flac`, downloads/materializes, fails on none |
|
||||
| Local audio workflow | Implemented | Prepare logic still supports `audio_dir`/`audio_files` | Prepare tests cover local behavior and conflict with `audio_s3` | Documented | Local+S3 conflict is enforced |
|
||||
| Manifest provenance for S3 audio | Implemented | S3 source metadata assignment in prepare stage | Covered by S3 prepare tests | Documented | ETag recorded as metadata, not checksum |
|
||||
| Archive run upload under `runs/{run_id}` | Implemented | `internal/stage/archive.go` | `internal/stage/archive_test.go` | Documented in `docs/archive-storage.md`, README, architecture | Successful/completed runs only |
|
||||
| Archive promotion rules | Implemented | Archive stage promotion handling | Archive tests cover required/optional/mapping behavior | Documented | Default promoted outputs: `transcripts/trimmed.json`, `artifacts/session_recap.md` |
|
||||
| `current/manifest.json` + `current/run_id.txt` last | Implemented | Archive stage upload order logic | Archive tests verify ordering and pointer content | Documented | `current/run_id.txt` is commit marker; written last |
|
||||
| Avoid upload of failed/incomplete runs | Implemented | Archive prerequisite checks | Archive tests cover prerequisite failure path | Documented | Failed runs stay local |
|
||||
| Spool/workdir cleanup after successful archive | Missing | `spool.delete_audio_after_archive` exists but no cleanup behavior in stages/app | No cleanup behavior tests found | Docs accurately call cleanup future work | Gap vs intended UX item 12 |
|
||||
| Minimal Seriatim config | Partial | Validation requires `seriatim.binary`; defaults fill timeout/schema/gap | Config load/validate tests | Docs mostly accurate | “Binary-only” works after defaults, but still validated post-defaults |
|
||||
| Minimal Audita config | Partial | Validation requires `audita.binary` and `audita.model`; defaults for timeout/base_url/etc in loader | Config tests in `internal/config` | Docs currently list `timeout`/`base_url` as required in README section | UX expectation “binary + llm_api_key_env only” does not hold because model is required |
|
||||
|
||||
## 3. Current Happy Path
|
||||
Shortest realistic command today is:
|
||||
|
||||
`narratio run --session /path/to/session.yml`
|
||||
|
||||
That works only if pipeline config is discoverable at `/usr/local/etc/narratio/pipeline.yml` or `/etc/narratio/pipeline.yml`.
|
||||
Otherwise minimum is:
|
||||
|
||||
`narratio run --config /path/to/pipeline.yml --session /path/to/session.yml`
|
||||
|
||||
`narratio run --session-id 2026-04-04` does not work today (flag not implemented).
|
||||
|
||||
## 4. Gaps to Intended UX
|
||||
1. Missing `--session-id` flow with session template injection (largest UX gap).
|
||||
2. No session config auto-discovery order when `--session` is omitted.
|
||||
3. No session template rendering engine / unresolved-variable handling.
|
||||
4. Cleanup policy not implemented (`spool.delete_audio_after_archive` is modeled only).
|
||||
5. Audita minimal config UX still stricter than intended (model required).
|
||||
6. Optional doc refinement: explicitly call out that `./pipeline.yml` is not in current default search order.
|
||||
|
||||
## 5. Recommended Next Implementation Prompt
|
||||
Implement session template and `--session-id` UX only:
|
||||
|
||||
> Add session discovery and template rendering support so `narratio run --session-id <id>` works with no `--session` in normal setups.
|
||||
> Requirements: define deterministic session discovery order; support rendering template variables in `session.yml` before strict YAML decode; inject CLI `--session-id` into template variables; fail clearly on unresolved variables; preserve strict field validation after render; keep existing `--session` explicit path behavior; add tests for discovery precedence, render success/failure, and CLI integration; update README/architecture/examples accordingly; do not change archive/prepare storage behavior.
|
||||
|
||||
Validation note: `go test ./...` passes for the inspected state.
|
||||
Reference in New Issue
Block a user