Compare commits
17 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
| 37daab7857 | |||
| 2356688cb9 | |||
| 1054b64d9f | |||
| 01fb02426c | |||
| 7dc79e052f | |||
| cb525c0f72 | |||
| 622677d038 | |||
| 550288e008 | |||
| e58e545686 | |||
| 6ff54c5a0f | |||
| 924b5d15c6 | |||
| b065663180 | |||
| 3ba564b00f | |||
| a3986cf0d6 | |||
| 539601bd16 | |||
| 6ca1c8d6b0 | |||
| 4b7b50981b |
5
.gitignore
vendored
5
.gitignore
vendored
@@ -2,6 +2,8 @@
|
||||
.codex
|
||||
AGENTS.md
|
||||
|
||||
.DS_Store
|
||||
|
||||
# ---> Go
|
||||
# If you prefer the allow list template instead of the deny list, see community template:
|
||||
# https://github.com/github/gitignore/blob/main/community/Golang/Go.AllowList.gitignore
|
||||
@@ -22,6 +24,9 @@ AGENTS.md
|
||||
# Dependency directories (remove the comment below to include it)
|
||||
# vendor/
|
||||
|
||||
# Go cache
|
||||
.gocache
|
||||
|
||||
# Go workspace file
|
||||
go.work
|
||||
go.work.sum
|
||||
|
||||
21
README.md
21
README.md
@@ -65,13 +65,15 @@ Optional secrets-from-files config:
|
||||
|
||||
YAML decoding is strict (`KnownFields(true)`), so unknown fields fail fast.
|
||||
|
||||
Maintainer note: application defaults are centralized in [`internal/config/defaults.go`](internal/config/defaults.go).
|
||||
|
||||
## Storage And Archive Foundations
|
||||
|
||||
Narratio now includes configuration and path-model foundations for archive support, plus implemented prepare-stage S3 audio input.
|
||||
|
||||
Implemented foundations:
|
||||
|
||||
- `pipeline.storage.s3` config shape (`bucket`, `root_prefix`, `region`, `endpoint`, `force_path_style`)
|
||||
- `pipeline.storage.s3` config shape (`bucket`, `root_prefix`, `region`, `endpoint`, `force_path_style`, `access_key_id_env`, `secret_access_key_env`)
|
||||
- `pipeline.spool` config shape (`root`, `delete_audio_after_archive`)
|
||||
- `pipeline.archive` config shape (`enabled`, `upload_run`, `promote_artifacts`)
|
||||
- promotion-rule validation (`from`/`to` required, relative-only paths, traversal rejected)
|
||||
@@ -83,6 +85,8 @@ Implemented foundations:
|
||||
Current defaults:
|
||||
|
||||
- `pipeline.storage.s3.root_prefix`: `dnd`
|
||||
- `pipeline.storage.s3.access_key_id_env`: `OBJECT_STORAGE_KEY_ID`
|
||||
- `pipeline.storage.s3.secret_access_key_env`: `OBJECT_STORAGE_KEY`
|
||||
- `pipeline.workspace.cleanup_after_archive`: `false`
|
||||
- `pipeline.spool.root`: `/var/spool/narratio`
|
||||
- `pipeline.spool.delete_audio_after_archive`: `false`
|
||||
@@ -110,7 +114,8 @@ Current boundaries:
|
||||
- `pipeline.workspace.cleanup_after_archive: true` removes only the run-scoped local workdir after successful archive commit
|
||||
- cleanup executes only after all selected stages for the command invocation succeed
|
||||
- cleanup does not run for failed, incomplete, skipped, or unarchived runs
|
||||
- local development `audio_dir`/`audio_files` source inputs are never deleted by spool cleanup
|
||||
- local development `audio_dir`/`audio_files` source inputs are never deleted by spool cleanup
|
||||
- S3 credentials are resolved from configured env-var names when both are present; if either is missing, Narratio falls back to the AWS SDK default credential chain
|
||||
|
||||
S3 input details and current boundaries are documented in [docs/s3-audio-input.md](docs/s3-audio-input.md).
|
||||
|
||||
@@ -364,10 +369,16 @@ For the initial implementation, only `session_recap` generation is supported.
|
||||
|
||||
Analyze-stage session recap behavior:
|
||||
|
||||
- available transcript input sources for configured artifacts: `processed_transcript`, `normalized_transcript`, `trimmed_transcript`
|
||||
- preferred transcript artifact source IDs:
|
||||
- `narratio.transcript.polished`
|
||||
- `narratio.transcript.full`
|
||||
- `narratio.transcript.trimmed`
|
||||
- backward-compatible aliases remain supported:
|
||||
- `processed_transcript`
|
||||
- `normalized_transcript`
|
||||
- `trimmed_transcript`
|
||||
- session recap should use gameplay-only transcript input (`source: trimmed_transcript`)
|
||||
- Narratio resolves `trimmed_transcript` from trim manifest output (`transcript_trimmed`) or fallback `transcripts/trimmed.json`
|
||||
- Narratio resolves `normalized_transcript` from normalize manifest output (`transcript_normalized`) or fallback `transcripts/normalized.json`
|
||||
- Narratio resolves transcript inputs from the artifact resolver (manifest producer outputs first, then canonical session paths)
|
||||
- missing trimmed transcript fails clearly and advises running trim stage first
|
||||
- `normalized_transcript` is the preferred full-transcript source for future table/meta-analysis artifacts
|
||||
- `processed_transcript` remains supported for advanced/debug use cases
|
||||
|
||||
@@ -114,6 +114,7 @@ CLI pipeline config path resolution:
|
||||
- when `--config` is omitted, Narratio searches defaults in order:
|
||||
- `/usr/local/etc/narratio/pipeline.yml`
|
||||
- `/etc/narratio/pipeline.yml`
|
||||
- default values are centralized in `internal/config/defaults.go`
|
||||
|
||||
CLI session config path resolution:
|
||||
|
||||
@@ -151,6 +152,8 @@ Storage and archive foundations:
|
||||
- `region`
|
||||
- `endpoint`
|
||||
- `force_path_style` (default `false`)
|
||||
- `access_key_id_env` (default `OBJECT_STORAGE_KEY_ID`)
|
||||
- `secret_access_key_env` (default `OBJECT_STORAGE_KEY`)
|
||||
- `pipeline.spool.root` defaults to `/var/spool/narratio`
|
||||
- `pipeline.workspace.cleanup_after_archive` defaults to `false`
|
||||
- `pipeline.spool.delete_audio_after_archive` defaults to `false`
|
||||
@@ -176,7 +179,9 @@ Session input foundations:
|
||||
Cross-config validation scope:
|
||||
|
||||
- `pipeline.storage.s3.bucket` is required only when an S3-dependent feature is explicitly configured (for current foundations, that includes `session.inputs.audio_s3`, and archive upload intent when using `storage.backend: s3`)
|
||||
- no AWS credentials are stored in Narratio config; credential resolution remains an external runtime concern
|
||||
- no AWS credential values are stored in Narratio config; only env-var names are configured
|
||||
- when both configured credential env vars resolve to non-empty values, the S3 backend uses them as static credentials
|
||||
- when either configured credential value is missing, the S3 backend falls back to the AWS SDK default credential chain
|
||||
|
||||
Remote object-store backend scope:
|
||||
|
||||
|
||||
130
docs/development/workspace-implementation-plan.md
Normal file
130
docs/development/workspace-implementation-plan.md
Normal file
@@ -0,0 +1,130 @@
|
||||
# Workspace Architecture Implementation Plan (Status)
|
||||
|
||||
This document tracks the implemented workspace architecture and remaining work for v1.0.
|
||||
|
||||
## Current Architecture (Implemented)
|
||||
|
||||
Narratio now uses a canonical campaign-aware local layout:
|
||||
|
||||
```text
|
||||
{workspace.root}/work/{campaign_id}/{session_id}/
|
||||
manifest.json
|
||||
current/
|
||||
manifest.json
|
||||
run_id.txt
|
||||
inputs/
|
||||
transcripts/
|
||||
artifacts/
|
||||
reports/
|
||||
logs/
|
||||
config/
|
||||
runs/
|
||||
{run_id}/
|
||||
manifest.json
|
||||
{stage}/
|
||||
outputs/
|
||||
logs/
|
||||
reports/
|
||||
config/
|
||||
scratch/
|
||||
```
|
||||
|
||||
Core behavior:
|
||||
|
||||
- Session manifest remains the skip/resume source of truth.
|
||||
- Each invocation creates a run manifest at `runs/{run_id}/manifest.json`.
|
||||
- Stage execution writes run-local artifacts and promotes durable outputs to canonical session paths.
|
||||
- Archive uploads run records under `runs/{run_id}/`, applies promotion rules, then publishes `current/manifest.json` and `current/run_id.txt`.
|
||||
- Analyze input resolution uses centralized artifact IDs with alias support.
|
||||
- Forced upstream reruns mark downstream succeeded stages `stale` so later runs do not skip stale outputs.
|
||||
|
||||
## Section 4 Sequence Status
|
||||
|
||||
### Step 1: Campaign-aware session path model
|
||||
|
||||
Status: complete.
|
||||
|
||||
Implemented:
|
||||
|
||||
- Campaign-aware session and run path helpers.
|
||||
- Campaign-aware artifact-store layout APIs.
|
||||
- Canonical session manifest pathing under `work/{campaign}/{session}`.
|
||||
|
||||
### Step 2: Session manifest + run manifest scaffolding
|
||||
|
||||
Status: complete.
|
||||
|
||||
Implemented:
|
||||
|
||||
- Invocation-scoped run manifest type and store methods.
|
||||
- Runner creates/saves run manifests per invocation.
|
||||
- Session manifest remains authoritative for idempotent stage skipping.
|
||||
|
||||
### Step 3: Run-local stage execution + promotion
|
||||
|
||||
Status: complete.
|
||||
|
||||
Implemented:
|
||||
|
||||
- Run-local stage directory layout under `runs/{run_id}/{stage}`.
|
||||
- Shared helpers for run-local output mapping and promotion to canonical durable paths.
|
||||
- Producer run provenance recorded on durable artifact outputs.
|
||||
|
||||
### Step 4: Archive alignment
|
||||
|
||||
Status: complete.
|
||||
|
||||
Implemented:
|
||||
|
||||
- Canonical run-root/session-root resolution.
|
||||
- Deterministic run-file collection and promotion source resolution.
|
||||
- Current-pointer publication ordering retained (`current/manifest.json` then `current/run_id.txt`).
|
||||
|
||||
### Step 5: Artifact registry/resolver (analyze first consumer)
|
||||
|
||||
Status: complete.
|
||||
|
||||
Implemented:
|
||||
|
||||
- Central artifact resolver with canonical IDs:
|
||||
- `narratio.transcript.merged`
|
||||
- `narratio.transcript.polished`
|
||||
- `narratio.transcript.full`
|
||||
- `narratio.transcript.trimmed`
|
||||
- `narratio.bounds.session`
|
||||
- `narratio.artifact.session_recap`
|
||||
- Backward-compatible aliases:
|
||||
- `processed_transcript`
|
||||
- `normalized_transcript`
|
||||
- `trimmed_transcript`
|
||||
- Analyze stage switched to resolver-based source resolution.
|
||||
|
||||
### Step 6: Minimal downstream invalidation for forced reruns
|
||||
|
||||
Status: complete.
|
||||
|
||||
Implemented:
|
||||
|
||||
- Deterministic downstream invalidation based on canonical stage order.
|
||||
- On forced successful rerun of stage `X`, downstream succeeded stages are marked `stale`.
|
||||
- Resume and non-forced runs naturally re-execute stale stages.
|
||||
|
||||
### Step 7: Legacy layout migration strategy
|
||||
|
||||
Status: intentionally skipped.
|
||||
|
||||
Decision:
|
||||
|
||||
- Automatic migration and legacy fallback compatibility are intentionally not implemented.
|
||||
- The codebase targets canonical-only local layout behavior.
|
||||
- Legacy local workspace state, if present, should be recreated or migrated manually outside Narratio.
|
||||
|
||||
## Remaining Work (v1.0)
|
||||
|
||||
No required workspace/run-history migration steps remain from Section 4.
|
||||
|
||||
Possible future enhancements (non-blocking):
|
||||
|
||||
- Full checksum/input-graph stale detection.
|
||||
- Optional retention-policy expansion for run-history cleanup.
|
||||
- Broader artifact-resolver adoption across additional stage consumers.
|
||||
744
docs/development/workspace.md
Normal file
744
docs/development/workspace.md
Normal file
@@ -0,0 +1,744 @@
|
||||
# Narratio Workspace, Run History, and Artifact Resolution Architecture
|
||||
|
||||
## 1. Purpose
|
||||
|
||||
This document defines the intended v1.0 architecture for Narratio's local workspace layout, run history model, durable session outputs, manifest responsibilities, and artifact resolution contract.
|
||||
|
||||
Narratio is an idempotent session orchestrator. The command:
|
||||
|
||||
```bash
|
||||
narratio run --session-id 2026-05-07
|
||||
```
|
||||
|
||||
means "bring the identified session to its desired completed state." It does **not** mean "always create an entirely new independent output tree and ignore prior session state."
|
||||
|
||||
This distinction drives the architecture:
|
||||
|
||||
* A **session** is the durable domain object and idempotency boundary.
|
||||
* A **run** is an execution attempt that may update the session's durable state.
|
||||
* Durable outputs live at the session level.
|
||||
* Run-specific outputs, logs, generated configs, scratch files, and diagnostics live under `runs/{run_id}/`.
|
||||
* Successful stage outputs are promoted from run-local locations into canonical session-level locations.
|
||||
* The session manifest records current durable state.
|
||||
* Run manifests record execution history and debugging/provenance details.
|
||||
|
||||
This model intentionally mirrors the S3 archive model: session-level current artifacts are distinct from run-record history.
|
||||
|
||||
## 2. Core Concepts
|
||||
|
||||
### 2.1 Session
|
||||
|
||||
A session is the stable unit of work identified by `campaign_id` and `session_id`.
|
||||
|
||||
Examples:
|
||||
|
||||
```text
|
||||
campaign_id = dilfs
|
||||
session_id = 2026-05-07
|
||||
```
|
||||
|
||||
The session directory represents the current durable local state for that session. Re-running Narratio for the same session should consult this state, skip already-completed stages by default, and produce no changes unless work is incomplete, stale, forced, or explicitly selected.
|
||||
|
||||
### 2.2 Run
|
||||
|
||||
A run is a particular execution attempt identified by a generated `run_id`, for example:
|
||||
|
||||
```text
|
||||
20260517T174748Z-abcd1234
|
||||
```
|
||||
|
||||
A run may execute all stages or only a sparse subset of stages. Sparse runs are expected and desirable when the user invokes `--force`, `run-stage`, or a stage-limited command.
|
||||
|
||||
Run directories are provenance/debug records. They should reflect what actually happened during that invocation, not a synthetic complete pipeline layout.
|
||||
|
||||
### 2.3 Durable Output
|
||||
|
||||
A durable output is a canonical session-level artifact intended for later stages, user consumption, archive promotion, or future idempotency decisions.
|
||||
|
||||
Examples:
|
||||
|
||||
```text
|
||||
transcripts/merged.json
|
||||
transcripts/processed.json
|
||||
transcripts/normalized.json
|
||||
transcripts/trimmed.json
|
||||
artifacts/session_recap.md
|
||||
```
|
||||
|
||||
Durable outputs live directly under the session directory, not under a particular run directory.
|
||||
|
||||
### 2.4 Run-Local Output
|
||||
|
||||
A run-local output is the file initially produced by a stage during a specific run. After validation, durable outputs are promoted from run-local paths to session-level canonical paths.
|
||||
|
||||
Run-local outputs, logs, generated configs, reports, and scratch files should remain under:
|
||||
|
||||
```text
|
||||
runs/{run_id}/{stage}/...
|
||||
```
|
||||
|
||||
## 3. Local Workspace Layout
|
||||
|
||||
The canonical local workspace layout is:
|
||||
|
||||
```text
|
||||
{workspace.root}/work/{campaign_id}/{session_id}/
|
||||
manifest.json
|
||||
current/
|
||||
manifest.json
|
||||
run_id.txt
|
||||
inputs/
|
||||
transcripts/
|
||||
artifacts/
|
||||
reports/
|
||||
logs/
|
||||
config/
|
||||
runs/
|
||||
{run_id}/
|
||||
manifest.json
|
||||
prepare/
|
||||
transcribe/
|
||||
merge/
|
||||
polish/
|
||||
normalize/
|
||||
trim/
|
||||
analyze/
|
||||
archive/
|
||||
notify/
|
||||
```
|
||||
|
||||
Not every directory must exist at all times. Directories should be created idempotently when needed.
|
||||
|
||||
### 3.1 Session Root
|
||||
|
||||
The session root is:
|
||||
|
||||
```text
|
||||
{workspace.root}/work/{campaign_id}/{session_id}/
|
||||
```
|
||||
|
||||
The session root is the stable local home for the session. It is the default base for resolving canonical artifact paths.
|
||||
|
||||
The only files that should live directly in the session root are core session-state files, primarily:
|
||||
|
||||
```text
|
||||
manifest.json
|
||||
```
|
||||
|
||||
Lock files may also be session-root scoped if the implementation uses file locks there, but transient locks should not be treated as durable artifacts.
|
||||
|
||||
### 3.2 Session-Level Canonical Directories
|
||||
|
||||
The following directories contain current durable session state:
|
||||
|
||||
```text
|
||||
inputs/
|
||||
transcripts/
|
||||
artifacts/
|
||||
reports/
|
||||
logs/
|
||||
config/
|
||||
current/
|
||||
```
|
||||
|
||||
Recommended meanings:
|
||||
|
||||
| Directory | Purpose |
|
||||
| -------------- | ----------------------------------------------------------------------------- |
|
||||
| `inputs/` | Materialized or copied input files used by the current durable session state. |
|
||||
| `transcripts/` | Canonical transcript tiers. |
|
||||
| `artifacts/` | User-facing and machine-readable generated artifacts. |
|
||||
| `reports/` | Canonical stage reports worth preserving at the session level. |
|
||||
| `logs/` | Optional session-level logs or promoted/latest logs. |
|
||||
| `config/` | Optional session-level generated config snapshots or promoted/latest configs. |
|
||||
| `current/` | Current published session pointers, mirroring the archive backend. |
|
||||
|
||||
Canonical durable outputs should use stable paths under these directories.
|
||||
|
||||
### 3.3 Run History Directory
|
||||
|
||||
Run history lives under:
|
||||
|
||||
```text
|
||||
{workspace.root}/work/{campaign_id}/{session_id}/runs/{run_id}/
|
||||
```
|
||||
|
||||
Each run directory records what happened during that invocation. A run may contain all stage directories or only a sparse subset.
|
||||
|
||||
Example full run:
|
||||
|
||||
```text
|
||||
runs/20260517T174748Z-abcd1234/
|
||||
manifest.json
|
||||
prepare/
|
||||
transcribe/
|
||||
merge/
|
||||
polish/
|
||||
normalize/
|
||||
trim/
|
||||
analyze/
|
||||
archive/
|
||||
notify/
|
||||
```
|
||||
|
||||
Example sparse forced analyze run:
|
||||
|
||||
```text
|
||||
runs/20260518T030000Z-efgh5678/
|
||||
manifest.json
|
||||
analyze/
|
||||
```
|
||||
|
||||
Example sparse polish-through-analyze rerun:
|
||||
|
||||
```text
|
||||
runs/20260518T041500Z-a1b2c3d4/
|
||||
manifest.json
|
||||
polish/
|
||||
normalize/
|
||||
trim/
|
||||
analyze/
|
||||
```
|
||||
|
||||
Run directories should not create stage folders for stages that were not selected, executed, skipped, or otherwise considered during that run unless there is a clear diagnostic reason to do so.
|
||||
|
||||
### 3.4 Stage Run-Local Directories
|
||||
|
||||
Each stage receives a run-local directory:
|
||||
|
||||
```text
|
||||
runs/{run_id}/{stage}/
|
||||
```
|
||||
|
||||
Within that stage directory, the stage may use subdirectories such as:
|
||||
|
||||
```text
|
||||
outputs/
|
||||
logs/
|
||||
reports/
|
||||
config/
|
||||
scratch/
|
||||
```
|
||||
|
||||
For example:
|
||||
|
||||
```text
|
||||
runs/{run_id}/polish/
|
||||
outputs/transcripts/processed.json
|
||||
reports/audita.polish.report.json
|
||||
logs/stdout.log
|
||||
logs/stderr.log
|
||||
config/audita.polish.generated.yml
|
||||
scratch/
|
||||
```
|
||||
|
||||
The exact internal layout of a stage directory may vary by stage, but it should be deterministic, documented, and generated through centralized path helpers rather than ad hoc path joins.
|
||||
|
||||
## 4. Promotion Model
|
||||
|
||||
Narratio uses stage-level promotion with immediate promotion after successful validation.
|
||||
|
||||
The stage lifecycle is:
|
||||
|
||||
1. Resolve required inputs from the current session state and/or run-local context.
|
||||
2. Create the run-local stage directory.
|
||||
3. Execute the stage, writing outputs under `runs/{run_id}/{stage}/...`.
|
||||
4. Validate run-local outputs.
|
||||
5. Promote durable outputs into session-level canonical paths.
|
||||
6. Update the session manifest.
|
||||
7. Update the run manifest.
|
||||
|
||||
Promotion means an atomic or effectively atomic copy/rename from a run-local path to a session-level canonical path.
|
||||
|
||||
Example:
|
||||
|
||||
```text
|
||||
runs/{run_id}/polish/outputs/transcripts/processed.json
|
||||
```
|
||||
|
||||
is promoted to:
|
||||
|
||||
```text
|
||||
transcripts/processed.json
|
||||
```
|
||||
|
||||
Promotion should be safe and deterministic:
|
||||
|
||||
* Validate before promotion.
|
||||
* Write promoted files atomically where possible.
|
||||
* Never leave partially written durable outputs.
|
||||
* Record the producing `run_id` in the session manifest.
|
||||
* Preserve run-local files for debugging unless retention policy deletes them.
|
||||
|
||||
## 5. Promotion Policy: Option A
|
||||
|
||||
Narratio uses immediate stage-level promotion.
|
||||
|
||||
If a selected stage succeeds, its durable outputs are promoted immediately, even if a later selected stage fails.
|
||||
|
||||
Example:
|
||||
|
||||
```bash
|
||||
narratio run --session-id 2026-05-07 --force --stages polish,normalize,trim,analyze
|
||||
```
|
||||
|
||||
If `polish` succeeds and `normalize` fails:
|
||||
|
||||
* `transcripts/processed.json` may be updated from the new run.
|
||||
* `normalize`, `trim`, and `analyze` should not be marked succeeded for the new input state.
|
||||
* Downstream outputs may now be stale relative to the newly promoted polished transcript.
|
||||
|
||||
This policy is simpler, transparent, and consistent with stage-level resumability. It does require explicit stale/invalidation handling.
|
||||
|
||||
## 6. Stale and Invalidation Semantics
|
||||
|
||||
Full checksum-based stale detection may be implemented later. Before that exists, Narratio should still use a simple deterministic invalidation rule for forced or explicit upstream reruns.
|
||||
|
||||
When a stage is successfully re-executed and promoted, downstream stages should be marked stale unless they are also re-executed successfully in the same command invocation.
|
||||
|
||||
Example stage order:
|
||||
|
||||
```text
|
||||
prepare -> transcribe -> merge -> polish -> normalize -> trim -> analyze -> archive -> notify
|
||||
```
|
||||
|
||||
If `polish` is forced and promoted, then the following downstream stages should be invalidated unless rerun successfully:
|
||||
|
||||
```text
|
||||
normalize
|
||||
trim
|
||||
analyze
|
||||
archive
|
||||
notify
|
||||
```
|
||||
|
||||
A stale stage is not equivalent to a failed stage. It means its current durable outputs may no longer correspond to current upstream inputs or configuration.
|
||||
|
||||
Minimum manifest state model:
|
||||
|
||||
```text
|
||||
pending
|
||||
running
|
||||
succeeded
|
||||
failed
|
||||
skipped
|
||||
stale
|
||||
```
|
||||
|
||||
If adding a new `stale` state is too invasive for v1.0, the implementation should at least record stale metadata or clear downstream success markers in a way that prevents accidental idempotent skips based on obsolete outputs.
|
||||
|
||||
## 7. Manifest Responsibilities
|
||||
|
||||
Narratio should distinguish between session manifests and run manifests.
|
||||
|
||||
The same underlying Go types may be reused where practical, but the concepts should remain separate.
|
||||
|
||||
### 7.1 Session Manifest
|
||||
|
||||
Path:
|
||||
|
||||
```text
|
||||
{workspace.root}/work/{campaign_id}/{session_id}/manifest.json
|
||||
```
|
||||
|
||||
The session manifest answers:
|
||||
|
||||
```text
|
||||
What is the current durable state of this session?
|
||||
```
|
||||
|
||||
It should record:
|
||||
|
||||
* campaign ID
|
||||
* session ID
|
||||
* current or latest run ID
|
||||
* current stage states
|
||||
* canonical durable output refs
|
||||
* artifact IDs and paths
|
||||
* producing run ID for each current stage output
|
||||
* relevant input/config checksums when available
|
||||
* stale/invalidated stage information
|
||||
* archive/current publication metadata
|
||||
|
||||
A session's durable state may be a composite of multiple runs.
|
||||
|
||||
For example:
|
||||
|
||||
```text
|
||||
transcripts/merged.json produced by run A
|
||||
transcripts/processed.json produced by run B
|
||||
transcripts/normalized.json produced by run B
|
||||
transcripts/trimmed.json produced by run B
|
||||
artifacts/session_recap.md produced by run C
|
||||
```
|
||||
|
||||
This is valid and expected.
|
||||
|
||||
### 7.2 Run Manifest
|
||||
|
||||
Path:
|
||||
|
||||
```text
|
||||
{workspace.root}/work/{campaign_id}/{session_id}/runs/{run_id}/manifest.json
|
||||
```
|
||||
|
||||
The run manifest answers:
|
||||
|
||||
```text
|
||||
What happened during this specific execution attempt?
|
||||
```
|
||||
|
||||
It should record:
|
||||
|
||||
* run ID
|
||||
* campaign ID
|
||||
* session ID
|
||||
* command mode and selected stages
|
||||
* force flags or stage selection flags
|
||||
* stages considered during this run
|
||||
* stages executed during this run
|
||||
* stages skipped during this run and reasons
|
||||
* run-local output paths
|
||||
* promoted output paths
|
||||
* logs
|
||||
* reports
|
||||
* generated configs
|
||||
* timings
|
||||
* errors
|
||||
* non-secret subprocess invocation metadata
|
||||
|
||||
Run manifests are primarily for debugging, auditability, and archive history.
|
||||
|
||||
## 8. Idempotency and Resume Behavior
|
||||
|
||||
The idempotency boundary is the session, not the run.
|
||||
|
||||
By default:
|
||||
|
||||
```bash
|
||||
narratio run --session-id 2026-05-07
|
||||
```
|
||||
|
||||
should consult the session manifest and skip stages that are already succeeded and not stale.
|
||||
|
||||
If all stages are already complete, the command should execute zero stages and report that the session is already complete.
|
||||
|
||||
Forced execution creates a new run record but updates session-level durable state only for stages that actually succeed and promote outputs.
|
||||
|
||||
Examples:
|
||||
|
||||
```bash
|
||||
narratio run --session-id 2026-05-07 --force
|
||||
```
|
||||
|
||||
Creates a new run and attempts to re-execute the selected/default stage set.
|
||||
|
||||
```bash
|
||||
narratio run-stage --session-id 2026-05-07 analyze --force
|
||||
```
|
||||
|
||||
Creates a sparse run that executes only `analyze`, then promotes updated analysis artifacts if successful.
|
||||
|
||||
```bash
|
||||
narratio resume --session-id 2026-05-07
|
||||
```
|
||||
|
||||
Uses the session manifest to determine what remains incomplete or stale. Resume does not need to resume the same `run_id` unless the implementation explicitly supports resuming an interrupted active run.
|
||||
|
||||
## 9. Artifact Resolution Contract
|
||||
|
||||
Narratio should provide a first-class artifact registry and resolver.
|
||||
|
||||
The resolver maps symbolic artifact source names to canonical session-level paths and manifest output kinds.
|
||||
|
||||
Stages and adapters should not hardcode path fragments when resolving cross-stage inputs. They should ask the artifact resolver for the current durable artifact by ID.
|
||||
|
||||
### 9.1 Canonical Artifact IDs
|
||||
|
||||
Preferred artifact IDs should be namespaced:
|
||||
|
||||
```text
|
||||
narratio.transcript.merged
|
||||
narratio.transcript.polished
|
||||
narratio.transcript.full
|
||||
narratio.transcript.trimmed
|
||||
narratio.bounds.session
|
||||
narratio.artifact.session_recap
|
||||
```
|
||||
|
||||
Recommended initial registry:
|
||||
|
||||
| Artifact ID | Canonical Path | Producer Stage | Output Kind | Meaning |
|
||||
| --------------------------------- | ------------------------------- | -------------- | ------------------------ | ------------------------------------- |
|
||||
| `narratio.transcript.merged` | `transcripts/merged.json` | `merge` | `transcript_merged` | Deterministic Seriatim merge. |
|
||||
| `narratio.transcript.polished` | `transcripts/processed.json` | `polish` | `transcript_processed` | Full Audita-polished transcript. |
|
||||
| `narratio.transcript.full` | `transcripts/normalized.json` | `normalize` | `transcript_normalized` | Preferred full normalized transcript. |
|
||||
| `narratio.transcript.trimmed` | `transcripts/trimmed.json` | `trim` | `transcript_trimmed` | Gameplay-only transcript. |
|
||||
| `narratio.bounds.session` | `artifacts/session_bounds.json` | `trim` | `session_bounds` | Trim bounds selected for the session. |
|
||||
| `narratio.artifact.session_recap` | `artifacts/session_recap.md` | `analyze` | `artifact_session_recap` | Generated session recap. |
|
||||
|
||||
### 9.2 Backward-Compatible Aliases
|
||||
|
||||
Existing source names should remain supported:
|
||||
|
||||
| Legacy Source | Preferred Artifact ID |
|
||||
| ----------------------- | ------------------------------ |
|
||||
| `processed_transcript` | `narratio.transcript.polished` |
|
||||
| `normalized_transcript` | `narratio.transcript.full` |
|
||||
| `trimmed_transcript` | `narratio.transcript.trimmed` |
|
||||
|
||||
These aliases may be supported silently for v1.0. Documentation should prefer namespaced IDs.
|
||||
|
||||
### 9.3 Resolver Behavior
|
||||
|
||||
Artifact resolution should follow this order:
|
||||
|
||||
1. Normalize aliases to canonical artifact IDs.
|
||||
2. Look for a current output reference in the session manifest.
|
||||
3. Fall back to the canonical session-level path.
|
||||
4. If the artifact is required, fail clearly if missing.
|
||||
5. If the artifact is optional and missing, omit it from the downstream invocation.
|
||||
6. Validate the artifact using the expected content validator.
|
||||
7. Return a resolved artifact record containing ID, path, producer stage, output kind, and provenance.
|
||||
|
||||
Example conceptual result:
|
||||
|
||||
```json
|
||||
{
|
||||
"id": "narratio.transcript.trimmed",
|
||||
"path": "/var/lib/narratio/work/dilfs/2026-05-07/transcripts/trimmed.json",
|
||||
"producer_stage": "trim",
|
||||
"producer_run_id": "20260517T174748Z-abcd1234",
|
||||
"output_kind": "transcript_trimmed",
|
||||
"content_type": "application/json"
|
||||
}
|
||||
```
|
||||
|
||||
### 9.4 Artifact Validation
|
||||
|
||||
Transcript artifacts must be valid JSON with a top-level `segments` array.
|
||||
|
||||
Markdown/text artifacts must exist and be non-empty when required.
|
||||
|
||||
Bounds artifacts must match the expected bounds schema and refer to segment IDs in the same transcript ID space used by the trim stage.
|
||||
|
||||
Validation should happen before a resolved artifact is passed to another stage or external subprocess.
|
||||
|
||||
## 10. Analyze Stage Implications
|
||||
|
||||
The analyze stage should consume artifacts through the artifact resolver.
|
||||
|
||||
Preferred Scriptorium config shape:
|
||||
|
||||
```yaml
|
||||
scriptorium:
|
||||
artifacts:
|
||||
session_recap:
|
||||
enabled: true
|
||||
prompt_id: "dnd.session_recap"
|
||||
output_path: "artifacts/session_recap.md"
|
||||
inputs:
|
||||
transcript:
|
||||
source: "narratio.transcript.trimmed"
|
||||
required: true
|
||||
```
|
||||
|
||||
Additional artifacts can choose different transcript tiers:
|
||||
|
||||
```yaml
|
||||
scriptorium:
|
||||
artifacts:
|
||||
table_summary:
|
||||
enabled: true
|
||||
prompt_id: "dnd.table_summary"
|
||||
output_path: "artifacts/table_summary.md"
|
||||
inputs:
|
||||
transcript:
|
||||
source: "narratio.transcript.full"
|
||||
required: true
|
||||
```
|
||||
|
||||
For v1.0, Narratio does not need a generic DAG engine. It may execute configured analyze artifacts in deterministic order and allow later artifacts to consume earlier artifacts only when that relationship is explicit and unambiguous.
|
||||
|
||||
Rules:
|
||||
|
||||
* Artifact inputs resolve from current session-level durable state.
|
||||
* Outputs are first written run-locally.
|
||||
* Successful analyze outputs are promoted to session-level `artifacts/` paths.
|
||||
* Manifest output refs record the producing run ID.
|
||||
* Optional inputs are omitted when unavailable.
|
||||
* Required missing inputs fail before invoking Scriptorium.
|
||||
|
||||
## 11. Archive Alignment
|
||||
|
||||
Local workspace semantics should mirror archive semantics.
|
||||
|
||||
Local session-level durable paths:
|
||||
|
||||
```text
|
||||
work/{campaign}/{session}/transcripts/trimmed.json
|
||||
work/{campaign}/{session}/artifacts/session_recap.md
|
||||
work/{campaign}/{session}/current/manifest.json
|
||||
work/{campaign}/{session}/current/run_id.txt
|
||||
work/{campaign}/{session}/runs/{run_id}/...
|
||||
```
|
||||
|
||||
should map naturally to remote archive paths:
|
||||
|
||||
```text
|
||||
{root_prefix}/campaigns/{campaign}/sessions/{session}/transcripts/trimmed.json
|
||||
{root_prefix}/campaigns/{campaign}/sessions/{session}/artifacts/session_recap.md
|
||||
{root_prefix}/campaigns/{campaign}/sessions/{session}/current/manifest.json
|
||||
{root_prefix}/campaigns/{campaign}/sessions/{session}/current/run_id.txt
|
||||
{root_prefix}/campaigns/{campaign}/sessions/{session}/runs/{run_id}/...
|
||||
```
|
||||
|
||||
The archive stage should publish run records and promoted current artifacts consistently with the local model.
|
||||
|
||||
`current/run_id.txt` remains the effective commit marker for the archived current session state.
|
||||
|
||||
## 12. Path Helper Requirements
|
||||
|
||||
All code should use centralized path helpers for workspace paths.
|
||||
|
||||
Stage code should not manually assemble durable cross-stage paths using raw string joins except through the path model.
|
||||
|
||||
Recommended helper surface:
|
||||
|
||||
```text
|
||||
SessionRoot(campaignID, sessionID)
|
||||
SessionManifestPath(campaignID, sessionID)
|
||||
SessionCurrentDir(campaignID, sessionID)
|
||||
SessionTranscriptsDir(campaignID, sessionID)
|
||||
SessionArtifactsDir(campaignID, sessionID)
|
||||
SessionReportsDir(campaignID, sessionID)
|
||||
SessionLogsDir(campaignID, sessionID)
|
||||
SessionConfigDir(campaignID, sessionID)
|
||||
RunsDir(campaignID, sessionID)
|
||||
RunRoot(campaignID, sessionID, runID)
|
||||
RunManifestPath(campaignID, sessionID, runID)
|
||||
RunStageDir(campaignID, sessionID, runID, stage)
|
||||
RunStageOutputsDir(campaignID, sessionID, runID, stage)
|
||||
RunStageLogsDir(campaignID, sessionID, runID, stage)
|
||||
RunStageReportsDir(campaignID, sessionID, runID, stage)
|
||||
RunStageConfigDir(campaignID, sessionID, runID, stage)
|
||||
CanonicalArtifactPath(campaignID, sessionID, artifactID)
|
||||
```
|
||||
|
||||
Path helpers should enforce safe relative paths for configured output paths:
|
||||
|
||||
* reject absolute paths unless explicitly allowed for a particular config field
|
||||
* reject `..` traversal
|
||||
* normalize separators
|
||||
* preserve deterministic output paths
|
||||
|
||||
## 13. Directory Creation Policy
|
||||
|
||||
Directory creation should be centralized and idempotent.
|
||||
|
||||
Recommended policy:
|
||||
|
||||
* `prepare` ensures the baseline session directory structure exists.
|
||||
* Every stage also calls shared layout helpers to ensure its required run-local directories exist before writing.
|
||||
* `run-stage` should not depend on a prior `prepare` invocation merely to create folders.
|
||||
* Missing directories should be created with appropriate permissions.
|
||||
* Directory creation should not imply stage success.
|
||||
|
||||
This provides consistent layout while keeping direct stage execution robust.
|
||||
|
||||
## 14. Cleanup and Retention
|
||||
|
||||
Cleanup must preserve the distinction between durable session state and run history.
|
||||
|
||||
Workspace cleanup after successful archive may remove selected local directories only according to explicit configuration.
|
||||
|
||||
Potential retention policies:
|
||||
|
||||
```text
|
||||
keep_all_runs
|
||||
keep_failed_runs
|
||||
keep_last_n_runs
|
||||
delete_run_after_success
|
||||
```
|
||||
|
||||
For v1.0, conservative retention is preferred:
|
||||
|
||||
* Do not delete durable session-level outputs unless explicitly requested.
|
||||
* Do not delete failed run directories by default.
|
||||
* If cleanup is enabled, remove only documented run-scoped or spool-scoped paths.
|
||||
* Local development audio inputs must never be deleted by workspace cleanup.
|
||||
|
||||
## 15. Canonical-Only Layout Policy
|
||||
|
||||
Narratio now supports only the canonical campaign-aware layout:
|
||||
|
||||
```text
|
||||
{workspace.root}/work/{campaign_id}/{session_id}/manifest.json
|
||||
{workspace.root}/work/{campaign_id}/{session_id}/runs/{run_id}/...
|
||||
```
|
||||
|
||||
Legacy session-only layout compatibility is intentionally not implemented.
|
||||
|
||||
If legacy workspace data exists, operators should recreate or manually migrate that data outside Narratio before running v1.0 commands.
|
||||
|
||||
## 16. Documentation Updates Required
|
||||
|
||||
The following documentation should be updated to reflect this architecture:
|
||||
|
||||
* `README.md`
|
||||
* `docs/architecture.md`
|
||||
* a dedicated workspace/run-history document, such as this file
|
||||
* S3/archive documentation
|
||||
* analyze/artifact configuration documentation
|
||||
* example pipeline files
|
||||
|
||||
Documentation should consistently use the following terms:
|
||||
|
||||
| Term | Meaning |
|
||||
| ---------------- | ---------------------------------------------------------------------- |
|
||||
| Session | Durable domain object and idempotency boundary. |
|
||||
| Run | Execution attempt that may update session state. |
|
||||
| Durable output | Canonical current session-level output. |
|
||||
| Run-local output | Output produced inside a specific run directory before promotion. |
|
||||
| Promotion | Validated copy/rename from run-local output to durable session output. |
|
||||
| Session manifest | Current durable state of the session. |
|
||||
| Run manifest | Execution record for a particular run. |
|
||||
| Artifact ID | Symbolic source name resolved by the artifact registry. |
|
||||
|
||||
## 17. Architectural Invariants
|
||||
|
||||
The following invariants should hold after implementation:
|
||||
|
||||
1. `session_id` remains the idempotency boundary for normal operator commands.
|
||||
2. `run_id` identifies an execution attempt, not the primary durable workspace.
|
||||
3. Session-level canonical artifacts are the default inputs for downstream stages.
|
||||
4. Run-local outputs are promoted only after validation.
|
||||
5. A session's current durable state may be composed of outputs from multiple runs.
|
||||
6. Sparse run directories are valid and expected.
|
||||
7. The session manifest records current stage/artifact state and producer run IDs.
|
||||
8. The run manifest records what happened during one invocation.
|
||||
9. Artifact consumers resolve symbolic artifact IDs through a registry/resolver.
|
||||
10. Local workspace semantics mirror S3 archive semantics.
|
||||
11. Directory creation is centralized and idempotent.
|
||||
12. Stage code uses path helpers rather than ad hoc path construction.
|
||||
13. Forced upstream reruns invalidate downstream stage success unless downstream stages are rerun successfully.
|
||||
14. Cleanup never removes durable session outputs or local development inputs unless explicitly configured to do so.
|
||||
|
||||
## 18. Implementation Guidance
|
||||
|
||||
A practical implementation sequence is:
|
||||
|
||||
1. Add this architecture document.
|
||||
2. Add or revise path model helpers for session roots, run roots, stage directories, and canonical artifact paths.
|
||||
3. Introduce session manifest versus run manifest concepts.
|
||||
4. Route stage outputs through run-local directories.
|
||||
5. Add promotion helpers with validation and atomic writes.
|
||||
6. Update existing stages to promote durable outputs to session-level canonical paths.
|
||||
7. Add artifact registry and resolver.
|
||||
8. Update analyze to use artifact IDs and aliases.
|
||||
9. Add simple downstream stale invalidation for forced upstream reruns.
|
||||
10. Align archive/local path behavior and documentation.
|
||||
11. Update examples and README.
|
||||
12. Add tests for idempotency, sparse forced runs, promotion, manifest provenance, and artifact resolution.
|
||||
|
||||
This sequence intentionally avoids introducing a generic DAG engine. The v1.0 goal is a clear, deterministic, stage-oriented orchestrator with stable session-level outputs and inspectable run history.
|
||||
@@ -283,9 +283,9 @@ Session recap:
|
||||
```bash
|
||||
scriptorium run \
|
||||
--prompt dnd.session_recap \
|
||||
--input transcript=/work/session-42/transcript.polished.md \
|
||||
--input glossary=/work/session-42/glossary.yml \
|
||||
--out /work/session-42/artifacts/session_recap.md
|
||||
--input transcript=/work/campaign-7/session-42/transcript.polished.md \
|
||||
--input glossary=/work/campaign-7/session-42/glossary.yml \
|
||||
--out /work/campaign-7/session-42/artifacts/session_recap.md
|
||||
```
|
||||
|
||||
Structured events:
|
||||
@@ -293,8 +293,8 @@ Structured events:
|
||||
```bash
|
||||
scriptorium run \
|
||||
--prompt dnd.structured_events \
|
||||
--input transcript=/work/session-42/transcript.polished.md \
|
||||
--out /work/session-42/artifacts/structured_events.json
|
||||
--input transcript=/work/campaign-7/session-42/transcript.polished.md \
|
||||
--out /work/campaign-7/session-42/artifacts/structured_events.json
|
||||
```
|
||||
|
||||
Glossary suggestions:
|
||||
@@ -302,9 +302,9 @@ Glossary suggestions:
|
||||
```bash
|
||||
scriptorium run \
|
||||
--prompt dnd.glossary_suggestions \
|
||||
--input transcript=/work/session-42/transcript.polished.md \
|
||||
--input previous_recap=/work/session-41/artifacts/session_recap.md \
|
||||
--out /work/session-42/artifacts/glossary_suggestions.md
|
||||
--input transcript=/work/campaign-7/session-42/transcript.polished.md \
|
||||
--input previous_recap=/work/campaign-7/session-41/artifacts/session_recap.md \
|
||||
--out /work/campaign-7/session-42/artifacts/glossary_suggestions.md
|
||||
```
|
||||
|
||||
Player-facing summary:
|
||||
@@ -312,9 +312,9 @@ Player-facing summary:
|
||||
```bash
|
||||
scriptorium run \
|
||||
--prompt dnd.player_summary \
|
||||
--input transcript=/work/session-42/transcript.polished.md \
|
||||
--input structured_events=/work/session-42/artifacts/structured_events.json \
|
||||
--out /work/session-42/artifacts/player_summary.md
|
||||
--input transcript=/work/campaign-7/session-42/transcript.polished.md \
|
||||
--input structured_events=/work/campaign-7/session-42/artifacts/structured_events.json \
|
||||
--out /work/campaign-7/session-42/artifacts/player_summary.md
|
||||
```
|
||||
|
||||
## 21. Non-Goals
|
||||
|
||||
File diff suppressed because it is too large
Load Diff
@@ -1,35 +0,0 @@
|
||||
# S3 Archive Foundations Runbook
|
||||
|
||||
This runbook documents the currently implemented storage/archive foundations and the boundaries of current behavior.
|
||||
|
||||
## Implemented Now
|
||||
|
||||
- config modeling for:
|
||||
- `pipeline.storage.s3`
|
||||
- `pipeline.spool`
|
||||
- `pipeline.archive`
|
||||
- `session.inputs.audio_s3`
|
||||
- promotion rule validation for safe relative paths
|
||||
- run ID generation and path/key helper functions
|
||||
- manifest run/path identity fields
|
||||
- remote storage backend layer:
|
||||
- object-store interface (`List`, `Download`, `Upload`, `Exists`)
|
||||
- fake backend for deterministic tests
|
||||
- S3-compatible backend using AWS SDK v2
|
||||
- config-based backend construction helper
|
||||
|
||||
## Not Implemented Yet
|
||||
|
||||
- prepare-stage S3 object listing or download
|
||||
- archive-stage S3 upload or promotion writes
|
||||
- writing `current/manifest.json` or `current/run_id.txt` in S3
|
||||
|
||||
## Operational Notes
|
||||
|
||||
- local audio workflows remain the active development path (`audio_dir` or `audio_files`)
|
||||
- `audio_s3` and local audio config are mutually exclusive
|
||||
- do not place AWS credentials in Narratio config files
|
||||
|
||||
## Next Implementation Target
|
||||
|
||||
Use the storage backend layer in prepare-stage session audio discovery/download flow, while preserving local audio input support.
|
||||
@@ -20,6 +20,8 @@ Not implemented:
|
||||
|
||||
- `storage.s3.bucket` must be set when S3 audio input is used.
|
||||
- `storage.s3.root_prefix` defaults to `dnd`.
|
||||
- `storage.s3.access_key_id_env` defaults to `OBJECT_STORAGE_KEY_ID`.
|
||||
- `storage.s3.secret_access_key_env` defaults to `OBJECT_STORAGE_KEY`.
|
||||
- `spool.root` defaults to `/var/spool/narratio`.
|
||||
|
||||
`session.yml`:
|
||||
@@ -54,7 +56,7 @@ When `inputs.audio_s3.prefix` is configured, `prepare`:
|
||||
4. downloads selected objects to spool audio:
|
||||
- `{spool.root}/{campaign}/{session_id}/{run_id}/audio/`
|
||||
5. materializes audio into workdir audio:
|
||||
- `{workspace.root}/work/{campaign}/{session_id}/{run_id}/audio/`
|
||||
- `{workspace.root}/work/{campaign}/{session_id}/runs/{run_id}/audio/`
|
||||
6. records input provenance in the manifest (bucket, key, metadata, local paths, checksum)
|
||||
|
||||
Notes:
|
||||
@@ -1,58 +0,0 @@
|
||||
# Storage Backends
|
||||
|
||||
This document describes the currently implemented remote object storage backend layer used by Narratio, and its intended role in later prepare/archive work.
|
||||
|
||||
## Implemented
|
||||
|
||||
Remote object store abstraction:
|
||||
|
||||
- `List(ctx, prefix)`
|
||||
- `Download(ctx, key, localPath)`
|
||||
- `Upload(ctx, localPath, key, opts)`
|
||||
- `Exists(ctx, key)`
|
||||
|
||||
Object metadata model includes:
|
||||
|
||||
- key
|
||||
- size
|
||||
- ETag (provider metadata only)
|
||||
- last modified time when available
|
||||
|
||||
Backends:
|
||||
|
||||
- fake storage backend for deterministic tests
|
||||
- S3-compatible backend implemented with AWS SDK for Go v2
|
||||
|
||||
Construction:
|
||||
|
||||
- config-based constructor builds S3 backend from `pipeline.storage.s3` values:
|
||||
- bucket
|
||||
- region
|
||||
- endpoint
|
||||
- force_path_style
|
||||
|
||||
## Key Invariant
|
||||
|
||||
- callers pass full bucket-relative object keys
|
||||
- storage backends do not prepend `root_prefix`
|
||||
- storage backends do not infer campaign/session/run paths
|
||||
|
||||
S3 session/run key builders remain separate and continue to live outside backend implementations.
|
||||
|
||||
## Security Boundary
|
||||
|
||||
- do not store AWS credentials in Narratio config
|
||||
- AWS credentials are resolved through standard AWS SDK credential chains
|
||||
- AWS SDK-specific types remain isolated to the storage adapter package
|
||||
|
||||
## Testing
|
||||
|
||||
- fake storage tests cover list/download/upload/exists and error paths
|
||||
- S3 backend tests use injected fake S3 API clients
|
||||
- tests do not require live S3 services, AWS credentials, or network access
|
||||
|
||||
## Not Implemented Yet
|
||||
|
||||
- prepare-stage S3 object listing or downloads
|
||||
- archive-stage S3 uploads or promotion writes
|
||||
- writing `current/manifest.json` or `current/run_id.txt` to S3
|
||||
@@ -8,6 +8,9 @@ storage:
|
||||
bucket: "my-dnd-archive"
|
||||
root_prefix: "dnd"
|
||||
region: "us-east-1"
|
||||
# Optional credential env-var names (defaulted when omitted):
|
||||
# access_key_id_env: "OBJECT_STORAGE_KEY_ID"
|
||||
# secret_access_key_env: "OBJECT_STORAGE_KEY"
|
||||
|
||||
spool:
|
||||
root: "/var/spool/narratio"
|
||||
|
||||
@@ -11,6 +11,7 @@ import (
|
||||
"time"
|
||||
|
||||
awsconfig "github.com/aws/aws-sdk-go-v2/config"
|
||||
"github.com/aws/aws-sdk-go-v2/credentials"
|
||||
"github.com/aws/aws-sdk-go-v2/service/s3"
|
||||
"github.com/aws/aws-sdk-go-v2/service/s3/types"
|
||||
"github.com/aws/smithy-go"
|
||||
@@ -35,6 +36,8 @@ type s3ClientOptions struct {
|
||||
Region string
|
||||
Endpoint string
|
||||
ForcePathStyle bool
|
||||
AccessKeyID string
|
||||
SecretKey string
|
||||
}
|
||||
|
||||
var newS3Client = func(ctx context.Context, opts s3ClientOptions) (s3API, error) {
|
||||
@@ -42,6 +45,15 @@ var newS3Client = func(ctx context.Context, opts s3ClientOptions) (s3API, error)
|
||||
if strings.TrimSpace(opts.Region) != "" {
|
||||
loadOpts = append(loadOpts, awsconfig.WithRegion(strings.TrimSpace(opts.Region)))
|
||||
}
|
||||
if strings.TrimSpace(opts.AccessKeyID) != "" && strings.TrimSpace(opts.SecretKey) != "" {
|
||||
loadOpts = append(loadOpts, awsconfig.WithCredentialsProvider(
|
||||
credentials.NewStaticCredentialsProvider(
|
||||
strings.TrimSpace(opts.AccessKeyID),
|
||||
strings.TrimSpace(opts.SecretKey),
|
||||
"",
|
||||
),
|
||||
))
|
||||
}
|
||||
awsCfg, err := awsconfig.LoadDefaultConfig(ctx, loadOpts...)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("load aws config: %w", err)
|
||||
@@ -67,6 +79,8 @@ func NewS3BackendFromConfig(ctx context.Context, cfg config.StorageS3Config) (*S
|
||||
Region: cfg.Region,
|
||||
Endpoint: cfg.Endpoint,
|
||||
ForcePathStyle: cfg.ForcePathStyle,
|
||||
AccessKeyID: s3CredentialFromEnv(orDefaultEnvName(cfg.AccessKeyIDEnv, config.DefaultS3AccessKeyIDEnv)),
|
||||
SecretKey: s3CredentialFromEnv(orDefaultEnvName(cfg.SecretKeyEnv, config.DefaultS3SecretAccessKeyEnv)),
|
||||
})
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("build s3 client: %w", err)
|
||||
@@ -78,6 +92,26 @@ func NewS3BackendFromConfig(ctx context.Context, cfg config.StorageS3Config) (*S
|
||||
}, nil
|
||||
}
|
||||
|
||||
func s3CredentialFromEnv(envVarName string) string {
|
||||
name := strings.TrimSpace(envVarName)
|
||||
if name == "" {
|
||||
return ""
|
||||
}
|
||||
value, ok := os.LookupEnv(name)
|
||||
if !ok {
|
||||
return ""
|
||||
}
|
||||
return strings.TrimSpace(value)
|
||||
}
|
||||
|
||||
func orDefaultEnvName(name, fallback string) string {
|
||||
trimmed := strings.TrimSpace(name)
|
||||
if trimmed == "" {
|
||||
return fallback
|
||||
}
|
||||
return trimmed
|
||||
}
|
||||
|
||||
// List returns objects under prefix.
|
||||
func (b *S3Backend) List(ctx context.Context, prefix string) ([]ObjectInfo, error) {
|
||||
normalizedPrefix := normalizeObjectKey(prefix)
|
||||
|
||||
@@ -187,6 +187,8 @@ func TestS3BackendExistsNotFound(t *testing.T) {
|
||||
func TestNewS3BackendFromConfigUsesClientOptions(t *testing.T) {
|
||||
original := newS3Client
|
||||
t.Cleanup(func() { newS3Client = original })
|
||||
t.Setenv("OBJECT_STORAGE_KEY_ID", "id-123")
|
||||
t.Setenv("OBJECT_STORAGE_KEY", "secret-abc")
|
||||
|
||||
var got s3ClientOptions
|
||||
newS3Client = func(_ context.Context, opts s3ClientOptions) (s3API, error) {
|
||||
@@ -209,6 +211,9 @@ func TestNewS3BackendFromConfigUsesClientOptions(t *testing.T) {
|
||||
if got.Region != "us-east-1" || got.Endpoint != "http://localhost:9000" || !got.ForcePathStyle {
|
||||
t.Fatalf("client options = %#v, want region/endpoint/path-style values", got)
|
||||
}
|
||||
if got.AccessKeyID != "id-123" || got.SecretKey != "secret-abc" {
|
||||
t.Fatalf("client options credentials = %#v, want env-resolved static credentials", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestNewS3BackendFromConfigRequiresBucket(t *testing.T) {
|
||||
@@ -218,6 +223,30 @@ func TestNewS3BackendFromConfigRequiresBucket(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
func TestNewS3BackendFromConfigFallsBackWhenCredentialEnvMissing(t *testing.T) {
|
||||
original := newS3Client
|
||||
t.Cleanup(func() { newS3Client = original })
|
||||
|
||||
var got s3ClientOptions
|
||||
newS3Client = func(_ context.Context, opts s3ClientOptions) (s3API, error) {
|
||||
got = opts
|
||||
return &fakeS3API{}, nil
|
||||
}
|
||||
|
||||
_, err := NewS3BackendFromConfig(context.Background(), config.StorageS3Config{
|
||||
Bucket: "my-archive",
|
||||
Region: "us-east-1",
|
||||
AccessKeyIDEnv: "MISSING_ACCESS_KEY_ID",
|
||||
SecretKeyEnv: "MISSING_SECRET_KEY",
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatalf("NewS3BackendFromConfig() error = %v", err)
|
||||
}
|
||||
if got.AccessKeyID != "" || got.SecretKey != "" {
|
||||
t.Fatalf("client options credentials = %#v, want empty fallback values", got)
|
||||
}
|
||||
}
|
||||
|
||||
func strPtr(v string) *string { return &v }
|
||||
func int64Ptr(v int64) *int64 { return &v }
|
||||
|
||||
|
||||
@@ -111,7 +111,7 @@ func TestExecuteRunStageUnknownFails(t *testing.T) {
|
||||
func TestExecuteRunStageNormalizeIsAccepted(t *testing.T) {
|
||||
workspaceRoot := t.TempDir()
|
||||
pipelinePath, sessionPath := writeValidConfigFiles(t, workspaceRoot, "https://example.com/transcribe")
|
||||
workRoot := filepath.Join(workspaceRoot, "work", "2026-05-03")
|
||||
workRoot := filepath.Join(workspaceRoot, "work", "sample-campaign", "2026-05-03")
|
||||
mustWriteTestFile(t, filepath.Join(workRoot, "transcripts", "processed.json"), `{"segments":[{"id":1}]}`)
|
||||
|
||||
var stdout bytes.Buffer
|
||||
@@ -156,7 +156,7 @@ func TestExecuteRunStageTranscribeUsesConfiguredWhisperXServer(t *testing.T) {
|
||||
t.Fatal("expected whisperx server to be called at least once")
|
||||
}
|
||||
|
||||
outPath := filepath.Join(workspaceRoot, "work", "2026-05-03", "transcripts", "raw", "alice.json")
|
||||
outPath := filepath.Join(workspaceRoot, "work", "sample-campaign", "2026-05-03", "transcripts", "raw", "alice.json")
|
||||
data, err := os.ReadFile(outPath)
|
||||
if err != nil {
|
||||
t.Fatalf("ReadFile(%q): %v", outPath, err)
|
||||
@@ -233,7 +233,7 @@ inputs:
|
||||
_ = os.Chdir(originalWD)
|
||||
})
|
||||
|
||||
workRoot := filepath.Join(workspaceRoot, "work", sessionID)
|
||||
workRoot := filepath.Join(workspaceRoot, "work", "sample-campaign", sessionID)
|
||||
mustWriteTestFile(t, filepath.Join(workRoot, "transcripts", "merged.json"), `{"schema":"seriatim-intermediate","segments":[]}`)
|
||||
mustWriteTestFile(t, filepath.Join(workRoot, "inputs", "glossary.yml"), "[]\n")
|
||||
|
||||
|
||||
@@ -57,7 +57,7 @@ func Plan(ctx context.Context, args []string, out io.Writer) error {
|
||||
}
|
||||
|
||||
store := artifacts.NewLocalStore(cfg.Pipeline.Workspace.Root)
|
||||
paths, err := store.EnsureLayout(cfg.Session.SessionID)
|
||||
paths, err := store.EnsureLayoutFor(cfg.Session.Campaign, cfg.Session.SessionID)
|
||||
if err != nil {
|
||||
return fmt.Errorf("plan: prepare workdir: %w", err)
|
||||
}
|
||||
|
||||
@@ -36,7 +36,7 @@ func TestPlanCreatesAndReusesWorkdir(t *testing.T) {
|
||||
t.Fatalf("first output = %q, want totals", got)
|
||||
}
|
||||
|
||||
sessionWorkdir := artifacts.SessionWorkDir(workspaceRoot, "2026-05-03")
|
||||
sessionWorkdir := artifacts.SessionWorkDirForCampaign(workspaceRoot, "sample-campaign", "2026-05-03")
|
||||
expectedDirs := []string{
|
||||
sessionWorkdir,
|
||||
filepath.Join(sessionWorkdir, "inputs"),
|
||||
@@ -63,7 +63,7 @@ func TestPlanCreatesAndReusesWorkdir(t *testing.T) {
|
||||
func TestPlanShowsRunAndSkipFromManifest(t *testing.T) {
|
||||
workspaceRoot := t.TempDir()
|
||||
pipelinePath, sessionPath := writeValidConfigFiles(t, workspaceRoot)
|
||||
manifestPath := filepath.Join(workspaceRoot, "work", "2026-05-03", "manifest.json")
|
||||
manifestPath := filepath.Join(workspaceRoot, "work", "sample-campaign", "2026-05-03", "manifest.json")
|
||||
|
||||
store := &manifest.LocalStore{}
|
||||
m := manifest.New("2026-05-03", time.Date(2026, 5, 3, 10, 0, 0, 0, time.UTC))
|
||||
|
||||
@@ -54,7 +54,7 @@ func runPostArchiveCleanup(ctx context.Context, env *Env, manifestPath string, m
|
||||
}
|
||||
workDir := strings.TrimSpace(m.LocalWorkDir)
|
||||
if workDir == "" {
|
||||
workDir = artifacts.SessionRunWorkDir(
|
||||
workDir = artifacts.SessionRunRootForCampaign(
|
||||
env.Config.Pipeline.Workspace.Root,
|
||||
strings.TrimSpace(env.Config.Session.Campaign),
|
||||
strings.TrimSpace(env.Config.Session.SessionID),
|
||||
|
||||
@@ -210,7 +210,7 @@ func TestPostArchiveCleanupNotRunWhenPromotionIsMissing(t *testing.T) {
|
||||
assertExists(t, seed.spoolAudioDir)
|
||||
assertExists(t, seed.runWorkDir)
|
||||
assertExists(t, filepath.Join(seed.runWorkDir, "manifest.json"))
|
||||
assertExists(t, artifacts.SessionRunWorkDir(cfg.Pipeline.Workspace.Root, cfg.Session.Campaign, cfg.Session.SessionID, runID))
|
||||
assertExists(t, artifacts.SessionRunRootForCampaign(cfg.Pipeline.Workspace.Root, cfg.Session.Campaign, cfg.Session.SessionID, runID))
|
||||
}
|
||||
|
||||
func TestPostArchiveCleanupNotRunWhenCurrentManifestUploadFails(t *testing.T) {
|
||||
@@ -271,8 +271,8 @@ func cleanupFixtureConfig(t *testing.T) (*config.Config, cleanupSeed) {
|
||||
cfg.Pipeline.Spool.Root = filepath.Join(t.TempDir(), "spool")
|
||||
|
||||
runID := "20260516T010203Z-1a2b3c4d"
|
||||
runWorkDir := artifacts.SessionRunWorkDir(cfg.Pipeline.Workspace.Root, cfg.Session.Campaign, cfg.Session.SessionID, runID)
|
||||
otherRunDir := artifacts.SessionRunWorkDir(cfg.Pipeline.Workspace.Root, cfg.Session.Campaign, cfg.Session.SessionID, "20260516T010204Z-5e6f7a8b")
|
||||
runWorkDir := artifacts.SessionRunRootForCampaign(cfg.Pipeline.Workspace.Root, cfg.Session.Campaign, cfg.Session.SessionID, runID)
|
||||
otherRunDir := artifacts.SessionRunRootForCampaign(cfg.Pipeline.Workspace.Root, cfg.Session.Campaign, cfg.Session.SessionID, "20260516T010204Z-5e6f7a8b")
|
||||
spoolAudioDir := artifacts.SessionSpoolAudioDir(cfg.Pipeline.Spool.Root, cfg.Session.Campaign, cfg.Session.SessionID, runID)
|
||||
|
||||
mustWriteFile(t, filepath.Join(runWorkDir, "manifest.json"), "{}\n")
|
||||
@@ -326,7 +326,11 @@ func archiveStageCleanupFixture(t *testing.T) (*config.Config, cleanupSeed, stri
|
||||
{From: "artifacts/session_recap.md", To: "artifacts/session_recap.md", Required: boolPtr(true)},
|
||||
},
|
||||
}
|
||||
writeArchiveFixtureRunFiles(t, seed.runWorkDir)
|
||||
writeArchiveFixtureRunFiles(
|
||||
t,
|
||||
seed.runWorkDir,
|
||||
artifacts.SessionWorkDirForCampaign(cfg.Pipeline.Workspace.Root, cfg.Session.Campaign, cfg.Session.SessionID),
|
||||
)
|
||||
|
||||
store := &manifest.LocalStore{}
|
||||
seedManifest, err := store.Load(context.Background(), manifestPathFor(cfg))
|
||||
@@ -345,16 +349,19 @@ func archiveStageCleanupFixture(t *testing.T) (*config.Config, cleanupSeed, stri
|
||||
return cfg, seed, runID
|
||||
}
|
||||
|
||||
func writeArchiveFixtureRunFiles(t *testing.T, runWorkDir string) {
|
||||
func writeArchiveFixtureRunFiles(t *testing.T, runWorkDir, sessionRoot string) {
|
||||
t.Helper()
|
||||
mustWriteFile(t, filepath.Join(runWorkDir, "inputs", "session.yml"), "session_id: 2026-05-03\n")
|
||||
mustWriteFile(t, filepath.Join(runWorkDir, "transcripts", "raw", "speaker.json"), "{}\n")
|
||||
mustWriteFile(t, filepath.Join(runWorkDir, "transcripts", "trimmed.json"), "{}\n")
|
||||
mustWriteFile(t, filepath.Join(runWorkDir, "artifacts", "session_recap.md"), "# recap\n")
|
||||
mustWriteFile(t, filepath.Join(runWorkDir, "reports", "audita.report.json"), "{}\n")
|
||||
mustWriteFile(t, filepath.Join(runWorkDir, "config", "audita.generated.yml"), "key: value\n")
|
||||
mustWriteFile(t, filepath.Join(runWorkDir, "prepare", "inputs", "session.yml"), "session_id: 2026-05-03\n")
|
||||
mustWriteFile(t, filepath.Join(runWorkDir, "transcribe", "outputs", "transcripts", "raw", "speaker.json"), "{}\n")
|
||||
mustWriteFile(t, filepath.Join(runWorkDir, "trim", "outputs", "transcripts", "trimmed.json"), "{}\n")
|
||||
mustWriteFile(t, filepath.Join(runWorkDir, "analyze", "outputs", "artifacts", "session_recap.md"), "# recap\n")
|
||||
mustWriteFile(t, filepath.Join(runWorkDir, "polish", "reports", "audita.report.json"), "{}\n")
|
||||
mustWriteFile(t, filepath.Join(runWorkDir, "merge", "config", "seriatim.generated.yml"), "key: value\n")
|
||||
mustWriteFile(t, filepath.Join(runWorkDir, "logs", "audita.stderr.log"), "stderr\n")
|
||||
mustWriteFile(t, filepath.Join(runWorkDir, "manifest.json"), "{}\n")
|
||||
|
||||
mustWriteFile(t, filepath.Join(sessionRoot, "transcripts", "trimmed.json"), "{}\n")
|
||||
mustWriteFile(t, filepath.Join(sessionRoot, "artifacts", "session_recap.md"), "# recap\n")
|
||||
}
|
||||
|
||||
type failKeyStore struct {
|
||||
|
||||
@@ -6,6 +6,7 @@ import (
|
||||
"fmt"
|
||||
"io"
|
||||
|
||||
"gitea.maximumdirect.net/eric/narratio/internal/artifacts"
|
||||
"gitea.maximumdirect.net/eric/narratio/internal/config"
|
||||
"gitea.maximumdirect.net/eric/narratio/internal/manifest"
|
||||
)
|
||||
@@ -83,7 +84,11 @@ func Resume(ctx context.Context, args []string, out io.Writer) error {
|
||||
}
|
||||
|
||||
func loadManifestIfPresent(ctx context.Context, cfg *config.Config) (*manifest.Manifest, error) {
|
||||
path := manifestPathFor(cfg)
|
||||
path := artifacts.SessionManifestPathForCampaign(
|
||||
cfg.Pipeline.Workspace.Root,
|
||||
cfg.Session.Campaign,
|
||||
cfg.Session.SessionID,
|
||||
)
|
||||
exists, err := fileExists(path)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("check manifest %q: %w", path, err)
|
||||
|
||||
@@ -16,7 +16,7 @@ import (
|
||||
func TestResumeStartsAfterCompletedStages(t *testing.T) {
|
||||
workspaceRoot := t.TempDir()
|
||||
pipelinePath, sessionPath := writeValidConfigFiles(t, workspaceRoot)
|
||||
manifestPath := filepath.Join(workspaceRoot, "work", "2026-05-03", "manifest.json")
|
||||
manifestPath := filepath.Join(workspaceRoot, "work", "sample-campaign", "2026-05-03", "manifest.json")
|
||||
|
||||
store := &manifest.LocalStore{}
|
||||
m := manifest.New("2026-05-03", time.Date(2026, 5, 3, 10, 0, 0, 0, time.UTC))
|
||||
@@ -25,7 +25,7 @@ func TestResumeStartsAfterCompletedStages(t *testing.T) {
|
||||
if err := store.Save(context.Background(), manifestPath, m); err != nil {
|
||||
t.Fatalf("save manifest: %v", err)
|
||||
}
|
||||
workRoot := filepath.Join(workspaceRoot, "work", "2026-05-03")
|
||||
workRoot := filepath.Join(workspaceRoot, "work", "sample-campaign", "2026-05-03")
|
||||
mustWriteTestFile(t, filepath.Join(workRoot, "transcripts", "raw", "alice.json"), `{"segments":[]}`)
|
||||
mustWriteTestFile(t, filepath.Join(workRoot, "inputs", "speakers.yml"), "match:\n - speaker: Alice\n match: [\"alice\"]\n")
|
||||
mustWriteTestFile(t, filepath.Join(workRoot, "inputs", "autocorrect.yml"), "[]\n")
|
||||
@@ -52,7 +52,7 @@ func TestResumeStartsAfterCompletedStages(t *testing.T) {
|
||||
func TestResumeNoRemainingStages(t *testing.T) {
|
||||
workspaceRoot := t.TempDir()
|
||||
pipelinePath, sessionPath := writeValidConfigFiles(t, workspaceRoot)
|
||||
manifestPath := filepath.Join(workspaceRoot, "work", "2026-05-03", "manifest.json")
|
||||
manifestPath := filepath.Join(workspaceRoot, "work", "sample-campaign", "2026-05-03", "manifest.json")
|
||||
|
||||
store := &manifest.LocalStore{}
|
||||
m := manifest.New("2026-05-03", time.Date(2026, 5, 3, 10, 0, 0, 0, time.UTC))
|
||||
@@ -81,7 +81,7 @@ func TestResumeForceRerunsSucceeded(t *testing.T) {
|
||||
}))
|
||||
defer srv.Close()
|
||||
pipelinePath, sessionPath := writeValidConfigFiles(t, workspaceRoot, srv.URL)
|
||||
manifestPath := filepath.Join(workspaceRoot, "work", "2026-05-03", "manifest.json")
|
||||
manifestPath := filepath.Join(workspaceRoot, "work", "sample-campaign", "2026-05-03", "manifest.json")
|
||||
|
||||
store := &manifest.LocalStore{}
|
||||
m := manifest.New("2026-05-03", time.Date(2026, 5, 3, 10, 0, 0, 0, time.UTC))
|
||||
@@ -105,8 +105,8 @@ func TestResumeForceRerunsSucceeded(t *testing.T) {
|
||||
func TestRunStageExecutesOnlySelectedStage(t *testing.T) {
|
||||
workspaceRoot := t.TempDir()
|
||||
pipelinePath, sessionPath := writeValidConfigFiles(t, workspaceRoot)
|
||||
manifestPath := filepath.Join(workspaceRoot, "work", "2026-05-03", "manifest.json")
|
||||
workRoot := filepath.Join(workspaceRoot, "work", "2026-05-03")
|
||||
manifestPath := filepath.Join(workspaceRoot, "work", "sample-campaign", "2026-05-03", "manifest.json")
|
||||
workRoot := filepath.Join(workspaceRoot, "work", "sample-campaign", "2026-05-03")
|
||||
mustWriteTestFile(t, filepath.Join(workRoot, "transcripts", "merged.json"), `{"segments":[]}`)
|
||||
mustWriteTestFile(t, filepath.Join(workRoot, "inputs", "glossary.yml"), "terms: []\n")
|
||||
|
||||
@@ -135,8 +135,8 @@ func TestRunStageExecutesOnlySelectedStage(t *testing.T) {
|
||||
func TestRunStageSkipAndForce(t *testing.T) {
|
||||
workspaceRoot := t.TempDir()
|
||||
pipelinePath, sessionPath := writeValidConfigFiles(t, workspaceRoot)
|
||||
manifestPath := filepath.Join(workspaceRoot, "work", "2026-05-03", "manifest.json")
|
||||
workRoot := filepath.Join(workspaceRoot, "work", "2026-05-03")
|
||||
manifestPath := filepath.Join(workspaceRoot, "work", "sample-campaign", "2026-05-03", "manifest.json")
|
||||
workRoot := filepath.Join(workspaceRoot, "work", "sample-campaign", "2026-05-03")
|
||||
mustWriteTestFile(t, filepath.Join(workRoot, "transcripts", "merged.json"), `{"segments":[]}`)
|
||||
mustWriteTestFile(t, filepath.Join(workRoot, "inputs", "glossary.yml"), "terms: []\n")
|
||||
|
||||
@@ -166,11 +166,57 @@ func TestRunStageSkipAndForce(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
func TestRunStageForceMarksDownstreamStaleAndResumeContinuesFromStale(t *testing.T) {
|
||||
workspaceRoot := t.TempDir()
|
||||
pipelinePath, sessionPath := writeValidConfigFiles(t, workspaceRoot)
|
||||
manifestPath := filepath.Join(workspaceRoot, "work", "sample-campaign", "2026-05-03", "manifest.json")
|
||||
workRoot := filepath.Join(workspaceRoot, "work", "sample-campaign", "2026-05-03")
|
||||
mustWriteTestFile(t, filepath.Join(workRoot, "transcripts", "merged.json"), `{"segments":[]}`)
|
||||
mustWriteTestFile(t, filepath.Join(workRoot, "inputs", "glossary.yml"), "terms: []\n")
|
||||
|
||||
store := &manifest.LocalStore{}
|
||||
seed := manifest.New("2026-05-03", time.Date(2026, 5, 3, 10, 0, 0, 0, time.UTC))
|
||||
for _, name := range []string{"prepare", "transcribe", "merge", "polish", "normalize", "trim", "analyze", "archive", "notify"} {
|
||||
seed.MarkStageSucceeded(name, time.Date(2026, 5, 3, 10, 1, 0, 0, time.UTC), nil)
|
||||
}
|
||||
if err := store.Save(context.Background(), manifestPath, seed); err != nil {
|
||||
t.Fatalf("save manifest: %v", err)
|
||||
}
|
||||
|
||||
var out bytes.Buffer
|
||||
err := RunStage(context.Background(), []string{"--config", pipelinePath, "--session", sessionPath, "--force", "polish"}, &out)
|
||||
if err != nil {
|
||||
t.Fatalf("RunStage(force) error = %v", err)
|
||||
}
|
||||
if !strings.Contains(out.String(), "stage=polish executed=1 skipped=0 force=true") {
|
||||
t.Fatalf("output = %q, want forced polish rerun", out.String())
|
||||
}
|
||||
|
||||
afterForce, err := store.Load(context.Background(), manifestPath)
|
||||
if err != nil {
|
||||
t.Fatalf("load manifest after force: %v", err)
|
||||
}
|
||||
for _, name := range []string{"normalize", "trim", "analyze", "archive", "notify"} {
|
||||
if afterForce.Stages[name] == nil || afterForce.Stages[name].Status != manifest.StatusStale {
|
||||
t.Fatalf("stage %q = %#v, want stale", name, afterForce.Stages[name])
|
||||
}
|
||||
}
|
||||
|
||||
out.Reset()
|
||||
err = Resume(context.Background(), []string{"--config", pipelinePath, "--session", sessionPath}, &out)
|
||||
if err != nil {
|
||||
t.Fatalf("Resume() error = %v", err)
|
||||
}
|
||||
if !strings.Contains(out.String(), "executed=5 skipped=0") {
|
||||
t.Fatalf("output = %q, want resume to execute normalize..notify", out.String())
|
||||
}
|
||||
}
|
||||
|
||||
func TestRunStageTrimExecutes(t *testing.T) {
|
||||
workspaceRoot := t.TempDir()
|
||||
pipelinePath, sessionPath := writeValidConfigFiles(t, workspaceRoot)
|
||||
manifestPath := filepath.Join(workspaceRoot, "work", "2026-05-03", "manifest.json")
|
||||
workRoot := filepath.Join(workspaceRoot, "work", "2026-05-03")
|
||||
manifestPath := filepath.Join(workspaceRoot, "work", "sample-campaign", "2026-05-03", "manifest.json")
|
||||
workRoot := filepath.Join(workspaceRoot, "work", "sample-campaign", "2026-05-03")
|
||||
mustWriteTestFile(t, filepath.Join(workRoot, "transcripts", "normalized.json"), `{"segments":[{"id":1},{"id":2}]}`)
|
||||
|
||||
var out bytes.Buffer
|
||||
@@ -198,8 +244,8 @@ func TestRunStageTrimExecutes(t *testing.T) {
|
||||
func TestRunStageNormalizeExecutes(t *testing.T) {
|
||||
workspaceRoot := t.TempDir()
|
||||
pipelinePath, sessionPath := writeValidConfigFiles(t, workspaceRoot)
|
||||
manifestPath := filepath.Join(workspaceRoot, "work", "2026-05-03", "manifest.json")
|
||||
workRoot := filepath.Join(workspaceRoot, "work", "2026-05-03")
|
||||
manifestPath := filepath.Join(workspaceRoot, "work", "sample-campaign", "2026-05-03", "manifest.json")
|
||||
workRoot := filepath.Join(workspaceRoot, "work", "sample-campaign", "2026-05-03")
|
||||
mustWriteTestFile(t, filepath.Join(workRoot, "transcripts", "processed.json"), `{"segments":[{"id":1},{"id":2}]}`)
|
||||
|
||||
var out bytes.Buffer
|
||||
|
||||
@@ -1,6 +1,8 @@
|
||||
package app
|
||||
|
||||
import (
|
||||
"time"
|
||||
|
||||
"gitea.maximumdirect.net/eric/narratio/internal/manifest"
|
||||
"gitea.maximumdirect.net/eric/narratio/internal/stage"
|
||||
)
|
||||
@@ -49,3 +51,43 @@ func firstNonSucceededIndex(stages []stage.Stage, m *manifest.Manifest) int {
|
||||
}
|
||||
return len(stages)
|
||||
}
|
||||
|
||||
func canonicalStageNames() []string {
|
||||
all := stage.All()
|
||||
out := make([]string, 0, len(all))
|
||||
for _, s := range all {
|
||||
if s == nil {
|
||||
continue
|
||||
}
|
||||
out = append(out, s.Name())
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
func downstreamStageNames(stageName string) []string {
|
||||
names := canonicalStageNames()
|
||||
for i, name := range names {
|
||||
if name != stageName {
|
||||
continue
|
||||
}
|
||||
return append([]string(nil), names[i+1:]...)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func invalidateDownstreamSucceededStages(m *manifest.Manifest, upstreamStage string, at time.Time) []string {
|
||||
if m == nil || m.Stages == nil {
|
||||
return nil
|
||||
}
|
||||
|
||||
invalidated := make([]string, 0)
|
||||
for _, downstream := range downstreamStageNames(upstreamStage) {
|
||||
sr := m.Stages[downstream]
|
||||
if sr == nil || sr.Status != manifest.StatusSucceeded {
|
||||
continue
|
||||
}
|
||||
m.MarkStageStale(downstream, at, "upstream stage rerun with force")
|
||||
invalidated = append(invalidated, downstream)
|
||||
}
|
||||
return invalidated
|
||||
}
|
||||
|
||||
@@ -1,6 +1,7 @@
|
||||
package app
|
||||
|
||||
import (
|
||||
"reflect"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
@@ -40,3 +41,48 @@ func TestDecideStageActions(t *testing.T) {
|
||||
t.Fatalf("forced prepare action = %q, want %q", forced[0].Action, stageActionRun)
|
||||
}
|
||||
}
|
||||
|
||||
func TestDownstreamStageNames(t *testing.T) {
|
||||
got := downstreamStageNames("polish")
|
||||
want := []string{"normalize", "trim", "analyze", "archive", "notify"}
|
||||
if !reflect.DeepEqual(got, want) {
|
||||
t.Fatalf("downstreamStageNames(polish) = %#v, want %#v", got, want)
|
||||
}
|
||||
|
||||
missing := downstreamStageNames("unknown")
|
||||
if len(missing) != 0 {
|
||||
t.Fatalf("downstreamStageNames(unknown) = %#v, want empty", missing)
|
||||
}
|
||||
}
|
||||
|
||||
func TestInvalidateDownstreamSucceededStages(t *testing.T) {
|
||||
now := time.Now().UTC()
|
||||
m := manifest.New("2026-05-03", now)
|
||||
m.MarkStageSucceeded("prepare", now, nil)
|
||||
m.MarkStageSucceeded("transcribe", now, nil)
|
||||
m.MarkStageSucceeded("merge", now, nil)
|
||||
m.MarkStageSucceeded("polish", now, nil)
|
||||
m.MarkStageSucceeded("normalize", now, nil)
|
||||
m.MarkStageSucceeded("trim", now, nil)
|
||||
m.MarkStageFailed("analyze", now, "analysis failed")
|
||||
m.MarkStageSucceeded("archive", now, nil)
|
||||
m.MarkStageSucceeded("notify", now, nil)
|
||||
|
||||
got := invalidateDownstreamSucceededStages(m, "polish", now.Add(1*time.Second))
|
||||
want := []string{"normalize", "trim", "archive", "notify"}
|
||||
if !reflect.DeepEqual(got, want) {
|
||||
t.Fatalf("invalidateDownstreamSucceededStages() = %#v, want %#v", got, want)
|
||||
}
|
||||
|
||||
for _, stageName := range want {
|
||||
if m.Stages[stageName].Status != manifest.StatusStale {
|
||||
t.Fatalf("%s status = %q, want stale", stageName, m.Stages[stageName].Status)
|
||||
}
|
||||
}
|
||||
if m.Stages["analyze"].Status != manifest.StatusFailed {
|
||||
t.Fatalf("analyze status = %q, want failed", m.Stages["analyze"].Status)
|
||||
}
|
||||
if m.Stages["prepare"].Status != manifest.StatusSucceeded {
|
||||
t.Fatalf("prepare status = %q, want succeeded", m.Stages["prepare"].Status)
|
||||
}
|
||||
}
|
||||
|
||||
@@ -5,7 +5,6 @@ import (
|
||||
"fmt"
|
||||
"log/slog"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
|
||||
"gitea.maximumdirect.net/eric/narratio/internal/adapters/audita"
|
||||
@@ -27,11 +26,13 @@ type RunOptions struct {
|
||||
}
|
||||
|
||||
type RunSummary struct {
|
||||
SessionID string
|
||||
ManifestPath string
|
||||
StageNames []string
|
||||
Executed []string
|
||||
Skipped []string
|
||||
SessionID string
|
||||
RunID string
|
||||
ManifestPath string
|
||||
RunManifestPath string
|
||||
StageNames []string
|
||||
Executed []string
|
||||
Skipped []string
|
||||
}
|
||||
|
||||
func executeStages(ctx context.Context, cfg *config.Config, stages []stage.Stage, opts RunOptions) (*RunSummary, error) {
|
||||
@@ -93,12 +94,12 @@ func executeStages(ctx context.Context, cfg *config.Config, stages []stage.Stage
|
||||
}
|
||||
|
||||
artifactStore := env.ArtifactStore
|
||||
paths, err := artifactStore.EnsureLayout(cfg.Session.SessionID)
|
||||
paths, err := artifactStore.EnsureLayoutFor(cfg.Session.Campaign, cfg.Session.SessionID)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("prepare workdir: %w", err)
|
||||
}
|
||||
|
||||
lock, err := artifactStore.AcquireSessionLock(cfg.Session.SessionID)
|
||||
lock, err := artifactStore.AcquireSessionLockFor(cfg.Session.Campaign, cfg.Session.SessionID)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("acquire session lock: %w", err)
|
||||
}
|
||||
@@ -111,7 +112,11 @@ func executeStages(ctx context.Context, cfg *config.Config, stages []stage.Stage
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
identityChanged, err := ensureManifestIdentity(cfg, m)
|
||||
runID, err := artifacts.NewRunID()
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("generate run id: %w", err)
|
||||
}
|
||||
identityChanged, err := ensureManifestIdentity(cfg, m, runID)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("initialize manifest identity: %w", err)
|
||||
}
|
||||
@@ -120,6 +125,29 @@ func executeStages(ctx context.Context, cfg *config.Config, stages []stage.Stage
|
||||
return nil, fmt.Errorf("save manifest identity %q: %w", manifestPath, err)
|
||||
}
|
||||
}
|
||||
runManifestPath := artifacts.SessionRunManifestPathForCampaign(
|
||||
cfg.Pipeline.Workspace.Root,
|
||||
cfg.Session.Campaign,
|
||||
cfg.Session.SessionID,
|
||||
runID,
|
||||
)
|
||||
runManifestStore := &manifest.LocalStore{}
|
||||
runManifest, err := runManifestStore.CreateRun(
|
||||
ctx,
|
||||
cfg.Session.SessionID,
|
||||
cfg.Session.Campaign,
|
||||
runID,
|
||||
opts.Force,
|
||||
requestedStageNames(stages),
|
||||
)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("create run manifest: %w", err)
|
||||
}
|
||||
runManifest.SessionManifestPath = manifestPath
|
||||
syncRunManifestIdentityFromSession(m, runManifest)
|
||||
if err := runManifestStore.SaveRun(ctx, runManifestPath, runManifest); err != nil {
|
||||
return nil, fmt.Errorf("save initial run manifest %q: %w", runManifestPath, err)
|
||||
}
|
||||
|
||||
stageEnv := env
|
||||
|
||||
@@ -134,12 +162,23 @@ func executeStages(ctx context.Context, cfg *config.Config, stages []stage.Stage
|
||||
|
||||
if d.Action == stageActionSkip {
|
||||
skipped = append(skipped, s.Name())
|
||||
skipAt := nowUTC()
|
||||
runManifest.SetStageAction(s.Name(), manifest.RunStageActionSkip, skipAt)
|
||||
runManifest.MarkStageSkipped(s.Name(), skipAt, "already_succeeded")
|
||||
if err := runManifestStore.SaveRun(ctx, runManifestPath, runManifest); err != nil {
|
||||
return nil, fmt.Errorf("save run manifest after skip %q: %w", s.Name(), err)
|
||||
}
|
||||
env.Logger.Info("skipping stage", "stage", s.Name(), "reason", "already_succeeded", "force", opts.Force)
|
||||
continue
|
||||
}
|
||||
executed = append(executed, s.Name())
|
||||
|
||||
now := nowUTC()
|
||||
runManifest.SetStageAction(s.Name(), manifest.RunStageActionRun, now)
|
||||
runManifest.MarkStageRunning(s.Name(), now)
|
||||
if err := runManifestStore.SaveRun(ctx, runManifestPath, runManifest); err != nil {
|
||||
return nil, fmt.Errorf("save run manifest before stage %q: %w", s.Name(), err)
|
||||
}
|
||||
m.MarkStageRunning(s.Name(), now)
|
||||
env.Logger.Info("starting stage", "stage", s.Name())
|
||||
if err := env.ManifestStore.Save(ctx, manifestPath, m); err != nil {
|
||||
@@ -149,35 +188,65 @@ func executeStages(ctx context.Context, cfg *config.Config, stages []stage.Stage
|
||||
|
||||
result, err := s.Run(ctx, stageEnv, m)
|
||||
if err != nil {
|
||||
m.MarkStageFailed(s.Name(), nowUTC(), err.Error())
|
||||
failedAt := nowUTC()
|
||||
m.MarkStageFailed(s.Name(), failedAt, err.Error())
|
||||
if saveErr := env.ManifestStore.Save(ctx, manifestPath, m); saveErr != nil {
|
||||
return nil, fmt.Errorf("stage %q failed (%v) and manifest save failed (%v)", s.Name(), err, saveErr)
|
||||
}
|
||||
runManifest.MarkStageFailed(s.Name(), failedAt, err.Error())
|
||||
syncRunManifestIdentityFromSession(m, runManifest)
|
||||
if saveErr := runManifestStore.SaveRun(ctx, runManifestPath, runManifest); saveErr != nil {
|
||||
return nil, fmt.Errorf("stage %q failed (%v) and run-manifest save failed (%v)", s.Name(), err, saveErr)
|
||||
}
|
||||
env.Logger.Info("stage failed", "stage", s.Name(), "error", err)
|
||||
return nil, fmt.Errorf("stage %q failed: %w", s.Name(), err)
|
||||
}
|
||||
|
||||
outputs := mapResultOutputs(result)
|
||||
m.MarkStageSucceeded(s.Name(), nowUTC(), outputs)
|
||||
outputs := mapResultOutputs(result, runID)
|
||||
succeededAt := nowUTC()
|
||||
m.MarkStageSucceeded(s.Name(), succeededAt, outputs)
|
||||
applyStageResultToManifest(m, s.Name(), result)
|
||||
if opts.Force {
|
||||
invalidateDownstreamSucceededStages(m, s.Name(), succeededAt)
|
||||
}
|
||||
|
||||
if err := env.ManifestStore.Save(ctx, manifestPath, m); err != nil {
|
||||
return nil, fmt.Errorf("save manifest after stage %q: %w", s.Name(), err)
|
||||
}
|
||||
runManifest.MarkStageSucceeded(s.Name(), succeededAt, outputs)
|
||||
applyStageResultToRunManifest(runManifest, s.Name(), result)
|
||||
syncRunManifestIdentityFromSession(m, runManifest)
|
||||
if err := runManifestStore.SaveRun(ctx, runManifestPath, runManifest); err != nil {
|
||||
return nil, fmt.Errorf("save run manifest after stage %q: %w", s.Name(), err)
|
||||
}
|
||||
env.Logger.Debug("manifest saved", "stage", s.Name(), "transition", "succeeded", "path", manifestPath)
|
||||
env.Logger.Info("stage succeeded", "stage", s.Name())
|
||||
}
|
||||
|
||||
if err := runPostArchiveCleanup(ctx, env, manifestPath, m, executed); err != nil {
|
||||
failedAt := nowUTC()
|
||||
runManifest.MarkFailed(failedAt, err.Error())
|
||||
syncRunManifestIdentityFromSession(m, runManifest)
|
||||
if saveErr := runManifestStore.SaveRun(ctx, runManifestPath, runManifest); saveErr != nil {
|
||||
return nil, fmt.Errorf("post-archive cleanup failed (%v) and run-manifest save failed (%v)", err, saveErr)
|
||||
}
|
||||
return nil, fmt.Errorf("post-archive cleanup: %w", err)
|
||||
}
|
||||
completedAt := nowUTC()
|
||||
runManifest.MarkSucceeded(completedAt)
|
||||
syncRunManifestIdentityFromSession(m, runManifest)
|
||||
if err := runManifestStore.SaveRun(ctx, runManifestPath, runManifest); err != nil {
|
||||
return nil, fmt.Errorf("save final run manifest %q: %w", runManifestPath, err)
|
||||
}
|
||||
|
||||
return &RunSummary{
|
||||
SessionID: cfg.Session.SessionID,
|
||||
ManifestPath: manifestPath,
|
||||
StageNames: runNames,
|
||||
Executed: executed,
|
||||
Skipped: skipped,
|
||||
SessionID: cfg.Session.SessionID,
|
||||
RunID: runID,
|
||||
ManifestPath: manifestPath,
|
||||
RunManifestPath: runManifestPath,
|
||||
StageNames: runNames,
|
||||
Executed: executed,
|
||||
Skipped: skipped,
|
||||
}, nil
|
||||
}
|
||||
|
||||
@@ -317,11 +386,12 @@ func fileExists(path string) (bool, error) {
|
||||
return false, err
|
||||
}
|
||||
|
||||
func mapResultOutputs(result *stage.StageResult) []manifest.ArtifactRecord {
|
||||
func mapResultOutputs(result *stage.StageResult, runID string) []manifest.ArtifactRecord {
|
||||
if result == nil || len(result.Outputs) == 0 {
|
||||
return nil
|
||||
}
|
||||
|
||||
runID = strings.TrimSpace(runID)
|
||||
out := make([]manifest.ArtifactRecord, 0, len(result.Outputs))
|
||||
for _, ref := range result.Outputs {
|
||||
localPath := ref.AbsolutePath
|
||||
@@ -329,10 +399,11 @@ func mapResultOutputs(result *stage.StageResult) []manifest.ArtifactRecord {
|
||||
localPath = ref.RelativePath
|
||||
}
|
||||
out = append(out, manifest.ArtifactRecord{
|
||||
Kind: ref.Kind,
|
||||
LocalPath: localPath,
|
||||
RemoteKey: ref.RemoteKey,
|
||||
Checksum: ref.Checksum,
|
||||
Kind: ref.Kind,
|
||||
LocalPath: localPath,
|
||||
ProducerRunID: runID,
|
||||
RemoteKey: ref.RemoteKey,
|
||||
Checksum: ref.Checksum,
|
||||
})
|
||||
}
|
||||
|
||||
@@ -358,7 +429,7 @@ func applyStageResultToManifest(m *manifest.Manifest, stageName string, result *
|
||||
}
|
||||
}
|
||||
|
||||
func ensureManifestIdentity(cfg *config.Config, m *manifest.Manifest) (bool, error) {
|
||||
func ensureManifestIdentity(cfg *config.Config, m *manifest.Manifest, runID string) (bool, error) {
|
||||
if cfg == nil || cfg.Pipeline == nil || cfg.Session == nil || m == nil {
|
||||
return false, nil
|
||||
}
|
||||
@@ -374,16 +445,13 @@ func ensureManifestIdentity(cfg *config.Config, m *manifest.Manifest) (bool, err
|
||||
m.Campaign = campaign
|
||||
changed = true
|
||||
}
|
||||
if m.RunID == "" {
|
||||
runID, err := artifacts.NewRunID()
|
||||
if err != nil {
|
||||
return false, err
|
||||
}
|
||||
runID = strings.TrimSpace(runID)
|
||||
if runID != "" && m.RunID != runID {
|
||||
m.RunID = runID
|
||||
changed = true
|
||||
}
|
||||
if m.LocalWorkDir == "" && campaign != "" && sessionID != "" && m.RunID != "" {
|
||||
m.LocalWorkDir = artifacts.SessionRunWorkDir(cfg.Pipeline.Workspace.Root, campaign, sessionID, m.RunID)
|
||||
m.LocalWorkDir = artifacts.SessionRunRootForCampaign(cfg.Pipeline.Workspace.Root, campaign, sessionID, m.RunID)
|
||||
changed = true
|
||||
}
|
||||
if m.LocalSpoolDir == "" && campaign != "" && sessionID != "" && m.RunID != "" && strings.TrimSpace(cfg.Pipeline.Spool.Root) != "" {
|
||||
@@ -410,8 +478,54 @@ func ensureManifestIdentity(cfg *config.Config, m *manifest.Manifest) (bool, err
|
||||
return changed, nil
|
||||
}
|
||||
|
||||
func requestedStageNames(stages []stage.Stage) []string {
|
||||
out := make([]string, 0, len(stages))
|
||||
for _, s := range stages {
|
||||
if s == nil {
|
||||
continue
|
||||
}
|
||||
out = append(out, s.Name())
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
func applyStageResultToRunManifest(m *manifest.RunManifest, stageName string, result *stage.StageResult) {
|
||||
if m == nil || result == nil {
|
||||
return
|
||||
}
|
||||
sr := m.Stages[stageName]
|
||||
if sr == nil {
|
||||
return
|
||||
}
|
||||
if len(result.Logs) > 0 {
|
||||
sr.Logs = append([]string(nil), result.Logs...)
|
||||
}
|
||||
if len(result.GeneratedConfigs) > 0 {
|
||||
sr.GeneratedConfigs = append([]string(nil), result.GeneratedConfigs...)
|
||||
}
|
||||
if len(result.Metadata) > 0 {
|
||||
sr.Metadata = result.Metadata
|
||||
}
|
||||
}
|
||||
|
||||
func syncRunManifestIdentityFromSession(session *manifest.Manifest, run *manifest.RunManifest) {
|
||||
if session == nil || run == nil {
|
||||
return
|
||||
}
|
||||
run.Campaign = session.Campaign
|
||||
run.LocalWorkDir = session.LocalWorkDir
|
||||
run.LocalSpoolDir = session.LocalSpoolDir
|
||||
run.S3Bucket = session.S3Bucket
|
||||
run.S3SessionPrefix = session.S3SessionPrefix
|
||||
run.S3RunPrefix = session.S3RunPrefix
|
||||
}
|
||||
|
||||
func manifestPathFor(cfg *config.Config) string {
|
||||
return filepath.Join(cfg.Pipeline.Workspace.Root, "work", cfg.Session.SessionID, "manifest.json")
|
||||
return artifacts.SessionManifestPathForCampaign(
|
||||
cfg.Pipeline.Workspace.Root,
|
||||
cfg.Session.Campaign,
|
||||
cfg.Session.SessionID,
|
||||
)
|
||||
}
|
||||
|
||||
func needsObjectStoreForRun(cfg *config.Config, stages []stage.Stage) bool {
|
||||
|
||||
@@ -235,6 +235,56 @@ func TestExecuteStagesForceRerunsSucceeded(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
func TestExecuteStagesForceSuccessInvalidatesDownstreamSucceededStages(t *testing.T) {
|
||||
cfg := testConfig(t)
|
||||
manifestPath := manifestPathFor(cfg)
|
||||
store := &manifest.LocalStore{}
|
||||
|
||||
existing := manifest.New(cfg.Session.SessionID, time.Date(2026, 5, 3, 1, 0, 0, 0, time.UTC))
|
||||
for _, stageName := range []string{"prepare", "transcribe", "merge", "polish", "normalize", "trim", "archive", "notify"} {
|
||||
existing.MarkStageSucceeded(stageName, time.Date(2026, 5, 3, 1, 1, 0, 0, time.UTC), nil)
|
||||
}
|
||||
existing.MarkStageFailed("analyze", time.Date(2026, 5, 3, 1, 1, 0, 0, time.UTC), "previous analyze failure")
|
||||
if err := os.MkdirAll(filepath.Dir(manifestPath), 0o755); err != nil {
|
||||
t.Fatalf("MkdirAll() error = %v", err)
|
||||
}
|
||||
if err := store.Save(context.Background(), manifestPath, existing); err != nil {
|
||||
t.Fatalf("Save manifest error = %v", err)
|
||||
}
|
||||
|
||||
runs := 0
|
||||
stageToRun := countingStage{name: "polish", runs: &runs}
|
||||
summary, err := executeStages(context.Background(), cfg, []stage.Stage{stageToRun}, RunOptions{Force: true})
|
||||
if err != nil {
|
||||
t.Fatalf("executeStages() error = %v", err)
|
||||
}
|
||||
if runs != 1 {
|
||||
t.Fatalf("runs = %d, want 1 with force", runs)
|
||||
}
|
||||
if len(summary.Executed) != 1 || summary.Executed[0] != "polish" || len(summary.Skipped) != 0 {
|
||||
t.Fatalf("summary = %#v, want executed polish", summary)
|
||||
}
|
||||
|
||||
loaded, err := store.Load(context.Background(), manifestPath)
|
||||
if err != nil {
|
||||
t.Fatalf("Load manifest error = %v", err)
|
||||
}
|
||||
if loaded.Stages["polish"] == nil || loaded.Stages["polish"].Status != manifest.StatusSucceeded {
|
||||
t.Fatalf("polish status = %#v, want succeeded", loaded.Stages["polish"])
|
||||
}
|
||||
for _, stageName := range []string{"normalize", "trim", "archive", "notify"} {
|
||||
if loaded.Stages[stageName] == nil || loaded.Stages[stageName].Status != manifest.StatusStale {
|
||||
t.Fatalf("%s status = %#v, want stale", stageName, loaded.Stages[stageName])
|
||||
}
|
||||
}
|
||||
if loaded.Stages["analyze"] == nil || loaded.Stages["analyze"].Status != manifest.StatusFailed {
|
||||
t.Fatalf("analyze status = %#v, want preserved failed", loaded.Stages["analyze"])
|
||||
}
|
||||
if loaded.Stages["transcribe"] == nil || loaded.Stages["transcribe"].Status != manifest.StatusSucceeded {
|
||||
t.Fatalf("transcribe status = %#v, want preserved succeeded", loaded.Stages["transcribe"])
|
||||
}
|
||||
}
|
||||
|
||||
func TestExecuteStagesFailureUpdatesManifest(t *testing.T) {
|
||||
cfg := testConfig(t)
|
||||
|
||||
@@ -280,7 +330,14 @@ func TestExecuteStagesLoadsExistingManifest(t *testing.T) {
|
||||
|
||||
existing := manifest.New(cfg.Session.SessionID, time.Date(2026, 5, 3, 1, 0, 0, 0, time.UTC))
|
||||
existing.MarkStageSucceeded("prepare", time.Date(2026, 5, 3, 1, 1, 0, 0, time.UTC), nil)
|
||||
audioPath := filepath.Join(cfg.Pipeline.Workspace.Root, "work", cfg.Session.SessionID, "audio", "alice.flac")
|
||||
audioPath := filepath.Join(
|
||||
cfg.Pipeline.Workspace.Root,
|
||||
"work",
|
||||
cfg.Session.Campaign,
|
||||
cfg.Session.SessionID,
|
||||
"audio",
|
||||
"alice.flac",
|
||||
)
|
||||
if err := os.MkdirAll(filepath.Dir(audioPath), 0o755); err != nil {
|
||||
t.Fatalf("MkdirAll() error = %v", err)
|
||||
}
|
||||
@@ -315,6 +372,208 @@ func TestExecuteStagesLoadsExistingManifest(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
func TestExecuteStagesCreatesRunManifestPerInvocation(t *testing.T) {
|
||||
cfg := testConfig(t)
|
||||
run1, err := executeStages(context.Background(), cfg, []stage.Stage{BuildFullPlan()[0]}, RunOptions{})
|
||||
if err != nil {
|
||||
t.Fatalf("first executeStages() error = %v", err)
|
||||
}
|
||||
run2, err := executeStages(context.Background(), cfg, []stage.Stage{BuildFullPlan()[0]}, RunOptions{Force: true})
|
||||
if err != nil {
|
||||
t.Fatalf("second executeStages() error = %v", err)
|
||||
}
|
||||
|
||||
if run1.RunID == "" || run2.RunID == "" {
|
||||
t.Fatalf("run ids must be set, got %q and %q", run1.RunID, run2.RunID)
|
||||
}
|
||||
if run1.RunID == run2.RunID {
|
||||
t.Fatalf("expected distinct run ids, got %q", run1.RunID)
|
||||
}
|
||||
if run1.RunManifestPath == "" || run2.RunManifestPath == "" {
|
||||
t.Fatalf("run manifest paths must be set, got %q and %q", run1.RunManifestPath, run2.RunManifestPath)
|
||||
}
|
||||
if run1.RunManifestPath == run2.RunManifestPath {
|
||||
t.Fatalf("expected distinct run manifest paths, got %q", run1.RunManifestPath)
|
||||
}
|
||||
for _, path := range []string{run1.RunManifestPath, run2.RunManifestPath} {
|
||||
if _, statErr := os.Stat(path); statErr != nil {
|
||||
t.Fatalf("run manifest missing at %q: %v", path, statErr)
|
||||
}
|
||||
}
|
||||
|
||||
store := &manifest.LocalStore{}
|
||||
sessionManifest, err := store.Load(context.Background(), run2.ManifestPath)
|
||||
if err != nil {
|
||||
t.Fatalf("Load session manifest error = %v", err)
|
||||
}
|
||||
if sessionManifest.RunID != run2.RunID {
|
||||
t.Fatalf("session manifest run_id = %q, want latest run id %q", sessionManifest.RunID, run2.RunID)
|
||||
}
|
||||
}
|
||||
|
||||
func TestExecuteStagesRunManifestRecordsSkippedStage(t *testing.T) {
|
||||
cfg := testConfig(t)
|
||||
store := &manifest.LocalStore{}
|
||||
manifestPath := manifestPathFor(cfg)
|
||||
existing := manifest.New(cfg.Session.SessionID, time.Date(2026, 5, 3, 1, 0, 0, 0, time.UTC))
|
||||
existing.MarkStageSucceeded("transcribe", time.Date(2026, 5, 3, 1, 1, 0, 0, time.UTC), nil)
|
||||
if err := os.MkdirAll(filepath.Dir(manifestPath), 0o755); err != nil {
|
||||
t.Fatalf("MkdirAll() error = %v", err)
|
||||
}
|
||||
if err := store.Save(context.Background(), manifestPath, existing); err != nil {
|
||||
t.Fatalf("Save manifest error = %v", err)
|
||||
}
|
||||
|
||||
summary, err := executeStages(context.Background(), cfg, []stage.Stage{BuildFullPlan()[1]}, RunOptions{})
|
||||
if err != nil {
|
||||
t.Fatalf("executeStages() error = %v", err)
|
||||
}
|
||||
if len(summary.Skipped) != 1 || summary.Skipped[0] != "transcribe" {
|
||||
t.Fatalf("summary = %#v, want skipped transcribe", summary)
|
||||
}
|
||||
|
||||
runManifest, err := store.LoadRun(context.Background(), summary.RunManifestPath)
|
||||
if err != nil {
|
||||
t.Fatalf("LoadRun() error = %v", err)
|
||||
}
|
||||
sr := runManifest.Stages["transcribe"]
|
||||
if sr == nil {
|
||||
t.Fatal("run manifest transcribe stage missing")
|
||||
}
|
||||
if sr.Action != manifest.RunStageActionSkip {
|
||||
t.Fatalf("action = %q, want %q", sr.Action, manifest.RunStageActionSkip)
|
||||
}
|
||||
if sr.Status != manifest.StatusSkipped {
|
||||
t.Fatalf("status = %q, want %q", sr.Status, manifest.StatusSkipped)
|
||||
}
|
||||
}
|
||||
|
||||
func TestExecuteStagesRunLocalArtifactsAndCanonicalPromotion(t *testing.T) {
|
||||
cfg := testConfig(t)
|
||||
stages := []stage.Stage{
|
||||
BuildFullPlan()[0], // prepare
|
||||
BuildFullPlan()[1], // transcribe
|
||||
BuildFullPlan()[2], // merge
|
||||
BuildFullPlan()[3], // polish
|
||||
BuildFullPlan()[4], // normalize
|
||||
BuildFullPlan()[5], // trim
|
||||
}
|
||||
|
||||
summary, err := executeStages(context.Background(), cfg, stages, RunOptions{})
|
||||
if err != nil {
|
||||
t.Fatalf("executeStages() error = %v", err)
|
||||
}
|
||||
if summary.RunID == "" {
|
||||
t.Fatal("run id must be set")
|
||||
}
|
||||
|
||||
runRoot := artifacts.SessionRunRootForCampaign(
|
||||
cfg.Pipeline.Workspace.Root,
|
||||
cfg.Session.Campaign,
|
||||
cfg.Session.SessionID,
|
||||
summary.RunID,
|
||||
)
|
||||
paths := artifacts.NewLocalStore(cfg.Pipeline.Workspace.Root).SessionPathsFor(cfg.Session.Campaign, cfg.Session.SessionID)
|
||||
|
||||
runLocalChecks := []string{
|
||||
filepath.Join(runRoot, "transcribe", "outputs", "transcripts", "raw", "alice.json"),
|
||||
filepath.Join(runRoot, "merge", "logs", "seriatim.stdout.log"),
|
||||
filepath.Join(runRoot, "polish", "config", "audita.generated.yml"),
|
||||
filepath.Join(runRoot, "normalize", "logs", "seriatim.normalize.stdout.log"),
|
||||
filepath.Join(runRoot, "trim", "outputs", "transcripts", "trimmed.json"),
|
||||
}
|
||||
for _, p := range runLocalChecks {
|
||||
if _, statErr := os.Stat(p); statErr != nil {
|
||||
t.Fatalf("run-local artifact missing at %q: %v", p, statErr)
|
||||
}
|
||||
}
|
||||
|
||||
canonicalChecks := []string{
|
||||
filepath.Join(paths.TranscriptsRawDir, "alice.json"),
|
||||
filepath.Join(paths.TranscriptsDir, "merged.json"),
|
||||
filepath.Join(paths.TranscriptsDir, "processed.json"),
|
||||
filepath.Join(paths.TranscriptsDir, "normalized.json"),
|
||||
filepath.Join(paths.TranscriptsDir, "trimmed.json"),
|
||||
}
|
||||
for _, p := range canonicalChecks {
|
||||
if _, statErr := os.Stat(p); statErr != nil {
|
||||
t.Fatalf("canonical promoted artifact missing at %q: %v", p, statErr)
|
||||
}
|
||||
}
|
||||
|
||||
store := &manifest.LocalStore{}
|
||||
sessionManifest, err := store.Load(context.Background(), summary.ManifestPath)
|
||||
if err != nil {
|
||||
t.Fatalf("Load manifest error = %v", err)
|
||||
}
|
||||
if got := sessionManifest.Stages["trim"]; got == nil || len(got.Outputs) == 0 {
|
||||
t.Fatalf("trim stage outputs missing in session manifest: %#v", got)
|
||||
}
|
||||
for _, out := range sessionManifest.Stages["trim"].Outputs {
|
||||
if strings.Contains(out.LocalPath, string(filepath.Separator)+"runs"+string(filepath.Separator)) {
|
||||
t.Fatalf("session manifest output should be canonical, got run-local path %q", out.LocalPath)
|
||||
}
|
||||
if out.ProducerRunID != summary.RunID {
|
||||
t.Fatalf("producer_run_id = %q, want %q", out.ProducerRunID, summary.RunID)
|
||||
}
|
||||
}
|
||||
|
||||
runManifest, err := store.LoadRun(context.Background(), summary.RunManifestPath)
|
||||
if err != nil {
|
||||
t.Fatalf("LoadRun() error = %v", err)
|
||||
}
|
||||
mergeStage := runManifest.Stages["merge"]
|
||||
if mergeStage == nil || len(mergeStage.Logs) == 0 {
|
||||
t.Fatalf("merge logs missing in run manifest: %#v", mergeStage)
|
||||
}
|
||||
for _, logPath := range mergeStage.Logs {
|
||||
if !strings.Contains(logPath, filepath.Join("runs", summary.RunID, "merge", "logs")) {
|
||||
t.Fatalf("run manifest merge log path = %q, want run-local merge logs path", logPath)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestExecuteStagesSkippedStagePreservesExistingOutputsProvenance(t *testing.T) {
|
||||
cfg := testConfig(t)
|
||||
store := &manifest.LocalStore{}
|
||||
manifestPath := manifestPathFor(cfg)
|
||||
|
||||
existing := manifest.New(cfg.Session.SessionID, time.Date(2026, 5, 3, 1, 0, 0, 0, time.UTC))
|
||||
existing.MarkStageSucceeded("transcribe", time.Date(2026, 5, 3, 1, 1, 0, 0, time.UTC), []manifest.ArtifactRecord{
|
||||
{
|
||||
Kind: "transcript_raw",
|
||||
LocalPath: "transcripts/raw/alice.json",
|
||||
ProducerRunID: "20260501T000000Z-deadbeef",
|
||||
},
|
||||
})
|
||||
if err := os.MkdirAll(filepath.Dir(manifestPath), 0o755); err != nil {
|
||||
t.Fatalf("MkdirAll() error = %v", err)
|
||||
}
|
||||
if err := store.Save(context.Background(), manifestPath, existing); err != nil {
|
||||
t.Fatalf("Save manifest error = %v", err)
|
||||
}
|
||||
|
||||
summary, err := executeStages(context.Background(), cfg, []stage.Stage{BuildFullPlan()[1]}, RunOptions{})
|
||||
if err != nil {
|
||||
t.Fatalf("executeStages() error = %v", err)
|
||||
}
|
||||
if len(summary.Skipped) != 1 || summary.Skipped[0] != "transcribe" {
|
||||
t.Fatalf("summary = %#v, want skipped transcribe", summary)
|
||||
}
|
||||
|
||||
loaded, err := store.Load(context.Background(), manifestPath)
|
||||
if err != nil {
|
||||
t.Fatalf("Load manifest error = %v", err)
|
||||
}
|
||||
got := loaded.Stages["transcribe"]
|
||||
if got == nil || len(got.Outputs) != 1 {
|
||||
t.Fatalf("transcribe outputs = %#v, want one preserved output", got)
|
||||
}
|
||||
if got.Outputs[0].ProducerRunID != "20260501T000000Z-deadbeef" {
|
||||
t.Fatalf("producer_run_id = %q, want preserved value", got.Outputs[0].ProducerRunID)
|
||||
}
|
||||
}
|
||||
|
||||
func TestAdapterBackedStageFailureMarksManifestFailed(t *testing.T) {
|
||||
cases := []struct {
|
||||
name string
|
||||
@@ -341,7 +600,7 @@ func TestAdapterBackedStageFailureMarksManifestFailed(t *testing.T) {
|
||||
tc.env.ArtifactStore = artifactStore
|
||||
tc.env.ManifestStore = &manifest.LocalStore{}
|
||||
if tc.name == "transcribe" {
|
||||
paths, ensureErr := artifactStore.EnsureLayout(cfg.Session.SessionID)
|
||||
paths, ensureErr := artifactStore.EnsureLayoutFor(cfg.Session.Campaign, cfg.Session.SessionID)
|
||||
if ensureErr != nil {
|
||||
t.Fatalf("EnsureLayout() error = %v", ensureErr)
|
||||
}
|
||||
@@ -356,7 +615,7 @@ func TestAdapterBackedStageFailureMarksManifestFailed(t *testing.T) {
|
||||
}
|
||||
}
|
||||
if tc.name == "merge" {
|
||||
paths, ensureErr := artifactStore.EnsureLayout(cfg.Session.SessionID)
|
||||
paths, ensureErr := artifactStore.EnsureLayoutFor(cfg.Session.Campaign, cfg.Session.SessionID)
|
||||
if ensureErr != nil {
|
||||
t.Fatalf("EnsureLayout() error = %v", ensureErr)
|
||||
}
|
||||
@@ -375,7 +634,7 @@ func TestAdapterBackedStageFailureMarksManifestFailed(t *testing.T) {
|
||||
}
|
||||
}
|
||||
if tc.name == "polish" {
|
||||
paths, ensureErr := artifactStore.EnsureLayout(cfg.Session.SessionID)
|
||||
paths, ensureErr := artifactStore.EnsureLayoutFor(cfg.Session.Campaign, cfg.Session.SessionID)
|
||||
if ensureErr != nil {
|
||||
t.Fatalf("EnsureLayout() error = %v", ensureErr)
|
||||
}
|
||||
@@ -387,7 +646,7 @@ func TestAdapterBackedStageFailureMarksManifestFailed(t *testing.T) {
|
||||
}
|
||||
}
|
||||
if tc.name == "analyze" {
|
||||
paths, ensureErr := artifactStore.EnsureLayout(cfg.Session.SessionID)
|
||||
paths, ensureErr := artifactStore.EnsureLayoutFor(cfg.Session.Campaign, cfg.Session.SessionID)
|
||||
if ensureErr != nil {
|
||||
t.Fatalf("EnsureLayout() error = %v", ensureErr)
|
||||
}
|
||||
|
||||
@@ -63,7 +63,7 @@ func TestExecuteStagesDefaultWiringUsesWhisperXHTTPClient(t *testing.T) {
|
||||
t.Fatal("audio file payload was empty")
|
||||
}
|
||||
|
||||
outPath := filepath.Join(cfg.Pipeline.Workspace.Root, "work", cfg.Session.SessionID, "transcripts", "raw", "alice.json")
|
||||
outPath := filepath.Join(cfg.Pipeline.Workspace.Root, "work", cfg.Session.Campaign, cfg.Session.SessionID, "transcripts", "raw", "alice.json")
|
||||
data, err := os.ReadFile(outPath)
|
||||
if err != nil {
|
||||
t.Fatalf("ReadFile(%q) error = %v", outPath, err)
|
||||
|
||||
294
internal/artifacts/artifact_resolver.go
Normal file
294
internal/artifacts/artifact_resolver.go
Normal file
@@ -0,0 +1,294 @@
|
||||
package artifacts
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
|
||||
"gitea.maximumdirect.net/eric/narratio/internal/manifest"
|
||||
)
|
||||
|
||||
const (
|
||||
ArtifactTranscriptMerged = "narratio.transcript.merged"
|
||||
ArtifactTranscriptPolished = "narratio.transcript.polished"
|
||||
ArtifactTranscriptFull = "narratio.transcript.full"
|
||||
ArtifactTranscriptTrimmed = "narratio.transcript.trimmed"
|
||||
ArtifactBoundsSession = "narratio.bounds.session"
|
||||
ArtifactSessionRecap = "narratio.artifact.session_recap"
|
||||
)
|
||||
|
||||
// ErrSessionArtifactNotFound is returned when no readable artifact exists for a known ID.
|
||||
var ErrSessionArtifactNotFound = errors.New("session artifact not found")
|
||||
|
||||
type artifactContentKind string
|
||||
|
||||
const (
|
||||
contentTranscriptJSON artifactContentKind = "transcript_json"
|
||||
contentJSON artifactContentKind = "json"
|
||||
contentText artifactContentKind = "text"
|
||||
)
|
||||
|
||||
type artifactSpec struct {
|
||||
ID string
|
||||
CanonicalRelPath string
|
||||
ProducerStage string
|
||||
OutputKind string
|
||||
ContentKind artifactContentKind
|
||||
}
|
||||
|
||||
var artifactRegistry = map[string]artifactSpec{
|
||||
ArtifactTranscriptMerged: {
|
||||
ID: ArtifactTranscriptMerged,
|
||||
CanonicalRelPath: "transcripts/merged.json",
|
||||
ProducerStage: "merge",
|
||||
OutputKind: "transcript_merged",
|
||||
ContentKind: contentTranscriptJSON,
|
||||
},
|
||||
ArtifactTranscriptPolished: {
|
||||
ID: ArtifactTranscriptPolished,
|
||||
CanonicalRelPath: "transcripts/processed.json",
|
||||
ProducerStage: "polish",
|
||||
OutputKind: "transcript_processed",
|
||||
ContentKind: contentTranscriptJSON,
|
||||
},
|
||||
ArtifactTranscriptFull: {
|
||||
ID: ArtifactTranscriptFull,
|
||||
CanonicalRelPath: "transcripts/normalized.json",
|
||||
ProducerStage: "normalize",
|
||||
OutputKind: "transcript_normalized",
|
||||
ContentKind: contentTranscriptJSON,
|
||||
},
|
||||
ArtifactTranscriptTrimmed: {
|
||||
ID: ArtifactTranscriptTrimmed,
|
||||
CanonicalRelPath: "transcripts/trimmed.json",
|
||||
ProducerStage: "trim",
|
||||
OutputKind: "transcript_trimmed",
|
||||
ContentKind: contentTranscriptJSON,
|
||||
},
|
||||
ArtifactBoundsSession: {
|
||||
ID: ArtifactBoundsSession,
|
||||
CanonicalRelPath: "artifacts/session_bounds.json",
|
||||
ProducerStage: "trim",
|
||||
OutputKind: "session_bounds",
|
||||
ContentKind: contentJSON,
|
||||
},
|
||||
ArtifactSessionRecap: {
|
||||
ID: ArtifactSessionRecap,
|
||||
CanonicalRelPath: "artifacts/session_recap.md",
|
||||
ProducerStage: "analyze",
|
||||
OutputKind: "session_recap",
|
||||
ContentKind: contentText,
|
||||
},
|
||||
}
|
||||
|
||||
var artifactAliases = map[string]string{
|
||||
"processed_transcript": ArtifactTranscriptPolished,
|
||||
"normalized_transcript": ArtifactTranscriptFull,
|
||||
"trimmed_transcript": ArtifactTranscriptTrimmed,
|
||||
}
|
||||
|
||||
// ResolvedSessionArtifact describes one session-level artifact lookup result.
|
||||
type ResolvedSessionArtifact struct {
|
||||
ID string
|
||||
Path string
|
||||
ProducerStage string
|
||||
OutputKind string
|
||||
ProducerRunID string
|
||||
Provenance string
|
||||
}
|
||||
|
||||
// SessionArtifactNotFoundError includes context when a known artifact cannot be read.
|
||||
type SessionArtifactNotFoundError struct {
|
||||
ArtifactID string
|
||||
}
|
||||
|
||||
func (e *SessionArtifactNotFoundError) Error() string {
|
||||
return fmt.Sprintf("%s: %q", ErrSessionArtifactNotFound, e.ArtifactID)
|
||||
}
|
||||
|
||||
func (e *SessionArtifactNotFoundError) Unwrap() error {
|
||||
return ErrSessionArtifactNotFound
|
||||
}
|
||||
|
||||
// NormalizeSessionArtifactSource maps legacy aliases to canonical IDs and validates IDs.
|
||||
func NormalizeSessionArtifactSource(source string) (string, error) {
|
||||
normalized := strings.TrimSpace(source)
|
||||
if normalized == "" {
|
||||
return "", fmt.Errorf("artifact source is required")
|
||||
}
|
||||
if alias, ok := artifactAliases[normalized]; ok {
|
||||
normalized = alias
|
||||
}
|
||||
if _, ok := artifactRegistry[normalized]; !ok {
|
||||
return "", fmt.Errorf("unsupported artifact source %q", source)
|
||||
}
|
||||
return normalized, nil
|
||||
}
|
||||
|
||||
// ResolveSessionArtifact resolves a symbolic source to a readable local session artifact path.
|
||||
// Resolution order is manifest producer outputs first, then canonical session path fallback.
|
||||
func ResolveSessionArtifact(paths SessionPaths, m *manifest.Manifest, source string) (ResolvedSessionArtifact, error) {
|
||||
id, err := NormalizeSessionArtifactSource(source)
|
||||
if err != nil {
|
||||
return ResolvedSessionArtifact{}, err
|
||||
}
|
||||
spec := artifactRegistry[id]
|
||||
|
||||
for _, candidate := range manifestArtifactCandidates(paths, m, spec) {
|
||||
exists, isDir, statErr := pathExists(candidate.Path)
|
||||
if statErr != nil {
|
||||
return ResolvedSessionArtifact{}, fmt.Errorf("stat %q: %w", candidate.Path, statErr)
|
||||
}
|
||||
if !exists || isDir {
|
||||
continue
|
||||
}
|
||||
resolved := candidate
|
||||
resolved.ID = spec.ID
|
||||
resolved.ProducerStage = spec.ProducerStage
|
||||
resolved.OutputKind = spec.OutputKind
|
||||
if err := validateResolvedContent(resolved.Path, spec.ContentKind); err != nil {
|
||||
return ResolvedSessionArtifact{}, fmt.Errorf("validate %q: %w", resolved.ID, err)
|
||||
}
|
||||
return resolved, nil
|
||||
}
|
||||
|
||||
fallbackPath := filepath.Join(paths.Root, filepath.FromSlash(spec.CanonicalRelPath))
|
||||
exists, isDir, statErr := pathExists(fallbackPath)
|
||||
if statErr != nil {
|
||||
return ResolvedSessionArtifact{}, fmt.Errorf("stat %q: %w", fallbackPath, statErr)
|
||||
}
|
||||
if exists && !isDir {
|
||||
if err := validateResolvedContent(fallbackPath, spec.ContentKind); err != nil {
|
||||
return ResolvedSessionArtifact{}, fmt.Errorf("validate %q: %w", spec.ID, err)
|
||||
}
|
||||
return ResolvedSessionArtifact{
|
||||
ID: spec.ID,
|
||||
Path: filepath.Clean(fallbackPath),
|
||||
ProducerStage: spec.ProducerStage,
|
||||
OutputKind: spec.OutputKind,
|
||||
Provenance: "fallback.canonical_path",
|
||||
}, nil
|
||||
}
|
||||
|
||||
return ResolvedSessionArtifact{}, &SessionArtifactNotFoundError{ArtifactID: spec.ID}
|
||||
}
|
||||
|
||||
func manifestArtifactCandidates(paths SessionPaths, m *manifest.Manifest, spec artifactSpec) []ResolvedSessionArtifact {
|
||||
if m == nil || len(m.Stages) == 0 || spec.ProducerStage == "" || spec.OutputKind == "" {
|
||||
return nil
|
||||
}
|
||||
sr := m.Stages[spec.ProducerStage]
|
||||
if sr == nil {
|
||||
return nil
|
||||
}
|
||||
|
||||
candidates := make([]ResolvedSessionArtifact, 0, len(sr.Outputs))
|
||||
for _, out := range sr.Outputs {
|
||||
if strings.TrimSpace(out.Kind) != spec.OutputKind {
|
||||
continue
|
||||
}
|
||||
if strings.TrimSpace(out.LocalPath) == "" {
|
||||
continue
|
||||
}
|
||||
resolved := filepath.Clean(ResolveSessionLocalPathForRead(paths, out.LocalPath))
|
||||
if resolved == "" {
|
||||
continue
|
||||
}
|
||||
candidates = append(candidates, ResolvedSessionArtifact{
|
||||
Path: resolved,
|
||||
ProducerRunID: strings.TrimSpace(out.ProducerRunID),
|
||||
Provenance: "manifest." + spec.ProducerStage + ".outputs",
|
||||
})
|
||||
}
|
||||
return dedupeResolvedArtifacts(candidates)
|
||||
}
|
||||
|
||||
func dedupeResolvedArtifacts(values []ResolvedSessionArtifact) []ResolvedSessionArtifact {
|
||||
seen := map[string]struct{}{}
|
||||
out := make([]ResolvedSessionArtifact, 0, len(values))
|
||||
for _, value := range values {
|
||||
key := filepath.Clean(strings.TrimSpace(value.Path))
|
||||
if key == "" {
|
||||
continue
|
||||
}
|
||||
if _, ok := seen[key]; ok {
|
||||
continue
|
||||
}
|
||||
seen[key] = struct{}{}
|
||||
value.Path = key
|
||||
out = append(out, value)
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
func pathExists(path string) (exists bool, isDir bool, err error) {
|
||||
info, err := os.Stat(path)
|
||||
if err == nil {
|
||||
return true, info.IsDir(), nil
|
||||
}
|
||||
if errors.Is(err, os.ErrNotExist) {
|
||||
return false, false, nil
|
||||
}
|
||||
return false, false, err
|
||||
}
|
||||
|
||||
func validateResolvedContent(path string, kind artifactContentKind) error {
|
||||
switch kind {
|
||||
case contentTranscriptJSON:
|
||||
return validateTranscriptSegmentsJSON(path)
|
||||
case contentJSON:
|
||||
return validateJSONContent(path)
|
||||
case contentText:
|
||||
return validateNonEmptyContent(path)
|
||||
default:
|
||||
return fmt.Errorf("unsupported content kind %q", kind)
|
||||
}
|
||||
}
|
||||
|
||||
func validateTranscriptSegmentsJSON(path string) error {
|
||||
data, err := os.ReadFile(path)
|
||||
if err != nil {
|
||||
return fmt.Errorf("read file: %w", err)
|
||||
}
|
||||
var payload map[string]any
|
||||
if err := json.Unmarshal(data, &payload); err != nil {
|
||||
return fmt.Errorf("decode json: %w", err)
|
||||
}
|
||||
segments, ok := payload["segments"]
|
||||
if !ok {
|
||||
return fmt.Errorf("top-level segments is required")
|
||||
}
|
||||
if _, ok := segments.([]any); !ok {
|
||||
return fmt.Errorf("top-level segments must be an array")
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func validateJSONContent(path string) error {
|
||||
data, err := os.ReadFile(path)
|
||||
if err != nil {
|
||||
return fmt.Errorf("read file: %w", err)
|
||||
}
|
||||
var payload any
|
||||
if err := json.Unmarshal(data, &payload); err != nil {
|
||||
return fmt.Errorf("decode json: %w", err)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func validateNonEmptyContent(path string) error {
|
||||
info, err := os.Stat(path)
|
||||
if err != nil {
|
||||
return fmt.Errorf("stat file: %w", err)
|
||||
}
|
||||
if info.IsDir() {
|
||||
return fmt.Errorf("path is a directory")
|
||||
}
|
||||
if info.Size() <= 0 {
|
||||
return fmt.Errorf("file is empty")
|
||||
}
|
||||
return nil
|
||||
}
|
||||
139
internal/artifacts/artifact_resolver_test.go
Normal file
139
internal/artifacts/artifact_resolver_test.go
Normal file
@@ -0,0 +1,139 @@
|
||||
package artifacts
|
||||
|
||||
import (
|
||||
"errors"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"gitea.maximumdirect.net/eric/narratio/internal/manifest"
|
||||
)
|
||||
|
||||
func TestNormalizeSessionArtifactSource(t *testing.T) {
|
||||
tests := []struct {
|
||||
name string
|
||||
source string
|
||||
wantID string
|
||||
wantErr string
|
||||
}{
|
||||
{name: "legacy alias processed", source: "processed_transcript", wantID: ArtifactTranscriptPolished},
|
||||
{name: "legacy alias normalized", source: "normalized_transcript", wantID: ArtifactTranscriptFull},
|
||||
{name: "legacy alias trimmed", source: "trimmed_transcript", wantID: ArtifactTranscriptTrimmed},
|
||||
{name: "canonical", source: ArtifactTranscriptTrimmed, wantID: ArtifactTranscriptTrimmed},
|
||||
{name: "unsupported", source: "narratio.unknown", wantErr: "unsupported artifact source"},
|
||||
}
|
||||
|
||||
for _, tt := range tests {
|
||||
t.Run(tt.name, func(t *testing.T) {
|
||||
got, err := NormalizeSessionArtifactSource(tt.source)
|
||||
if tt.wantErr != "" {
|
||||
if err == nil || !strings.Contains(err.Error(), tt.wantErr) {
|
||||
t.Fatalf("NormalizeSessionArtifactSource() error = %v, want contains %q", err, tt.wantErr)
|
||||
}
|
||||
return
|
||||
}
|
||||
if err != nil {
|
||||
t.Fatalf("NormalizeSessionArtifactSource() error = %v", err)
|
||||
}
|
||||
if got != tt.wantID {
|
||||
t.Fatalf("NormalizeSessionArtifactSource() = %q, want %q", got, tt.wantID)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestResolveSessionArtifactPrefersManifestOutput(t *testing.T) {
|
||||
workspace := t.TempDir()
|
||||
paths := buildSessionPaths(workspace, "campaign", "session")
|
||||
if err := os.MkdirAll(paths.ArtifactsDir, 0o755); err != nil {
|
||||
t.Fatalf("MkdirAll() error = %v", err)
|
||||
}
|
||||
manifestPath := filepath.Join(paths.ArtifactsDir, "normalized.from-manifest.json")
|
||||
if err := os.WriteFile(manifestPath, []byte(`{"segments":[]}`), 0o644); err != nil {
|
||||
t.Fatalf("WriteFile() error = %v", err)
|
||||
}
|
||||
canonicalPath := filepath.Join(paths.TranscriptsDir, "normalized.json")
|
||||
if err := os.MkdirAll(filepath.Dir(canonicalPath), 0o755); err != nil {
|
||||
t.Fatalf("MkdirAll() error = %v", err)
|
||||
}
|
||||
if err := os.WriteFile(canonicalPath, []byte(`{"segments":[{"id":123}]}`), 0o644); err != nil {
|
||||
t.Fatalf("WriteFile() error = %v", err)
|
||||
}
|
||||
|
||||
m := manifest.New("session", time.Now().UTC())
|
||||
m.MarkStageSucceeded("normalize", time.Now().UTC(), []manifest.ArtifactRecord{
|
||||
{Kind: "transcript_normalized", LocalPath: manifestPath, ProducerRunID: "run-123"},
|
||||
})
|
||||
|
||||
resolved, err := ResolveSessionArtifact(paths, m, "normalized_transcript")
|
||||
if err != nil {
|
||||
t.Fatalf("ResolveSessionArtifact() error = %v", err)
|
||||
}
|
||||
if resolved.Path != manifestPath {
|
||||
t.Fatalf("resolved path = %q, want %q", resolved.Path, manifestPath)
|
||||
}
|
||||
if resolved.Provenance != "manifest.normalize.outputs" {
|
||||
t.Fatalf("provenance = %q, want manifest.normalize.outputs", resolved.Provenance)
|
||||
}
|
||||
if resolved.ProducerRunID != "run-123" {
|
||||
t.Fatalf("producer run id = %q, want run-123", resolved.ProducerRunID)
|
||||
}
|
||||
}
|
||||
|
||||
func TestResolveSessionArtifactFallsBackToCanonicalPath(t *testing.T) {
|
||||
workspace := t.TempDir()
|
||||
paths := buildSessionPaths(workspace, "campaign", "session")
|
||||
canonicalPath := filepath.Join(paths.TranscriptsDir, "trimmed.json")
|
||||
if err := os.MkdirAll(filepath.Dir(canonicalPath), 0o755); err != nil {
|
||||
t.Fatalf("MkdirAll() error = %v", err)
|
||||
}
|
||||
if err := os.WriteFile(canonicalPath, []byte(`{"segments":[]}`), 0o644); err != nil {
|
||||
t.Fatalf("WriteFile() error = %v", err)
|
||||
}
|
||||
|
||||
resolved, err := ResolveSessionArtifact(paths, nil, ArtifactTranscriptTrimmed)
|
||||
if err != nil {
|
||||
t.Fatalf("ResolveSessionArtifact() error = %v", err)
|
||||
}
|
||||
if resolved.Path != canonicalPath {
|
||||
t.Fatalf("resolved path = %q, want %q", resolved.Path, canonicalPath)
|
||||
}
|
||||
if resolved.Provenance != "fallback.canonical_path" {
|
||||
t.Fatalf("provenance = %q, want fallback.canonical_path", resolved.Provenance)
|
||||
}
|
||||
}
|
||||
|
||||
func TestResolveSessionArtifactMissingReturnsTypedError(t *testing.T) {
|
||||
workspace := t.TempDir()
|
||||
paths := buildSessionPaths(workspace, "campaign", "session")
|
||||
|
||||
_, err := ResolveSessionArtifact(paths, nil, ArtifactTranscriptTrimmed)
|
||||
if err == nil {
|
||||
t.Fatal("expected error, got nil")
|
||||
}
|
||||
if !errors.Is(err, ErrSessionArtifactNotFound) {
|
||||
t.Fatalf("errors.Is(err, ErrSessionArtifactNotFound) = false; err=%v", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestResolveSessionArtifactValidatesTranscriptShape(t *testing.T) {
|
||||
workspace := t.TempDir()
|
||||
paths := buildSessionPaths(workspace, "campaign", "session")
|
||||
canonicalPath := filepath.Join(paths.TranscriptsDir, "processed.json")
|
||||
if err := os.MkdirAll(filepath.Dir(canonicalPath), 0o755); err != nil {
|
||||
t.Fatalf("MkdirAll() error = %v", err)
|
||||
}
|
||||
if err := os.WriteFile(canonicalPath, []byte(`{"not_segments":[]}`), 0o644); err != nil {
|
||||
t.Fatalf("WriteFile() error = %v", err)
|
||||
}
|
||||
|
||||
_, err := ResolveSessionArtifact(paths, nil, ArtifactTranscriptPolished)
|
||||
if err == nil {
|
||||
t.Fatal("expected error, got nil")
|
||||
}
|
||||
if !strings.Contains(err.Error(), "top-level segments is required") {
|
||||
t.Fatalf("error = %q, want segments validation error", err.Error())
|
||||
}
|
||||
}
|
||||
@@ -30,13 +30,13 @@ func NewLocalStore(workspaceRoot string) *LocalStore {
|
||||
return &LocalStore{WorkspaceRoot: workspaceRoot}
|
||||
}
|
||||
|
||||
// SessionPaths resolves canonical paths for a session workdir.
|
||||
func (s *LocalStore) SessionPaths(sessionID string) SessionPaths {
|
||||
return buildSessionPaths(s.WorkspaceRoot, sessionID)
|
||||
// SessionPathsFor resolves canonical campaign-aware paths for a session workdir.
|
||||
func (s *LocalStore) SessionPathsFor(campaign, sessionID string) SessionPaths {
|
||||
return buildSessionPaths(s.WorkspaceRoot, campaign, sessionID)
|
||||
}
|
||||
|
||||
// EnsureLayout creates and verifies the canonical session workdir directory layout.
|
||||
func (s *LocalStore) EnsureLayout(sessionID string) (SessionPaths, error) {
|
||||
// EnsureLayoutFor creates and verifies campaign-aware session layout.
|
||||
func (s *LocalStore) EnsureLayoutFor(campaign, sessionID string) (SessionPaths, error) {
|
||||
if strings.TrimSpace(s.WorkspaceRoot) == "" {
|
||||
return SessionPaths{}, fmt.Errorf("workspace root is required")
|
||||
}
|
||||
@@ -44,7 +44,22 @@ func (s *LocalStore) EnsureLayout(sessionID string) (SessionPaths, error) {
|
||||
return SessionPaths{}, fmt.Errorf("sessionID is required")
|
||||
}
|
||||
|
||||
paths := s.SessionPaths(sessionID)
|
||||
campaign = strings.TrimSpace(campaign)
|
||||
if campaign == "" {
|
||||
return SessionPaths{}, fmt.Errorf("campaign is required")
|
||||
}
|
||||
|
||||
return s.ensureLayout(s.SessionPathsFor(campaign, sessionID))
|
||||
}
|
||||
|
||||
func (s *LocalStore) ensureLayout(paths SessionPaths) (SessionPaths, error) {
|
||||
if strings.TrimSpace(s.WorkspaceRoot) == "" {
|
||||
return SessionPaths{}, fmt.Errorf("workspace root is required")
|
||||
}
|
||||
if strings.TrimSpace(paths.SessionID) == "" {
|
||||
return SessionPaths{}, fmt.Errorf("sessionID is required")
|
||||
}
|
||||
|
||||
dirs := []string{
|
||||
paths.Root,
|
||||
paths.InputsDir,
|
||||
@@ -53,8 +68,11 @@ func (s *LocalStore) EnsureLayout(sessionID string) (SessionPaths, error) {
|
||||
paths.TranscriptsRawDir,
|
||||
paths.TranscriptsTrimmedDir,
|
||||
paths.ArtifactsDir,
|
||||
paths.ReportsDir,
|
||||
paths.ConfigDir,
|
||||
paths.LogsDir,
|
||||
paths.CurrentDir,
|
||||
paths.RunsDir,
|
||||
}
|
||||
|
||||
for _, dir := range dirs {
|
||||
@@ -66,13 +84,16 @@ func (s *LocalStore) EnsureLayout(sessionID string) (SessionPaths, error) {
|
||||
return paths, nil
|
||||
}
|
||||
|
||||
// CopyInput copies an input file into the session workdir under destRelativePath.
|
||||
func (s *LocalStore) CopyInput(sessionID, srcPath, destRelativePath string) (Ref, error) {
|
||||
paths, err := s.EnsureLayout(sessionID)
|
||||
// CopyInputFor copies an input file into the campaign-aware session workdir under destRelativePath.
|
||||
func (s *LocalStore) CopyInputFor(campaign, sessionID, srcPath, destRelativePath string) (Ref, error) {
|
||||
paths, err := s.EnsureLayoutFor(campaign, sessionID)
|
||||
if err != nil {
|
||||
return Ref{}, err
|
||||
}
|
||||
return s.copyInputWithPaths(paths, sessionID, srcPath, destRelativePath)
|
||||
}
|
||||
|
||||
func (s *LocalStore) copyInputWithPaths(paths SessionPaths, sessionID, srcPath, destRelativePath string) (Ref, error) {
|
||||
destAbs, err := resolveInRoot(paths.Root, destRelativePath)
|
||||
if err != nil {
|
||||
return Ref{}, fmt.Errorf("copy input: %w", err)
|
||||
@@ -173,13 +194,16 @@ func (s *LocalStore) Checksum(path string) (string, error) {
|
||||
return digest, nil
|
||||
}
|
||||
|
||||
// AcquireSessionLock acquires an exclusive lock file for a session workdir.
|
||||
func (s *LocalStore) AcquireSessionLock(sessionID string) (*LockHandle, error) {
|
||||
paths, err := s.EnsureLayout(sessionID)
|
||||
// AcquireSessionLockFor acquires an exclusive lock file for a campaign/session workdir.
|
||||
func (s *LocalStore) AcquireSessionLockFor(campaign, sessionID string) (*LockHandle, error) {
|
||||
paths, err := s.EnsureLayoutFor(campaign, sessionID)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return s.acquireSessionLockForPaths(paths)
|
||||
}
|
||||
|
||||
func (s *LocalStore) acquireSessionLockForPaths(paths SessionPaths) (*LockHandle, error) {
|
||||
f, err := os.OpenFile(paths.LockPath, os.O_CREATE|os.O_EXCL|os.O_WRONLY, 0o644)
|
||||
if err != nil {
|
||||
if errors.Is(err, os.ErrExist) {
|
||||
|
||||
@@ -10,9 +10,9 @@ import (
|
||||
|
||||
func TestEnsureLayoutCreatesExpectedDirectories(t *testing.T) {
|
||||
store := NewLocalStore(t.TempDir())
|
||||
paths, err := store.EnsureLayout("session-1")
|
||||
paths, err := store.EnsureLayoutFor("sample-campaign", "session-1")
|
||||
if err != nil {
|
||||
t.Fatalf("EnsureLayout() error = %v", err)
|
||||
t.Fatalf("EnsureLayoutFor() error = %v", err)
|
||||
}
|
||||
|
||||
checkDirExists(t, paths.Root)
|
||||
@@ -22,8 +22,11 @@ func TestEnsureLayoutCreatesExpectedDirectories(t *testing.T) {
|
||||
checkDirExists(t, paths.TranscriptsRawDir)
|
||||
checkDirExists(t, paths.TranscriptsTrimmedDir)
|
||||
checkDirExists(t, paths.ArtifactsDir)
|
||||
checkDirExists(t, paths.ReportsDir)
|
||||
checkDirExists(t, paths.ConfigDir)
|
||||
checkDirExists(t, paths.LogsDir)
|
||||
checkDirExists(t, paths.CurrentDir)
|
||||
checkDirExists(t, paths.RunsDir)
|
||||
|
||||
if filepath.Base(paths.ManifestPath) != "manifest.json" {
|
||||
t.Fatalf("ManifestPath = %q, want basename manifest.json", paths.ManifestPath)
|
||||
@@ -33,6 +36,17 @@ func TestEnsureLayoutCreatesExpectedDirectories(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
func TestEnsureLayoutForRequiresCampaign(t *testing.T) {
|
||||
store := NewLocalStore(t.TempDir())
|
||||
_, err := store.EnsureLayoutFor("", "session-1")
|
||||
if err == nil {
|
||||
t.Fatal("expected campaign-required error, got nil")
|
||||
}
|
||||
if !strings.Contains(err.Error(), "campaign is required") {
|
||||
t.Fatalf("error = %v, want campaign-required error", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestChecksumCalculation(t *testing.T) {
|
||||
store := NewLocalStore(t.TempDir())
|
||||
path := filepath.Join(t.TempDir(), "sample.txt")
|
||||
@@ -53,9 +67,9 @@ func TestChecksumCalculation(t *testing.T) {
|
||||
func TestLockAcquireRelease(t *testing.T) {
|
||||
store := NewLocalStore(t.TempDir())
|
||||
|
||||
lock, err := store.AcquireSessionLock("session-1")
|
||||
lock, err := store.AcquireSessionLockFor("sample-campaign", "session-1")
|
||||
if err != nil {
|
||||
t.Fatalf("AcquireSessionLock() error = %v", err)
|
||||
t.Fatalf("AcquireSessionLockFor() error = %v", err)
|
||||
}
|
||||
|
||||
exists, err := store.Exists(lock.path)
|
||||
@@ -81,15 +95,15 @@ func TestLockAcquireRelease(t *testing.T) {
|
||||
|
||||
func TestLockConflict(t *testing.T) {
|
||||
store := NewLocalStore(t.TempDir())
|
||||
lock1, err := store.AcquireSessionLock("session-1")
|
||||
lock1, err := store.AcquireSessionLockFor("sample-campaign", "session-1")
|
||||
if err != nil {
|
||||
t.Fatalf("first AcquireSessionLock() error = %v", err)
|
||||
t.Fatalf("first AcquireSessionLockFor() error = %v", err)
|
||||
}
|
||||
defer func() {
|
||||
_ = store.ReleaseSessionLock(lock1)
|
||||
}()
|
||||
|
||||
_, err = store.AcquireSessionLock("session-1")
|
||||
_, err = store.AcquireSessionLockFor("sample-campaign", "session-1")
|
||||
if err == nil {
|
||||
t.Fatal("expected lock conflict error, got nil")
|
||||
}
|
||||
@@ -138,9 +152,9 @@ func TestCopyInput(t *testing.T) {
|
||||
t.Fatalf("WriteFile() error = %v", err)
|
||||
}
|
||||
|
||||
ref, err := store.CopyInput("session-1", srcPath, "inputs/speakers.yml")
|
||||
ref, err := store.CopyInputFor("sample-campaign", "session-1", srcPath, "inputs/speakers.yml")
|
||||
if err != nil {
|
||||
t.Fatalf("CopyInput() error = %v", err)
|
||||
t.Fatalf("CopyInputFor() error = %v", err)
|
||||
}
|
||||
|
||||
if ref.Kind != "input" {
|
||||
|
||||
@@ -2,11 +2,15 @@ package artifacts
|
||||
|
||||
import (
|
||||
"path/filepath"
|
||||
|
||||
"gitea.maximumdirect.net/eric/narratio/internal/config"
|
||||
)
|
||||
|
||||
// SessionPaths contains canonical local paths for one session work directory.
|
||||
type SessionPaths struct {
|
||||
WorkspaceRoot string
|
||||
CampaignID string
|
||||
SessionID string
|
||||
Root string
|
||||
InputsDir string
|
||||
AudioDir string
|
||||
@@ -14,42 +18,73 @@ type SessionPaths struct {
|
||||
TranscriptsRawDir string
|
||||
TranscriptsTrimmedDir string
|
||||
ArtifactsDir string
|
||||
ReportsDir string
|
||||
ConfigDir string
|
||||
LogsDir string
|
||||
CurrentDir string
|
||||
RunsDir string
|
||||
ManifestPath string
|
||||
LockPath string
|
||||
}
|
||||
|
||||
// SessionWorkDir returns the work directory for one session.
|
||||
func SessionWorkDir(rootDir, sessionID string) string {
|
||||
return filepath.Join(rootDir, "work", sessionID)
|
||||
// SessionWorkDirForCampaign returns the canonical campaign-aware work directory for one session.
|
||||
func SessionWorkDirForCampaign(rootDir, campaign, sessionID string) string {
|
||||
return filepath.Join(rootDir, config.PathWorkDirSegment, campaign, sessionID)
|
||||
}
|
||||
|
||||
// SessionRunWorkDir returns the campaign/session/run scoped local work directory.
|
||||
func SessionRunWorkDir(rootDir, campaign, sessionID, runID string) string {
|
||||
return filepath.Join(rootDir, "work", campaign, sessionID, runID)
|
||||
// SessionManifestPathForCampaign returns the canonical session manifest path.
|
||||
func SessionManifestPathForCampaign(rootDir, campaign, sessionID string) string {
|
||||
return filepath.Join(SessionWorkDirForCampaign(rootDir, campaign, sessionID), config.PathManifestFile)
|
||||
}
|
||||
|
||||
// SessionRunsDirForCampaign returns the canonical runs directory for one session.
|
||||
func SessionRunsDirForCampaign(rootDir, campaign, sessionID string) string {
|
||||
return filepath.Join(SessionWorkDirForCampaign(rootDir, campaign, sessionID), config.PathRunsDirSegment)
|
||||
}
|
||||
|
||||
// SessionRunRootForCampaign returns the canonical run root under runs/{run_id}.
|
||||
func SessionRunRootForCampaign(rootDir, campaign, sessionID, runID string) string {
|
||||
return filepath.Join(SessionRunsDirForCampaign(rootDir, campaign, sessionID), runID)
|
||||
}
|
||||
|
||||
// SessionRunManifestPathForCampaign returns the canonical run manifest path under runs/{run_id}/manifest.json.
|
||||
func SessionRunManifestPathForCampaign(rootDir, campaign, sessionID, runID string) string {
|
||||
return filepath.Join(SessionRunRootForCampaign(rootDir, campaign, sessionID, runID), config.PathManifestFile)
|
||||
}
|
||||
|
||||
// SessionRunStageDirForCampaign returns the canonical stage directory under runs/{run_id}/{stage}.
|
||||
func SessionRunStageDirForCampaign(rootDir, campaign, sessionID, runID, stageName string) string {
|
||||
return filepath.Join(SessionRunRootForCampaign(rootDir, campaign, sessionID, runID), stageName)
|
||||
}
|
||||
|
||||
// SessionSpoolAudioDir returns the campaign/session/run scoped local spool audio path.
|
||||
func SessionSpoolAudioDir(spoolRoot, campaign, sessionID, runID string) string {
|
||||
return filepath.Join(spoolRoot, campaign, sessionID, runID, "audio")
|
||||
return filepath.Join(spoolRoot, campaign, sessionID, runID, config.PathAudioDirSegment)
|
||||
}
|
||||
|
||||
func buildSessionPaths(workspaceRoot, sessionID string) SessionPaths {
|
||||
root := SessionWorkDir(workspaceRoot, sessionID)
|
||||
transcripts := filepath.Join(root, "transcripts")
|
||||
func buildSessionPaths(workspaceRoot, campaign, sessionID string) SessionPaths {
|
||||
root := SessionWorkDirForCampaign(workspaceRoot, campaign, sessionID)
|
||||
return buildSessionPathsFromRoot(workspaceRoot, campaign, sessionID, root)
|
||||
}
|
||||
|
||||
func buildSessionPathsFromRoot(workspaceRoot, campaign, sessionID, root string) SessionPaths {
|
||||
return SessionPaths{
|
||||
WorkspaceRoot: workspaceRoot,
|
||||
CampaignID: campaign,
|
||||
SessionID: sessionID,
|
||||
Root: root,
|
||||
InputsDir: filepath.Join(root, "inputs"),
|
||||
AudioDir: filepath.Join(root, "audio"),
|
||||
TranscriptsDir: transcripts,
|
||||
TranscriptsRawDir: filepath.Join(transcripts, "raw"),
|
||||
TranscriptsTrimmedDir: filepath.Join(transcripts, "trimmed"),
|
||||
ArtifactsDir: filepath.Join(root, "artifacts"),
|
||||
ConfigDir: filepath.Join(root, "config"),
|
||||
LogsDir: filepath.Join(root, "logs"),
|
||||
ManifestPath: filepath.Join(root, "manifest.json"),
|
||||
LockPath: filepath.Join(root, ".lock"),
|
||||
InputsDir: filepath.Join(root, config.PathInputsDirSegment),
|
||||
AudioDir: filepath.Join(root, config.PathAudioDirSegment),
|
||||
TranscriptsDir: filepath.Join(root, config.PathTranscriptsSegment),
|
||||
TranscriptsRawDir: filepath.Join(root, filepath.FromSlash(config.PathTranscriptsRaw)),
|
||||
TranscriptsTrimmedDir: filepath.Join(root, filepath.FromSlash(config.PathTranscriptsTrimmed)),
|
||||
ArtifactsDir: filepath.Join(root, config.PathArtifactsDirSegment),
|
||||
ReportsDir: filepath.Join(root, config.PathReportsDirSegment),
|
||||
ConfigDir: filepath.Join(root, config.PathConfigDirSegment),
|
||||
LogsDir: filepath.Join(root, config.PathLogsDirSegment),
|
||||
CurrentDir: filepath.Join(root, config.PathCurrentDirSegment),
|
||||
RunsDir: filepath.Join(root, config.PathRunsDirSegment),
|
||||
ManifestPath: filepath.Join(root, config.PathManifestFile),
|
||||
LockPath: filepath.Join(root, config.PathLockFile),
|
||||
}
|
||||
}
|
||||
|
||||
@@ -5,12 +5,47 @@ import (
|
||||
"testing"
|
||||
)
|
||||
|
||||
func TestSessionRunWorkDir(t *testing.T) {
|
||||
func TestSessionWorkDirForCampaign(t *testing.T) {
|
||||
root := "/tmp/workspace"
|
||||
got := SessionRunWorkDir(root, "forsaken", "2026-04-19", "20260515T031522Z-a1b2c3d4")
|
||||
want := filepath.Join(root, "work", "forsaken", "2026-04-19", "20260515T031522Z-a1b2c3d4")
|
||||
got := SessionWorkDirForCampaign(root, "forsaken", "2026-04-19")
|
||||
want := filepath.Join(root, "work", "forsaken", "2026-04-19")
|
||||
if got != want {
|
||||
t.Fatalf("SessionRunWorkDir() = %q, want %q", got, want)
|
||||
t.Fatalf("SessionWorkDirForCampaign() = %q, want %q", got, want)
|
||||
}
|
||||
}
|
||||
|
||||
func TestSessionManifestPathForCampaign(t *testing.T) {
|
||||
root := "/tmp/workspace"
|
||||
got := SessionManifestPathForCampaign(root, "forsaken", "2026-04-19")
|
||||
want := filepath.Join(root, "work", "forsaken", "2026-04-19", "manifest.json")
|
||||
if got != want {
|
||||
t.Fatalf("SessionManifestPathForCampaign() = %q, want %q", got, want)
|
||||
}
|
||||
}
|
||||
|
||||
func TestSessionRunRootAndStageDirForCampaign(t *testing.T) {
|
||||
root := "/tmp/workspace"
|
||||
runID := "20260515T031522Z-a1b2c3d4"
|
||||
runRoot := SessionRunRootForCampaign(root, "forsaken", "2026-04-19", runID)
|
||||
wantRoot := filepath.Join(root, "work", "forsaken", "2026-04-19", "runs", runID)
|
||||
if runRoot != wantRoot {
|
||||
t.Fatalf("SessionRunRootForCampaign() = %q, want %q", runRoot, wantRoot)
|
||||
}
|
||||
|
||||
stageDir := SessionRunStageDirForCampaign(root, "forsaken", "2026-04-19", runID, "transcribe")
|
||||
wantStage := filepath.Join(root, "work", "forsaken", "2026-04-19", "runs", runID, "transcribe")
|
||||
if stageDir != wantStage {
|
||||
t.Fatalf("SessionRunStageDirForCampaign() = %q, want %q", stageDir, wantStage)
|
||||
}
|
||||
}
|
||||
|
||||
func TestSessionRunManifestPathForCampaign(t *testing.T) {
|
||||
root := "/tmp/workspace"
|
||||
runID := "20260515T031522Z-a1b2c3d4"
|
||||
got := SessionRunManifestPathForCampaign(root, "forsaken", "2026-04-19", runID)
|
||||
want := filepath.Join(root, "work", "forsaken", "2026-04-19", "runs", runID, "manifest.json")
|
||||
if got != want {
|
||||
t.Fatalf("SessionRunManifestPathForCampaign() = %q, want %q", got, want)
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
@@ -8,7 +8,7 @@ import (
|
||||
|
||||
func TestResolveSessionLocalPathForRead(t *testing.T) {
|
||||
workspace := t.TempDir()
|
||||
paths := buildSessionPaths(workspace, "s-1")
|
||||
paths := buildSessionPaths(workspace, "sample-campaign", "s-1")
|
||||
if err := os.MkdirAll(paths.TranscriptsRawDir, 0o755); err != nil {
|
||||
t.Fatalf("MkdirAll() error = %v", err)
|
||||
}
|
||||
@@ -41,7 +41,7 @@ func TestResolveSessionLocalPathForReadRelativeWorkspaceRootQualifiedPath(t *tes
|
||||
t.Fatalf("Rel() error = %v", err)
|
||||
}
|
||||
|
||||
paths := buildSessionPaths(workspaceRel, "s-1")
|
||||
paths := buildSessionPaths(workspaceRel, "sample-campaign", "s-1")
|
||||
target := filepath.Join(paths.TranscriptsRawDir, "alice.json")
|
||||
if err := os.MkdirAll(filepath.Dir(target), 0o755); err != nil {
|
||||
t.Fatalf("MkdirAll() error = %v", err)
|
||||
@@ -50,7 +50,7 @@ func TestResolveSessionLocalPathForReadRelativeWorkspaceRootQualifiedPath(t *tes
|
||||
t.Fatalf("WriteFile() error = %v", err)
|
||||
}
|
||||
|
||||
manifestPath := filepath.Join(workspaceRel, "work", "s-1", "transcripts", "raw", "alice.json")
|
||||
manifestPath := filepath.Join(workspaceRel, "work", "sample-campaign", "s-1", "transcripts", "raw", "alice.json")
|
||||
got := ResolveSessionLocalPathForRead(paths, manifestPath)
|
||||
if got != filepath.Clean(manifestPath) {
|
||||
t.Fatalf("resolution = %q, want %q", got, filepath.Clean(manifestPath))
|
||||
|
||||
@@ -11,19 +11,19 @@ type S3Store struct {
|
||||
Prefix string
|
||||
}
|
||||
|
||||
// SessionPaths is not implemented for S3-backed storage.
|
||||
func (s *S3Store) SessionPaths(_ string) SessionPaths {
|
||||
// SessionPathsFor is not implemented for S3-backed storage.
|
||||
func (s *S3Store) SessionPathsFor(_, _ string) SessionPaths {
|
||||
return SessionPaths{}
|
||||
}
|
||||
|
||||
// EnsureLayout returns a not-yet-implemented error in the scaffold.
|
||||
func (s *S3Store) EnsureLayout(_ string) (SessionPaths, error) {
|
||||
return SessionPaths{}, fmt.Errorf("artifacts s3 ensure layout: not yet implemented")
|
||||
// EnsureLayoutFor returns a not-yet-implemented error in the scaffold.
|
||||
func (s *S3Store) EnsureLayoutFor(_, _ string) (SessionPaths, error) {
|
||||
return SessionPaths{}, fmt.Errorf("artifacts s3 ensure layout for campaign/session: not yet implemented")
|
||||
}
|
||||
|
||||
// CopyInput returns a not-yet-implemented error in the scaffold.
|
||||
func (s *S3Store) CopyInput(_, _, _ string) (Ref, error) {
|
||||
return Ref{}, fmt.Errorf("artifacts s3 copy input: not yet implemented")
|
||||
// CopyInputFor returns a not-yet-implemented error in the scaffold.
|
||||
func (s *S3Store) CopyInputFor(_, _, _, _ string) (Ref, error) {
|
||||
return Ref{}, fmt.Errorf("artifacts s3 copy input for campaign/session: not yet implemented")
|
||||
}
|
||||
|
||||
// Exists returns a not-yet-implemented error in the scaffold.
|
||||
@@ -46,9 +46,9 @@ func (s *S3Store) Checksum(_ string) (string, error) {
|
||||
return "", fmt.Errorf("artifacts s3 checksum: not yet implemented")
|
||||
}
|
||||
|
||||
// AcquireSessionLock returns a not-yet-implemented error in the scaffold.
|
||||
func (s *S3Store) AcquireSessionLock(_ string) (*LockHandle, error) {
|
||||
return nil, fmt.Errorf("artifacts s3 acquire lock: not yet implemented")
|
||||
// AcquireSessionLockFor returns a not-yet-implemented error in the scaffold.
|
||||
func (s *S3Store) AcquireSessionLockFor(_, _ string) (*LockHandle, error) {
|
||||
return nil, fmt.Errorf("artifacts s3 acquire lock for campaign/session: not yet implemented")
|
||||
}
|
||||
|
||||
// ReleaseSessionLock returns a not-yet-implemented error in the scaffold.
|
||||
|
||||
@@ -3,6 +3,8 @@ package artifacts
|
||||
import (
|
||||
"path"
|
||||
"strings"
|
||||
|
||||
"gitea.maximumdirect.net/eric/narratio/internal/config"
|
||||
)
|
||||
|
||||
// S3SessionPrefix builds the canonical S3 session prefix.
|
||||
@@ -10,9 +12,9 @@ import (
|
||||
func S3SessionPrefix(rootPrefix, campaign, sessionID string) string {
|
||||
prefix := path.Join(
|
||||
cleanS3PathPart(rootPrefix),
|
||||
"campaigns",
|
||||
config.S3CampaignsSegment,
|
||||
cleanS3PathPart(campaign),
|
||||
"sessions",
|
||||
config.S3SessionsSegment,
|
||||
cleanS3PathPart(sessionID),
|
||||
)
|
||||
return ensureS3TrailingSlash(prefix)
|
||||
@@ -21,7 +23,7 @@ func S3SessionPrefix(rootPrefix, campaign, sessionID string) string {
|
||||
// S3RunPrefix builds the canonical S3 run prefix.
|
||||
// Format: {session_prefix}/runs/{run_id}/
|
||||
func S3RunPrefix(sessionPrefix, runID string) string {
|
||||
prefix := path.Join(strings.TrimSuffix(cleanS3Key(sessionPrefix), "/"), "runs", cleanS3PathPart(runID))
|
||||
prefix := path.Join(strings.TrimSuffix(cleanS3Key(sessionPrefix), "/"), config.S3RunsSegment, cleanS3PathPart(runID))
|
||||
return ensureS3TrailingSlash(prefix)
|
||||
}
|
||||
|
||||
@@ -35,13 +37,13 @@ func S3AudioPrefix(sessionPrefix, audioPrefix string) string {
|
||||
// S3CurrentManifestKey returns the current manifest pointer key.
|
||||
// Format: {session_prefix}/current/manifest.json
|
||||
func S3CurrentManifestKey(sessionPrefix string) string {
|
||||
return path.Join(strings.TrimSuffix(cleanS3Key(sessionPrefix), "/"), "current", "manifest.json")
|
||||
return path.Join(strings.TrimSuffix(cleanS3Key(sessionPrefix), "/"), config.S3CurrentSegment, config.S3ManifestFile)
|
||||
}
|
||||
|
||||
// S3CurrentRunPointerKey returns the current run pointer key.
|
||||
// Format: {session_prefix}/current/run_id.txt
|
||||
func S3CurrentRunPointerKey(sessionPrefix string) string {
|
||||
return path.Join(strings.TrimSuffix(cleanS3Key(sessionPrefix), "/"), "current", "run_id.txt")
|
||||
return path.Join(strings.TrimSuffix(cleanS3Key(sessionPrefix), "/"), config.S3CurrentSegment, config.S3RunIDFile)
|
||||
}
|
||||
|
||||
// S3PromotedArtifactKey returns the destination key for one promoted artifact.
|
||||
|
||||
@@ -15,13 +15,13 @@ type Ref struct {
|
||||
|
||||
// Store is the local artifact/workdir abstraction used by orchestration code.
|
||||
type Store interface {
|
||||
SessionPaths(sessionID string) SessionPaths
|
||||
EnsureLayout(sessionID string) (SessionPaths, error)
|
||||
CopyInput(sessionID, srcPath, destRelativePath string) (Ref, error)
|
||||
SessionPathsFor(campaign, sessionID string) SessionPaths
|
||||
EnsureLayoutFor(campaign, sessionID string) (SessionPaths, error)
|
||||
CopyInputFor(campaign, sessionID, srcPath, destRelativePath string) (Ref, error)
|
||||
Exists(path string) (bool, error)
|
||||
ExistsRef(ref Ref) (bool, error)
|
||||
WriteFileAtomic(path string, data []byte, perm os.FileMode) error
|
||||
Checksum(path string) (string, error)
|
||||
AcquireSessionLock(sessionID string) (*LockHandle, error)
|
||||
AcquireSessionLockFor(campaign, sessionID string) (*LockHandle, error)
|
||||
ReleaseSessionLock(lock *LockHandle) error
|
||||
}
|
||||
|
||||
@@ -60,6 +60,8 @@ type StorageS3Config struct {
|
||||
Region string `yaml:"region"`
|
||||
Endpoint string `yaml:"endpoint"`
|
||||
ForcePathStyle bool `yaml:"force_path_style"`
|
||||
AccessKeyIDEnv string `yaml:"access_key_id_env"`
|
||||
SecretKeyEnv string `yaml:"secret_access_key_env"`
|
||||
}
|
||||
|
||||
// SpoolConfig configures local spool storage for staged data.
|
||||
|
||||
@@ -8,8 +8,74 @@ const (
|
||||
DefaultSessionConfigPathLocal = "./session.yml"
|
||||
DefaultSessionConfigPathUsrLocal = "/usr/local/etc/narratio/session.yml"
|
||||
DefaultSessionConfigPathEtc = "/etc/narratio/session.yml"
|
||||
DefaultS3AccessKeyIDEnv = "OBJECT_STORAGE_KEY_ID"
|
||||
DefaultS3SecretAccessKeyEnv = "OBJECT_STORAGE_KEY"
|
||||
DefaultStorageS3RootPrefix = "dnd"
|
||||
DefaultSpoolRoot = "/var/spool/narratio"
|
||||
|
||||
DefaultWhisperXLanguage = "en"
|
||||
DefaultWhisperXTimeout = "30m"
|
||||
DefaultWhisperXRetryDelay = "2s"
|
||||
DefaultWhisperXConcurrency = 2
|
||||
DefaultWhisperXRetries = 3
|
||||
|
||||
DefaultSeriatimBinary = "seriatim"
|
||||
DefaultSeriatimTimeout = "10m"
|
||||
DefaultSeriatimOutputSchema = "seriatim-intermediate"
|
||||
DefaultSeriatimCoalesceGap = 3.0
|
||||
DefaultSeriatimReport = true
|
||||
|
||||
DefaultAuditaBinary = "audita"
|
||||
DefaultAuditaTimeout = "3h"
|
||||
DefaultAuditaReport = true
|
||||
|
||||
DefaultScriptoriumBinary = "scriptorium"
|
||||
DefaultScriptoriumTimeout = "10m"
|
||||
|
||||
DefaultTrimBoundsTimeout = "10m"
|
||||
DefaultTrimSeriatimReport = false
|
||||
|
||||
DefaultNormalizeOutputPath = "transcripts/normalized.json"
|
||||
DefaultNormalizeOutputSchema = "seriatim-intermediate"
|
||||
DefaultNormalizeReport = true
|
||||
|
||||
DefaultArchiveEnabled = true
|
||||
DefaultArchiveUploadRun = true
|
||||
|
||||
PathWorkDirSegment = "work"
|
||||
PathInputsDirSegment = "inputs"
|
||||
PathAudioDirSegment = "audio"
|
||||
PathTranscriptsSegment = "transcripts"
|
||||
PathTranscriptsRaw = "transcripts/raw"
|
||||
PathTranscriptsTrimmed = "transcripts/trimmed"
|
||||
PathArtifactsDirSegment = "artifacts"
|
||||
PathReportsDirSegment = "reports"
|
||||
PathConfigDirSegment = "config"
|
||||
PathLogsDirSegment = "logs"
|
||||
PathCurrentDirSegment = "current"
|
||||
PathRunsDirSegment = "runs"
|
||||
PathManifestFile = "manifest.json"
|
||||
PathLockFile = ".lock"
|
||||
PathTranscriptMerged = "transcripts/merged.json"
|
||||
PathTranscriptProcessed = "transcripts/processed.json"
|
||||
PathTranscriptNormalized = "transcripts/normalized.json"
|
||||
PathTranscriptTrimmed = "transcripts/trimmed.json"
|
||||
|
||||
S3CampaignsSegment = "campaigns"
|
||||
S3SessionsSegment = "sessions"
|
||||
S3RunsSegment = "runs"
|
||||
S3CurrentSegment = "current"
|
||||
S3ManifestFile = "manifest.json"
|
||||
S3RunIDFile = "run_id.txt"
|
||||
)
|
||||
|
||||
// DefaultArchivePromoteArtifacts defines the default archive promotion rules.
|
||||
// Callers should copy this slice before mutating.
|
||||
var DefaultArchivePromoteArtifacts = []ArchivePromotionRule{
|
||||
{From: PathTranscriptTrimmed, To: PathTranscriptTrimmed},
|
||||
{From: "artifacts/session_recap.md", To: "artifacts/session_recap.md"},
|
||||
}
|
||||
|
||||
// DefaultPipelineConfigSearchPaths defines the default search order for
|
||||
// pipeline.yml when callers do not provide an explicit path.
|
||||
//
|
||||
|
||||
@@ -174,7 +174,13 @@ func applyStorageDefaults(cfg *StorageConfig) {
|
||||
cfg.S3 = &StorageS3Config{}
|
||||
}
|
||||
if cfg.S3.RootPrefix == "" {
|
||||
cfg.S3.RootPrefix = "dnd"
|
||||
cfg.S3.RootPrefix = DefaultStorageS3RootPrefix
|
||||
}
|
||||
if cfg.S3.AccessKeyIDEnv == "" {
|
||||
cfg.S3.AccessKeyIDEnv = DefaultS3AccessKeyIDEnv
|
||||
}
|
||||
if cfg.S3.SecretKeyEnv == "" {
|
||||
cfg.S3.SecretKeyEnv = DefaultS3SecretAccessKeyEnv
|
||||
}
|
||||
}
|
||||
|
||||
@@ -183,7 +189,7 @@ func applySpoolDefaults(cfg *SpoolConfig) {
|
||||
return
|
||||
}
|
||||
if cfg.Root == "" {
|
||||
cfg.Root = "/var/spool/narratio"
|
||||
cfg.Root = DefaultSpoolRoot
|
||||
}
|
||||
}
|
||||
|
||||
@@ -196,16 +202,13 @@ func applyArchiveDefaults(cfg **ArchiveConfig) {
|
||||
}
|
||||
|
||||
if (*cfg).Enabled == nil {
|
||||
(*cfg).Enabled = boolPtr(true)
|
||||
(*cfg).Enabled = boolPtr(DefaultArchiveEnabled)
|
||||
}
|
||||
if (*cfg).UploadRun == nil {
|
||||
(*cfg).UploadRun = boolPtr(true)
|
||||
(*cfg).UploadRun = boolPtr(DefaultArchiveUploadRun)
|
||||
}
|
||||
if len((*cfg).PromoteArtifacts) == 0 {
|
||||
(*cfg).PromoteArtifacts = []ArchivePromotionRule{
|
||||
{From: "transcripts/trimmed.json", To: "transcripts/trimmed.json", Required: boolPtr(true)},
|
||||
{From: "artifacts/session_recap.md", To: "artifacts/session_recap.md", Required: boolPtr(true)},
|
||||
}
|
||||
(*cfg).PromoteArtifacts = append([]ArchivePromotionRule(nil), DefaultArchivePromoteArtifacts...)
|
||||
}
|
||||
for i := range (*cfg).PromoteArtifacts {
|
||||
if (*cfg).PromoteArtifacts[i].Required == nil {
|
||||
@@ -219,19 +222,19 @@ func applyWhisperXDefaults(cfg *WhisperXConfig) {
|
||||
return
|
||||
}
|
||||
if cfg.Language == "" {
|
||||
cfg.Language = "en"
|
||||
cfg.Language = DefaultWhisperXLanguage
|
||||
}
|
||||
if cfg.Timeout == "" {
|
||||
cfg.Timeout = "30m"
|
||||
cfg.Timeout = DefaultWhisperXTimeout
|
||||
}
|
||||
if cfg.RetryDelay == "" {
|
||||
cfg.RetryDelay = "2s"
|
||||
cfg.RetryDelay = DefaultWhisperXRetryDelay
|
||||
}
|
||||
if cfg.Concurrency == nil {
|
||||
cfg.Concurrency = intPtr(2)
|
||||
cfg.Concurrency = intPtr(DefaultWhisperXConcurrency)
|
||||
}
|
||||
if cfg.Retries == nil {
|
||||
cfg.Retries = intPtr(3)
|
||||
cfg.Retries = intPtr(DefaultWhisperXRetries)
|
||||
}
|
||||
}
|
||||
|
||||
@@ -245,19 +248,19 @@ func applySeriatimDefaults(cfg *SeriatimConfig) {
|
||||
return
|
||||
}
|
||||
if cfg.Binary == "" {
|
||||
cfg.Binary = "seriatim"
|
||||
cfg.Binary = DefaultSeriatimBinary
|
||||
}
|
||||
if cfg.Timeout == "" {
|
||||
cfg.Timeout = "10m"
|
||||
cfg.Timeout = DefaultSeriatimTimeout
|
||||
}
|
||||
if cfg.OutputSchema == "" {
|
||||
cfg.OutputSchema = "seriatim-intermediate"
|
||||
cfg.OutputSchema = DefaultSeriatimOutputSchema
|
||||
}
|
||||
if cfg.CoalesceGap == nil {
|
||||
cfg.CoalesceGap = float64Ptr(3.0)
|
||||
cfg.CoalesceGap = float64Ptr(DefaultSeriatimCoalesceGap)
|
||||
}
|
||||
if cfg.Report == nil {
|
||||
cfg.Report = boolPtr(true)
|
||||
cfg.Report = boolPtr(DefaultSeriatimReport)
|
||||
}
|
||||
}
|
||||
|
||||
@@ -266,13 +269,13 @@ func applyAuditaDefaults(cfg *AuditaConfig) {
|
||||
return
|
||||
}
|
||||
if cfg.Binary == "" {
|
||||
cfg.Binary = "audita"
|
||||
cfg.Binary = DefaultAuditaBinary
|
||||
}
|
||||
if cfg.Timeout == "" {
|
||||
cfg.Timeout = "3h"
|
||||
cfg.Timeout = DefaultAuditaTimeout
|
||||
}
|
||||
if cfg.Report == nil {
|
||||
cfg.Report = boolPtr(true)
|
||||
cfg.Report = boolPtr(DefaultAuditaReport)
|
||||
}
|
||||
}
|
||||
|
||||
@@ -281,10 +284,10 @@ func applyScriptoriumDefaults(cfg *ScriptoriumConfig) {
|
||||
return
|
||||
}
|
||||
if cfg.Binary == "" {
|
||||
cfg.Binary = "scriptorium"
|
||||
cfg.Binary = DefaultScriptoriumBinary
|
||||
}
|
||||
if cfg.Timeout == "" {
|
||||
cfg.Timeout = "10m"
|
||||
cfg.Timeout = DefaultScriptoriumTimeout
|
||||
}
|
||||
}
|
||||
|
||||
@@ -293,10 +296,10 @@ func applyTrimDefaults(cfg *TrimConfig) {
|
||||
return
|
||||
}
|
||||
if cfg.Bounds.Timeout == "" {
|
||||
cfg.Bounds.Timeout = "10m"
|
||||
cfg.Bounds.Timeout = DefaultTrimBoundsTimeout
|
||||
}
|
||||
if cfg.Seriatim.Report == nil {
|
||||
cfg.Seriatim.Report = boolPtr(false)
|
||||
cfg.Seriatim.Report = boolPtr(DefaultTrimSeriatimReport)
|
||||
}
|
||||
}
|
||||
|
||||
@@ -305,13 +308,13 @@ func applyNormalizeDefaults(cfg *NormalizeConfig) {
|
||||
return
|
||||
}
|
||||
if cfg.OutputSchema == "" {
|
||||
cfg.OutputSchema = defaultNormalizeOutputSchema
|
||||
cfg.OutputSchema = DefaultNormalizeOutputSchema
|
||||
}
|
||||
if cfg.OutputPath == "" && !cfg.outputPathWasSet() {
|
||||
cfg.OutputPath = defaultNormalizeOutputPath
|
||||
cfg.OutputPath = DefaultNormalizeOutputPath
|
||||
}
|
||||
if cfg.Report == nil {
|
||||
cfg.Report = boolPtr(true)
|
||||
cfg.Report = boolPtr(DefaultNormalizeReport)
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
@@ -6,11 +6,6 @@ import (
|
||||
"gopkg.in/yaml.v3"
|
||||
)
|
||||
|
||||
const (
|
||||
defaultNormalizeOutputPath = "transcripts/normalized.json"
|
||||
defaultNormalizeOutputSchema = "seriatim-intermediate"
|
||||
)
|
||||
|
||||
// UnmarshalYAML tracks explicit normalize.output_path presence so validation can
|
||||
// distinguish omitted vs explicitly empty values.
|
||||
func (cfg *NormalizeConfig) UnmarshalYAML(node *yaml.Node) error {
|
||||
|
||||
@@ -116,6 +116,37 @@ func TestScriptoriumLoadAndValidate(t *testing.T) {
|
||||
output_kind: session_recap
|
||||
`,
|
||||
},
|
||||
{
|
||||
name: "canonical artifact source is accepted",
|
||||
scriptoriumYAML: `scriptorium:
|
||||
binary: scriptorium
|
||||
artifacts:
|
||||
session_recap:
|
||||
enabled: true
|
||||
prompt_id: dnd.session_recap
|
||||
output_path: artifacts/session_recap.md
|
||||
inputs:
|
||||
transcript:
|
||||
source: narratio.transcript.trimmed
|
||||
required: true
|
||||
`,
|
||||
},
|
||||
{
|
||||
name: "unknown artifact source fails validation",
|
||||
scriptoriumYAML: `scriptorium:
|
||||
binary: scriptorium
|
||||
artifacts:
|
||||
session_recap:
|
||||
enabled: true
|
||||
prompt_id: dnd.session_recap
|
||||
output_path: artifacts/session_recap.md
|
||||
inputs:
|
||||
transcript:
|
||||
source: narratio.unknown
|
||||
required: true
|
||||
`,
|
||||
wantValidateErr: `pipeline.scriptorium.artifacts.session_recap.inputs.transcript.source "narratio.unknown" is unsupported`,
|
||||
},
|
||||
{
|
||||
name: "artifact render_debug override is accepted",
|
||||
scriptoriumYAML: `scriptorium:
|
||||
|
||||
@@ -24,6 +24,12 @@ storage:
|
||||
if cfg.Pipeline.Storage.S3.RootPrefix != "dnd" {
|
||||
t.Fatalf("storage.s3.root_prefix = %q, want dnd", cfg.Pipeline.Storage.S3.RootPrefix)
|
||||
}
|
||||
if cfg.Pipeline.Storage.S3.AccessKeyIDEnv != DefaultS3AccessKeyIDEnv {
|
||||
t.Fatalf("storage.s3.access_key_id_env = %q, want %q", cfg.Pipeline.Storage.S3.AccessKeyIDEnv, DefaultS3AccessKeyIDEnv)
|
||||
}
|
||||
if cfg.Pipeline.Storage.S3.SecretKeyEnv != DefaultS3SecretAccessKeyEnv {
|
||||
t.Fatalf("storage.s3.secret_access_key_env = %q, want %q", cfg.Pipeline.Storage.S3.SecretKeyEnv, DefaultS3SecretAccessKeyEnv)
|
||||
}
|
||||
if cfg.Pipeline.Storage.S3.ForcePathStyle {
|
||||
t.Fatalf("storage.s3.force_path_style = true, want false default")
|
||||
}
|
||||
@@ -33,6 +39,77 @@ storage:
|
||||
}
|
||||
}
|
||||
|
||||
func TestStorageS3CredentialEnvNamesLoadAndValidate(t *testing.T) {
|
||||
pipelineYAML := testPipelineBaseYAML + `
|
||||
storage:
|
||||
backend: s3
|
||||
s3:
|
||||
bucket: my-dnd-archive
|
||||
access_key_id_env: CUSTOM_KEY_ID
|
||||
secret_access_key_env: CUSTOM_SECRET
|
||||
`
|
||||
pipelinePath, sessionPath := writeConfigFiles(t, pipelineYAML, testSessionBaseYAML)
|
||||
|
||||
cfg, err := Load(pipelinePath, sessionPath)
|
||||
if err != nil {
|
||||
t.Fatalf("Load() error = %v", err)
|
||||
}
|
||||
if cfg.Pipeline.Storage.S3.AccessKeyIDEnv != "CUSTOM_KEY_ID" {
|
||||
t.Fatalf("storage.s3.access_key_id_env = %q, want CUSTOM_KEY_ID", cfg.Pipeline.Storage.S3.AccessKeyIDEnv)
|
||||
}
|
||||
if cfg.Pipeline.Storage.S3.SecretKeyEnv != "CUSTOM_SECRET" {
|
||||
t.Fatalf("storage.s3.secret_access_key_env = %q, want CUSTOM_SECRET", cfg.Pipeline.Storage.S3.SecretKeyEnv)
|
||||
}
|
||||
if err := Validate(cfg); err != nil {
|
||||
t.Fatalf("Validate() error = %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestStorageS3CredentialEnvValidation(t *testing.T) {
|
||||
tests := []struct {
|
||||
name string
|
||||
pipelineYML string
|
||||
wantErr string
|
||||
}{
|
||||
{
|
||||
name: "invalid access key env name",
|
||||
pipelineYML: testPipelineBaseYAML + `
|
||||
storage:
|
||||
backend: s3
|
||||
s3:
|
||||
bucket: my-dnd-archive
|
||||
access_key_id_env: "123BAD"
|
||||
`,
|
||||
wantErr: "pipeline.storage.s3.access_key_id_env must be a valid environment variable name",
|
||||
},
|
||||
{
|
||||
name: "invalid secret key env name",
|
||||
pipelineYML: testPipelineBaseYAML + `
|
||||
storage:
|
||||
backend: s3
|
||||
s3:
|
||||
bucket: my-dnd-archive
|
||||
secret_access_key_env: "bad-name"
|
||||
`,
|
||||
wantErr: "pipeline.storage.s3.secret_access_key_env must be a valid environment variable name",
|
||||
},
|
||||
}
|
||||
|
||||
for _, tt := range tests {
|
||||
t.Run(tt.name, func(t *testing.T) {
|
||||
pipelinePath, sessionPath := writeConfigFiles(t, tt.pipelineYML, testSessionBaseYAML)
|
||||
cfg, err := Load(pipelinePath, sessionPath)
|
||||
if err != nil {
|
||||
t.Fatalf("Load() error = %v", err)
|
||||
}
|
||||
err = Validate(cfg)
|
||||
if err == nil || !strings.Contains(err.Error(), tt.wantErr) {
|
||||
t.Fatalf("Validate() error = %v, want to contain %q", err, tt.wantErr)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestSpoolAndArchiveDefaults(t *testing.T) {
|
||||
pipelinePath, sessionPath := writeConfigFiles(t, testPipelineBaseYAML, testSessionBaseYAML)
|
||||
|
||||
|
||||
@@ -91,6 +91,12 @@ func validateStorage(cfg StorageConfig) error {
|
||||
if cfg.S3.Endpoint != "" && strings.TrimSpace(cfg.S3.Endpoint) == "" {
|
||||
return fmt.Errorf("pipeline.storage.s3.endpoint must be non-empty when provided")
|
||||
}
|
||||
if err := validateEnvVarNameField("pipeline.storage.s3.access_key_id_env", cfg.S3.AccessKeyIDEnv); err != nil {
|
||||
return err
|
||||
}
|
||||
if err := validateEnvVarNameField("pipeline.storage.s3.secret_access_key_env", cfg.S3.SecretKeyEnv); err != nil {
|
||||
return err
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
@@ -337,9 +343,13 @@ func validateScriptorium(cfg *ScriptoriumConfig) error {
|
||||
if trimmedInputName == "" {
|
||||
return fmt.Errorf("pipeline.scriptorium.artifacts.%s.inputs keys must be non-empty", artifactName)
|
||||
}
|
||||
if strings.TrimSpace(inputCfg.Source) == "" {
|
||||
source := strings.TrimSpace(inputCfg.Source)
|
||||
if source == "" {
|
||||
return fmt.Errorf("pipeline.scriptorium.artifacts.%s.inputs.%s.source is required", artifactName, inputName)
|
||||
}
|
||||
if !isSupportedScriptoriumInputSource(source) {
|
||||
return fmt.Errorf("pipeline.scriptorium.artifacts.%s.inputs.%s.source %q is unsupported", artifactName, inputName, inputCfg.Source)
|
||||
}
|
||||
}
|
||||
for varName, varValue := range artifactCfg.Vars {
|
||||
if strings.TrimSpace(varName) == "" {
|
||||
@@ -429,7 +439,46 @@ func archiveUploadConfiguredForS3(pipeline *PipelineConfig) bool {
|
||||
return enabled && upload
|
||||
}
|
||||
|
||||
func isSupportedScriptoriumInputSource(source string) bool {
|
||||
switch strings.TrimSpace(source) {
|
||||
case "previous_session_artifact":
|
||||
return true
|
||||
case "processed_transcript":
|
||||
return true
|
||||
case "normalized_transcript":
|
||||
return true
|
||||
case "trimmed_transcript":
|
||||
return true
|
||||
case "narratio.transcript.merged":
|
||||
return true
|
||||
case "narratio.transcript.polished":
|
||||
return true
|
||||
case "narratio.transcript.full":
|
||||
return true
|
||||
case "narratio.transcript.trimmed":
|
||||
return true
|
||||
case "narratio.bounds.session":
|
||||
return true
|
||||
case "narratio.artifact.session_recap":
|
||||
return true
|
||||
default:
|
||||
return false
|
||||
}
|
||||
}
|
||||
|
||||
var windowsAbsPathRE = regexp.MustCompile(`^[A-Za-z]:[\\/].*`)
|
||||
var envVarNameRE = regexp.MustCompile(`^[A-Za-z_][A-Za-z0-9_]*$`)
|
||||
|
||||
func validateEnvVarNameField(fieldName, value string) error {
|
||||
trimmed := strings.TrimSpace(value)
|
||||
if trimmed == "" {
|
||||
return fmt.Errorf("%s must be non-empty", fieldName)
|
||||
}
|
||||
if !envVarNameRE.MatchString(trimmed) {
|
||||
return fmt.Errorf("%s must be a valid environment variable name", fieldName)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func validateRelativeSafePath(fieldName, value string) error {
|
||||
trimmed := strings.TrimSpace(value)
|
||||
|
||||
@@ -29,8 +29,10 @@ type InputRecord struct {
|
||||
type ArtifactRecord struct {
|
||||
Kind string `json:"kind"`
|
||||
LocalPath string `json:"local_path"`
|
||||
RemoteKey string `json:"remote_key,omitempty"`
|
||||
Checksum string `json:"checksum,omitempty"`
|
||||
// ProducerRunID identifies the run that produced this durable artifact.
|
||||
ProducerRunID string `json:"producer_run_id,omitempty"`
|
||||
RemoteKey string `json:"remote_key,omitempty"`
|
||||
Checksum string `json:"checksum,omitempty"`
|
||||
}
|
||||
|
||||
// StageRecord tracks lifecycle and provenance for one pipeline stage.
|
||||
@@ -120,6 +122,15 @@ func (m *Manifest) MarkStageSkipped(name string, at time.Time, reason string) {
|
||||
m.UpdatedAt = at
|
||||
}
|
||||
|
||||
// MarkStageStale marks a stage as stale so it is not skipped as idempotently complete.
|
||||
func (m *Manifest) MarkStageStale(name string, at time.Time, reason string) {
|
||||
s := m.ensureStage(name, at)
|
||||
s.Status = StatusStale
|
||||
s.Error = &ErrorRecord{Message: strings.TrimSpace(reason), Code: "stale", At: timePtr(at)}
|
||||
s.UpdatedAt = at
|
||||
m.UpdatedAt = at
|
||||
}
|
||||
|
||||
func (m *Manifest) ensureStage(name string, at time.Time) *StageRecord {
|
||||
if m.Stages == nil {
|
||||
m.Stages = map[string]*StageRecord{}
|
||||
|
||||
164
internal/manifest/run_manifest.go
Normal file
164
internal/manifest/run_manifest.go
Normal file
@@ -0,0 +1,164 @@
|
||||
package manifest
|
||||
|
||||
import (
|
||||
"strings"
|
||||
"time"
|
||||
)
|
||||
|
||||
type RunManifestStatus string
|
||||
|
||||
const (
|
||||
RunManifestStatusRunning RunManifestStatus = "running"
|
||||
RunManifestStatusSucceeded RunManifestStatus = "succeeded"
|
||||
RunManifestStatusFailed RunManifestStatus = "failed"
|
||||
)
|
||||
|
||||
type RunStageAction string
|
||||
|
||||
const (
|
||||
RunStageActionRun RunStageAction = "run"
|
||||
RunStageActionSkip RunStageAction = "skip"
|
||||
)
|
||||
|
||||
// RunStageRecord tracks lifecycle and provenance for one stage within a single invocation.
|
||||
type RunStageRecord struct {
|
||||
Name string `json:"name"`
|
||||
Action RunStageAction `json:"action"`
|
||||
Status StageStatus `json:"status"`
|
||||
CreatedAt time.Time `json:"created_at"`
|
||||
UpdatedAt time.Time `json:"updated_at"`
|
||||
StartedAt *time.Time `json:"started_at,omitempty"`
|
||||
CompletedAt *time.Time `json:"completed_at,omitempty"`
|
||||
Outputs []ArtifactRecord `json:"outputs,omitempty"`
|
||||
Logs []string `json:"logs,omitempty"`
|
||||
GeneratedConfigs []string `json:"generated_configs,omitempty"`
|
||||
Error *ErrorRecord `json:"error,omitempty"`
|
||||
Metadata map[string]any `json:"metadata,omitempty"`
|
||||
}
|
||||
|
||||
// RunManifest is the invocation-scoped execution record under runs/{run_id}/manifest.json.
|
||||
type RunManifest struct {
|
||||
SessionID string `json:"session_id"`
|
||||
Campaign string `json:"campaign,omitempty"`
|
||||
RunID string `json:"run_id"`
|
||||
Force bool `json:"force"`
|
||||
RequestedStages []string `json:"requested_stages,omitempty"`
|
||||
SessionManifestPath string `json:"session_manifest_path,omitempty"`
|
||||
LocalWorkDir string `json:"local_workdir,omitempty"`
|
||||
LocalSpoolDir string `json:"local_spool_dir,omitempty"`
|
||||
S3Bucket string `json:"s3_bucket,omitempty"`
|
||||
S3SessionPrefix string `json:"s3_session_prefix,omitempty"`
|
||||
S3RunPrefix string `json:"s3_run_prefix,omitempty"`
|
||||
CreatedAt time.Time `json:"created_at"`
|
||||
UpdatedAt time.Time `json:"updated_at"`
|
||||
StartedAt *time.Time `json:"started_at,omitempty"`
|
||||
CompletedAt *time.Time `json:"completed_at,omitempty"`
|
||||
Status RunManifestStatus `json:"status"`
|
||||
LastError *ErrorRecord `json:"last_error,omitempty"`
|
||||
Stages map[string]*RunStageRecord `json:"stages"`
|
||||
Metadata map[string]any `json:"metadata,omitempty"`
|
||||
}
|
||||
|
||||
// NewRun constructs a new run manifest with deterministic timestamps.
|
||||
func NewRun(sessionID, campaign, runID string, force bool, requestedStages []string, now time.Time) *RunManifest {
|
||||
return &RunManifest{
|
||||
SessionID: strings.TrimSpace(sessionID),
|
||||
Campaign: strings.TrimSpace(campaign),
|
||||
RunID: strings.TrimSpace(runID),
|
||||
Force: force,
|
||||
RequestedStages: append([]string(nil), requestedStages...),
|
||||
CreatedAt: now,
|
||||
UpdatedAt: now,
|
||||
StartedAt: timePtr(now),
|
||||
Status: RunManifestStatusRunning,
|
||||
Stages: map[string]*RunStageRecord{},
|
||||
}
|
||||
}
|
||||
|
||||
func (m *RunManifest) SetStageAction(name string, action RunStageAction, at time.Time) {
|
||||
s := m.ensureStage(name, at)
|
||||
s.Action = action
|
||||
s.UpdatedAt = at
|
||||
m.UpdatedAt = at
|
||||
}
|
||||
|
||||
func (m *RunManifest) MarkStageRunning(name string, at time.Time) {
|
||||
s := m.ensureStage(name, at)
|
||||
s.Status = StatusRunning
|
||||
s.StartedAt = timePtr(at)
|
||||
s.CompletedAt = nil
|
||||
s.Error = nil
|
||||
s.UpdatedAt = at
|
||||
m.UpdatedAt = at
|
||||
}
|
||||
|
||||
func (m *RunManifest) MarkStageSucceeded(name string, at time.Time, outputs []ArtifactRecord) {
|
||||
s := m.ensureStage(name, at)
|
||||
s.Status = StatusSucceeded
|
||||
s.CompletedAt = timePtr(at)
|
||||
s.Error = nil
|
||||
s.Outputs = append([]ArtifactRecord(nil), outputs...)
|
||||
s.UpdatedAt = at
|
||||
m.UpdatedAt = at
|
||||
}
|
||||
|
||||
func (m *RunManifest) MarkStageFailed(name string, at time.Time, message string) {
|
||||
s := m.ensureStage(name, at)
|
||||
s.Status = StatusFailed
|
||||
s.CompletedAt = timePtr(at)
|
||||
s.Error = &ErrorRecord{Message: strings.TrimSpace(message), At: timePtr(at)}
|
||||
s.UpdatedAt = at
|
||||
m.LastError = &ErrorRecord{Message: strings.TrimSpace(message), At: timePtr(at)}
|
||||
m.UpdatedAt = at
|
||||
m.Status = RunManifestStatusFailed
|
||||
m.CompletedAt = timePtr(at)
|
||||
}
|
||||
|
||||
func (m *RunManifest) MarkStageSkipped(name string, at time.Time, reason string) {
|
||||
s := m.ensureStage(name, at)
|
||||
s.Status = StatusSkipped
|
||||
s.CompletedAt = timePtr(at)
|
||||
s.Error = &ErrorRecord{Message: strings.TrimSpace(reason), Code: "skipped", At: timePtr(at)}
|
||||
s.UpdatedAt = at
|
||||
m.UpdatedAt = at
|
||||
}
|
||||
|
||||
func (m *RunManifest) MarkSucceeded(at time.Time) {
|
||||
m.Status = RunManifestStatusSucceeded
|
||||
m.CompletedAt = timePtr(at)
|
||||
m.UpdatedAt = at
|
||||
}
|
||||
|
||||
func (m *RunManifest) MarkFailed(at time.Time, message string) {
|
||||
m.Status = RunManifestStatusFailed
|
||||
m.CompletedAt = timePtr(at)
|
||||
m.LastError = &ErrorRecord{Message: strings.TrimSpace(message), At: timePtr(at)}
|
||||
m.UpdatedAt = at
|
||||
}
|
||||
|
||||
func (m *RunManifest) ensureStage(name string, at time.Time) *RunStageRecord {
|
||||
if m.Stages == nil {
|
||||
m.Stages = map[string]*RunStageRecord{}
|
||||
}
|
||||
|
||||
stageName := strings.TrimSpace(name)
|
||||
s, ok := m.Stages[stageName]
|
||||
if !ok || s == nil {
|
||||
s = &RunStageRecord{
|
||||
Name: stageName,
|
||||
Action: RunStageActionRun,
|
||||
Status: StatusPending,
|
||||
CreatedAt: at,
|
||||
UpdatedAt: at,
|
||||
}
|
||||
m.Stages[stageName] = s
|
||||
}
|
||||
if s.Name == "" {
|
||||
s.Name = stageName
|
||||
}
|
||||
if s.CreatedAt.IsZero() {
|
||||
s.CreatedAt = at
|
||||
}
|
||||
|
||||
return s
|
||||
}
|
||||
50
internal/manifest/run_manifest_test.go
Normal file
50
internal/manifest/run_manifest_test.go
Normal file
@@ -0,0 +1,50 @@
|
||||
package manifest
|
||||
|
||||
import (
|
||||
"testing"
|
||||
"time"
|
||||
)
|
||||
|
||||
func TestRunManifestStageMarkHelpers(t *testing.T) {
|
||||
rm := NewRun("2026-05-03", "forsaken", "20260517T000000Z-abcdef12", false, []string{"prepare"}, time.Date(2026, 5, 3, 10, 0, 0, 0, time.UTC))
|
||||
if rm.Status != RunManifestStatusRunning {
|
||||
t.Fatalf("status = %q, want %q", rm.Status, RunManifestStatusRunning)
|
||||
}
|
||||
|
||||
runningAt := time.Date(2026, 5, 3, 10, 1, 0, 0, time.UTC)
|
||||
rm.SetStageAction("prepare", RunStageActionRun, runningAt)
|
||||
rm.MarkStageRunning("prepare", runningAt)
|
||||
rm.MarkStageSucceeded("prepare", runningAt.Add(30*time.Second), []ArtifactRecord{
|
||||
{Kind: "input", LocalPath: "inputs/session.yml"},
|
||||
})
|
||||
|
||||
stage := rm.Stages["prepare"]
|
||||
if stage == nil {
|
||||
t.Fatal("prepare stage missing")
|
||||
}
|
||||
if stage.Action != RunStageActionRun {
|
||||
t.Fatalf("action = %q, want %q", stage.Action, RunStageActionRun)
|
||||
}
|
||||
if stage.Status != StatusSucceeded {
|
||||
t.Fatalf("status = %q, want %q", stage.Status, StatusSucceeded)
|
||||
}
|
||||
|
||||
skippedAt := runningAt.Add(1 * time.Minute)
|
||||
rm.SetStageAction("notify", RunStageActionSkip, skippedAt)
|
||||
rm.MarkStageSkipped("notify", skippedAt, "already_succeeded")
|
||||
skipped := rm.Stages["notify"]
|
||||
if skipped == nil {
|
||||
t.Fatal("notify stage missing")
|
||||
}
|
||||
if skipped.Action != RunStageActionSkip {
|
||||
t.Fatalf("action = %q, want %q", skipped.Action, RunStageActionSkip)
|
||||
}
|
||||
if skipped.Status != StatusSkipped {
|
||||
t.Fatalf("status = %q, want %q", skipped.Status, StatusSkipped)
|
||||
}
|
||||
|
||||
rm.MarkSucceeded(skippedAt.Add(10 * time.Second))
|
||||
if rm.Status != RunManifestStatusSucceeded {
|
||||
t.Fatalf("status = %q, want %q", rm.Status, RunManifestStatusSucceeded)
|
||||
}
|
||||
}
|
||||
@@ -129,6 +129,89 @@ func (s *LocalStore) Save(ctx context.Context, path string, m *Manifest) error {
|
||||
return nil
|
||||
}
|
||||
|
||||
// CreateRun returns a new in-memory run manifest for one invocation.
|
||||
func (s *LocalStore) CreateRun(
|
||||
ctx context.Context,
|
||||
sessionID, campaign, runID string,
|
||||
force bool,
|
||||
requestedStages []string,
|
||||
) (*RunManifest, error) {
|
||||
if err := checkContext(ctx); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if strings.TrimSpace(sessionID) == "" {
|
||||
return nil, fmt.Errorf("create run manifest: session_id is required")
|
||||
}
|
||||
if strings.TrimSpace(runID) == "" {
|
||||
return nil, fmt.Errorf("create run manifest: run_id is required")
|
||||
}
|
||||
|
||||
now := time.Now().UTC()
|
||||
return NewRun(sessionID, campaign, runID, force, requestedStages, now), nil
|
||||
}
|
||||
|
||||
// LoadRun reads and validates a local JSON run manifest from path.
|
||||
func (s *LocalStore) LoadRun(ctx context.Context, path string) (*RunManifest, error) {
|
||||
if err := checkContext(ctx); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if strings.TrimSpace(path) == "" {
|
||||
return nil, fmt.Errorf("load run manifest: path is required")
|
||||
}
|
||||
|
||||
data, err := os.ReadFile(path)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("load run manifest %q: %w", path, err)
|
||||
}
|
||||
|
||||
var m RunManifest
|
||||
if err := json.Unmarshal(data, &m); err != nil {
|
||||
return nil, fmt.Errorf("decode run manifest %q: %w", path, err)
|
||||
}
|
||||
|
||||
if err := validateLoadedRunManifest(&m); err != nil {
|
||||
return nil, fmt.Errorf("run manifest %q invalid: %w", path, err)
|
||||
}
|
||||
normalizeRunManifest(&m)
|
||||
|
||||
return &m, nil
|
||||
}
|
||||
|
||||
// SaveRun writes the run manifest to path atomically via temp file + rename.
|
||||
func (s *LocalStore) SaveRun(ctx context.Context, path string, m *RunManifest) error {
|
||||
if err := checkContext(ctx); err != nil {
|
||||
return err
|
||||
}
|
||||
if strings.TrimSpace(path) == "" {
|
||||
return fmt.Errorf("save run manifest: path is required")
|
||||
}
|
||||
if m == nil {
|
||||
return fmt.Errorf("save run manifest: manifest is nil")
|
||||
}
|
||||
if strings.TrimSpace(m.SessionID) == "" {
|
||||
return fmt.Errorf("save run manifest: session_id is required")
|
||||
}
|
||||
if strings.TrimSpace(m.RunID) == "" {
|
||||
return fmt.Errorf("save run manifest: run_id is required")
|
||||
}
|
||||
if m.CreatedAt.IsZero() {
|
||||
return fmt.Errorf("save run manifest: created_at is required")
|
||||
}
|
||||
|
||||
m.UpdatedAt = time.Now().UTC()
|
||||
if m.Stages == nil {
|
||||
m.Stages = map[string]*RunStageRecord{}
|
||||
}
|
||||
|
||||
data, err := json.MarshalIndent(m, "", " ")
|
||||
if err != nil {
|
||||
return fmt.Errorf("save run manifest: marshal: %w", err)
|
||||
}
|
||||
data = append(data, '\n')
|
||||
|
||||
return writeJSONAtomically(ctx, path, ".run-manifest.json.tmp-*", data)
|
||||
}
|
||||
|
||||
func validateLoadedManifest(m *Manifest) error {
|
||||
if m == nil {
|
||||
return fmt.Errorf("manifest is nil")
|
||||
@@ -161,6 +244,88 @@ func normalizeManifest(m *Manifest) {
|
||||
}
|
||||
}
|
||||
|
||||
func validateLoadedRunManifest(m *RunManifest) error {
|
||||
if m == nil {
|
||||
return fmt.Errorf("manifest is nil")
|
||||
}
|
||||
if strings.TrimSpace(m.SessionID) == "" {
|
||||
return fmt.Errorf("session_id is required")
|
||||
}
|
||||
if strings.TrimSpace(m.RunID) == "" {
|
||||
return fmt.Errorf("run_id is required")
|
||||
}
|
||||
if m.CreatedAt.IsZero() {
|
||||
return fmt.Errorf("created_at is required")
|
||||
}
|
||||
if m.UpdatedAt.IsZero() {
|
||||
return fmt.Errorf("updated_at is required")
|
||||
}
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
func normalizeRunManifest(m *RunManifest) {
|
||||
if m.Stages == nil {
|
||||
m.Stages = map[string]*RunStageRecord{}
|
||||
}
|
||||
for name, stage := range m.Stages {
|
||||
if stage == nil {
|
||||
stage = &RunStageRecord{
|
||||
Name: name,
|
||||
Action: RunStageActionRun,
|
||||
Status: StatusPending,
|
||||
CreatedAt: m.CreatedAt,
|
||||
UpdatedAt: m.UpdatedAt,
|
||||
}
|
||||
m.Stages[name] = stage
|
||||
}
|
||||
if stage.Name == "" {
|
||||
stage.Name = name
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func writeJSONAtomically(ctx context.Context, path, tempPattern string, data []byte) error {
|
||||
dir := filepath.Dir(path)
|
||||
if err := os.MkdirAll(dir, 0o755); err != nil {
|
||||
return fmt.Errorf("create directory %q: %w", dir, err)
|
||||
}
|
||||
|
||||
tmp, err := os.CreateTemp(dir, tempPattern)
|
||||
if err != nil {
|
||||
return fmt.Errorf("create temp file: %w", err)
|
||||
}
|
||||
tmpName := tmp.Name()
|
||||
removeTmp := true
|
||||
defer func() {
|
||||
if removeTmp {
|
||||
_ = os.Remove(tmpName)
|
||||
}
|
||||
}()
|
||||
|
||||
if _, err := tmp.Write(data); err != nil {
|
||||
_ = tmp.Close()
|
||||
return fmt.Errorf("write temp file: %w", err)
|
||||
}
|
||||
if err := tmp.Sync(); err != nil {
|
||||
_ = tmp.Close()
|
||||
return fmt.Errorf("sync temp file: %w", err)
|
||||
}
|
||||
if err := tmp.Close(); err != nil {
|
||||
return fmt.Errorf("close temp file: %w", err)
|
||||
}
|
||||
if err := checkContext(ctx); err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
if err := os.Rename(tmpName, path); err != nil {
|
||||
return fmt.Errorf("rename temp file: %w", err)
|
||||
}
|
||||
removeTmp = false
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
func checkContext(ctx context.Context) error {
|
||||
if ctx == nil {
|
||||
return nil
|
||||
|
||||
@@ -150,3 +150,73 @@ func TestLoadRejectsInvalidManifest(t *testing.T) {
|
||||
t.Fatalf("error = %q, want session_id validation", err.Error())
|
||||
}
|
||||
}
|
||||
|
||||
func TestLocalStoreCreateSaveLoadRunManifestRoundTrip(t *testing.T) {
|
||||
store := &LocalStore{}
|
||||
ctx := context.Background()
|
||||
|
||||
run, err := store.CreateRun(
|
||||
ctx,
|
||||
"2026-05-03",
|
||||
"forsaken",
|
||||
"20260517T000000Z-abcdef12",
|
||||
true,
|
||||
[]string{"prepare", "transcribe"},
|
||||
)
|
||||
if err != nil {
|
||||
t.Fatalf("CreateRun() error = %v", err)
|
||||
}
|
||||
run.SessionManifestPath = "/var/lib/narratio/work/forsaken/2026-05-03/manifest.json"
|
||||
run.MarkStageRunning("prepare", time.Date(2026, 5, 3, 12, 1, 0, 0, time.UTC))
|
||||
run.MarkStageSucceeded("prepare", time.Date(2026, 5, 3, 12, 2, 0, 0, time.UTC), []ArtifactRecord{
|
||||
{Kind: "input", LocalPath: "inputs/session.yml"},
|
||||
})
|
||||
run.MarkSucceeded(time.Date(2026, 5, 3, 12, 3, 0, 0, time.UTC))
|
||||
|
||||
path := filepath.Join(t.TempDir(), "run-manifest.json")
|
||||
if err := store.SaveRun(ctx, path, run); err != nil {
|
||||
t.Fatalf("SaveRun() error = %v", err)
|
||||
}
|
||||
|
||||
loaded, err := store.LoadRun(ctx, path)
|
||||
if err != nil {
|
||||
t.Fatalf("LoadRun() error = %v", err)
|
||||
}
|
||||
|
||||
if loaded.SessionID != "2026-05-03" {
|
||||
t.Fatalf("SessionID = %q, want %q", loaded.SessionID, "2026-05-03")
|
||||
}
|
||||
if loaded.Campaign != "forsaken" {
|
||||
t.Fatalf("Campaign = %q, want %q", loaded.Campaign, "forsaken")
|
||||
}
|
||||
if loaded.RunID != "20260517T000000Z-abcdef12" {
|
||||
t.Fatalf("RunID = %q, want %q", loaded.RunID, "20260517T000000Z-abcdef12")
|
||||
}
|
||||
if loaded.Status != RunManifestStatusSucceeded {
|
||||
t.Fatalf("Status = %q, want %q", loaded.Status, RunManifestStatusSucceeded)
|
||||
}
|
||||
if loaded.Stages["prepare"] == nil || loaded.Stages["prepare"].Status != StatusSucceeded {
|
||||
t.Fatalf("prepare stage = %#v, want succeeded", loaded.Stages["prepare"])
|
||||
}
|
||||
if loaded.Stages["prepare"].Action != RunStageActionRun {
|
||||
t.Fatalf("prepare action = %q, want %q", loaded.Stages["prepare"].Action, RunStageActionRun)
|
||||
}
|
||||
}
|
||||
|
||||
func TestLoadRunRejectsInvalidManifest(t *testing.T) {
|
||||
store := &LocalStore{}
|
||||
ctx := context.Background()
|
||||
|
||||
path := filepath.Join(t.TempDir(), "run-manifest.json")
|
||||
if err := os.WriteFile(path, []byte(`{"session_id":"2026-05-03"}`), 0o644); err != nil {
|
||||
t.Fatalf("WriteFile() error = %v", err)
|
||||
}
|
||||
|
||||
_, err := store.LoadRun(ctx, path)
|
||||
if err == nil {
|
||||
t.Fatal("expected error, got nil")
|
||||
}
|
||||
if !strings.Contains(err.Error(), "run_id is required") {
|
||||
t.Fatalf("error = %q, want run_id validation", err.Error())
|
||||
}
|
||||
}
|
||||
|
||||
@@ -3,6 +3,7 @@ package stage
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"os"
|
||||
"path/filepath"
|
||||
@@ -58,7 +59,11 @@ func (analyzeStage) Run(ctx context.Context, env *Env, m *manifest.Manifest) (*S
|
||||
return nil, fmt.Errorf("analyze: session id is required")
|
||||
}
|
||||
|
||||
paths := env.ArtifactStore.SessionPaths(sessionID)
|
||||
paths := sessionPathsForEnv(env, sessionID)
|
||||
runLayout, err := resolveRunStageLayout(env, m, paths, sessionID, "analyze")
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("analyze: resolve run-stage layout: %w", err)
|
||||
}
|
||||
if env.Config.Pipeline.Scriptorium == nil {
|
||||
return &StageResult{
|
||||
Metadata: map[string]any{
|
||||
@@ -83,27 +88,7 @@ func (analyzeStage) Run(ctx context.Context, env *Env, m *manifest.Manifest) (*S
|
||||
}, nil
|
||||
}
|
||||
|
||||
processedTranscriptPath, processedSource, err := discoverProcessedTranscript(m, paths)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("analyze: resolve processed transcript: %w", err)
|
||||
}
|
||||
normalizedTranscriptPath, normalizedSource, err := discoverNormalizedTranscript(m, paths)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("analyze: resolve normalized transcript: %w", err)
|
||||
}
|
||||
trimmedTranscriptPath, trimmedSource, err := discoverTrimmedTranscript(m, paths)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("analyze: resolve trimmed transcript: %w", err)
|
||||
}
|
||||
|
||||
transcriptInputs := analyzeTranscriptInputs{
|
||||
ProcessedPath: processedTranscriptPath,
|
||||
ProcessedSource: processedSource,
|
||||
NormalizedPath: normalizedTranscriptPath,
|
||||
NormalizedSource: normalizedSource,
|
||||
TrimmedPath: trimmedTranscriptPath,
|
||||
TrimmedSource: trimmedSource,
|
||||
}
|
||||
transcriptRefs := discoverAnalyzeTranscriptRefs(m, paths)
|
||||
|
||||
inputPaths := map[string]string{}
|
||||
omittedOptionalInputs := []string{}
|
||||
@@ -111,7 +96,7 @@ func (analyzeStage) Run(ctx context.Context, env *Env, m *manifest.Manifest) (*S
|
||||
inputNames := sortedScriptoriumInputNames(artifactCfg.Inputs)
|
||||
for _, inputName := range inputNames {
|
||||
inputCfg := artifactCfg.Inputs[inputName]
|
||||
resolvedPath, resolved, resolveErr := resolveScriptoriumInput(inputName, inputCfg, transcriptInputs, paths, sessionDir)
|
||||
resolvedPath, resolved, resolveErr := resolveScriptoriumInput(inputName, inputCfg, m, paths, sessionDir)
|
||||
if resolveErr != nil {
|
||||
return nil, fmt.Errorf("analyze: resolve input %q: %w", inputName, resolveErr)
|
||||
}
|
||||
@@ -130,13 +115,22 @@ func (analyzeStage) Run(ctx context.Context, env *Env, m *manifest.Manifest) (*S
|
||||
return nil, fmt.Errorf("analyze: resolve vars: %w", err)
|
||||
}
|
||||
|
||||
outputPath, err := resolveScriptoriumOutputPath(paths, artifactCfg.OutputPath)
|
||||
canonicalOutputPath, err := resolveScriptoriumOutputPath(paths, artifactCfg.OutputPath)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("analyze: resolve output path: %w", err)
|
||||
}
|
||||
outputPath, err := runLocalPathForCanonical(runLayout, paths, canonicalOutputPath)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("analyze: resolve run-local output path: %w", err)
|
||||
}
|
||||
stdoutLogPath := filepath.Join(paths.LogsDir, "scriptorium."+artifactName+".stdout.log")
|
||||
stderrLogPath := filepath.Join(paths.LogsDir, "scriptorium."+artifactName+".stderr.log")
|
||||
generatedConfigPath := filepath.Join(paths.ConfigDir, "scriptorium."+artifactName+".generated.yml")
|
||||
if runLayout.Enabled {
|
||||
stdoutLogPath = filepath.Join(runLayout.LogsDir, "scriptorium."+artifactName+".stdout.log")
|
||||
stderrLogPath = filepath.Join(runLayout.LogsDir, "scriptorium."+artifactName+".stderr.log")
|
||||
generatedConfigPath = filepath.Join(runLayout.ConfigDir, "scriptorium."+artifactName+".generated.yml")
|
||||
}
|
||||
|
||||
timeout, err := resolveScriptoriumTimeout(env.Config.Pipeline.Scriptorium.Timeout, artifactCfg.Timeout)
|
||||
if err != nil {
|
||||
@@ -157,20 +151,28 @@ func (analyzeStage) Run(ctx context.Context, env *Env, m *manifest.Manifest) (*S
|
||||
"omitted_optional_inputs": omittedOptionalInputs,
|
||||
"vars": vars,
|
||||
"timeout": timeout.String(),
|
||||
"processed_transcript_path": processedTranscriptPath,
|
||||
"processed_transcript_source": processedSource,
|
||||
"normalized_transcript_path": normalizedTranscriptPath,
|
||||
"normalized_transcript_source": normalizedSource,
|
||||
"trimmed_transcript_path": trimmedTranscriptPath,
|
||||
"trimmed_transcript_source": trimmedSource,
|
||||
"processed_transcript_path": transcriptRefs.ProcessedPath,
|
||||
"processed_transcript_source": transcriptRefs.ProcessedSource,
|
||||
"normalized_transcript_path": transcriptRefs.NormalizedPath,
|
||||
"normalized_transcript_source": transcriptRefs.NormalizedSource,
|
||||
"trimmed_transcript_path": transcriptRefs.TrimmedPath,
|
||||
"trimmed_transcript_source": transcriptRefs.TrimmedSource,
|
||||
"render_debug_enabled": resolveRenderDebugEnabled(env.Config.Pipeline.Scriptorium.RenderDebug, artifactCfg.RenderDebug),
|
||||
}
|
||||
|
||||
if meta["render_debug_enabled"] == true {
|
||||
renderOutputPath := filepath.Join(paths.ArtifactsDir, artifactName+".render.json")
|
||||
if runLayout.Enabled {
|
||||
renderOutputPath = filepath.Join(runLayout.ReportsDir, artifactName+".render.json")
|
||||
}
|
||||
renderStdoutPath := filepath.Join(paths.LogsDir, "scriptorium."+artifactName+".render.stdout.log")
|
||||
renderStderrPath := filepath.Join(paths.LogsDir, "scriptorium."+artifactName+".render.stderr.log")
|
||||
renderGeneratedConfigPath := filepath.Join(paths.ConfigDir, "scriptorium."+artifactName+".render.generated.yml")
|
||||
if runLayout.Enabled {
|
||||
renderStdoutPath = filepath.Join(runLayout.LogsDir, "scriptorium."+artifactName+".render.stdout.log")
|
||||
renderStderrPath = filepath.Join(runLayout.LogsDir, "scriptorium."+artifactName+".render.stderr.log")
|
||||
renderGeneratedConfigPath = filepath.Join(runLayout.ConfigDir, "scriptorium."+artifactName+".render.generated.yml")
|
||||
}
|
||||
|
||||
renderReq := scriptorium.RenderArtifactRequest{
|
||||
Binary: env.Config.Pipeline.Scriptorium.Binary,
|
||||
@@ -257,17 +259,19 @@ func (analyzeStage) Run(ctx context.Context, env *Env, m *manifest.Manifest) (*S
|
||||
if err := requireNonEmptyFile(finalOutputPath, artifactName+" output"); err != nil {
|
||||
return nil, fmt.Errorf("analyze: %w", err)
|
||||
}
|
||||
|
||||
artifactRef := artifacts.Ref{
|
||||
Kind: artifactName,
|
||||
Category: "artifacts",
|
||||
SessionID: sessionID,
|
||||
AbsolutePath: finalOutputPath,
|
||||
promotedArtifact, err := promoteRunLocalOutput(env.ArtifactStore, finalOutputPath, canonicalOutputPath, artifacts.Ref{
|
||||
Kind: artifactName,
|
||||
Category: "artifacts",
|
||||
SessionID: sessionID,
|
||||
})
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("analyze: promote artifact output: %w", err)
|
||||
}
|
||||
|
||||
logPaths = append(logPaths, stdoutLogPath, stderrLogPath)
|
||||
generatedConfigs = append(generatedConfigs, generatedConfigPath)
|
||||
meta["output_path"] = finalOutputPath
|
||||
meta["run_output_path"] = finalOutputPath
|
||||
meta["output_path"] = canonicalOutputPath
|
||||
meta["generated_config_path"] = generatedConfigPath
|
||||
meta["stdout_log_path"] = stdoutLogPath
|
||||
meta["stderr_log_path"] = stderrLogPath
|
||||
@@ -286,7 +290,7 @@ func (analyzeStage) Run(ctx context.Context, env *Env, m *manifest.Manifest) (*S
|
||||
}
|
||||
|
||||
return &StageResult{
|
||||
Outputs: []artifacts.Ref{artifactRef},
|
||||
Outputs: []artifacts.Ref{promotedArtifact},
|
||||
Logs: logPaths,
|
||||
GeneratedConfigs: generatedConfigs,
|
||||
Metadata: meta,
|
||||
@@ -350,40 +354,6 @@ func discoverProcessedTranscript(m *manifest.Manifest, paths artifacts.SessionPa
|
||||
return "", "", nil
|
||||
}
|
||||
|
||||
func discoverTrimmedTranscript(m *manifest.Manifest, paths artifacts.SessionPaths) (string, string, error) {
|
||||
candidates := []string{}
|
||||
if m != nil && m.Stages != nil {
|
||||
if sr := m.Stages["trim"]; sr != nil {
|
||||
for _, out := range sr.Outputs {
|
||||
if out.Kind != "transcript_trimmed" {
|
||||
continue
|
||||
}
|
||||
p := strings.TrimSpace(out.LocalPath)
|
||||
if p == "" {
|
||||
continue
|
||||
}
|
||||
resolved := artifacts.ResolveSessionLocalPathForRead(paths, p)
|
||||
candidates = append(candidates, filepath.Clean(resolved))
|
||||
}
|
||||
}
|
||||
}
|
||||
deduped := dedupeAndSortPaths(candidates)
|
||||
for _, p := range deduped {
|
||||
if info, err := os.Stat(p); err == nil && !info.IsDir() {
|
||||
return p, "manifest.trim.outputs", nil
|
||||
}
|
||||
}
|
||||
|
||||
fallback := filepath.Join(paths.TranscriptsDir, "trimmed.json")
|
||||
if info, err := os.Stat(fallback); err == nil && !info.IsDir() {
|
||||
return filepath.Clean(fallback), "fallback.transcripts_dir", nil
|
||||
}
|
||||
if len(deduped) > 0 {
|
||||
return deduped[0], "manifest.trim.outputs", nil
|
||||
}
|
||||
return "", "", nil
|
||||
}
|
||||
|
||||
type analyzeTranscriptInputs struct {
|
||||
ProcessedPath string
|
||||
ProcessedSource string
|
||||
@@ -393,38 +363,36 @@ type analyzeTranscriptInputs struct {
|
||||
TrimmedSource string
|
||||
}
|
||||
|
||||
func discoverAnalyzeTranscriptRefs(m *manifest.Manifest, paths artifacts.SessionPaths) analyzeTranscriptInputs {
|
||||
processedPath, processedSource := discoverAnalyzeArtifactRef(m, paths, artifacts.ArtifactTranscriptPolished)
|
||||
normalizedPath, normalizedSource := discoverAnalyzeArtifactRef(m, paths, artifacts.ArtifactTranscriptFull)
|
||||
trimmedPath, trimmedSource := discoverAnalyzeArtifactRef(m, paths, artifacts.ArtifactTranscriptTrimmed)
|
||||
return analyzeTranscriptInputs{
|
||||
ProcessedPath: processedPath,
|
||||
ProcessedSource: processedSource,
|
||||
NormalizedPath: normalizedPath,
|
||||
NormalizedSource: normalizedSource,
|
||||
TrimmedPath: trimmedPath,
|
||||
TrimmedSource: trimmedSource,
|
||||
}
|
||||
}
|
||||
|
||||
func discoverAnalyzeArtifactRef(m *manifest.Manifest, paths artifacts.SessionPaths, source string) (string, string) {
|
||||
resolved, err := artifacts.ResolveSessionArtifact(paths, m, source)
|
||||
if err != nil {
|
||||
return "", ""
|
||||
}
|
||||
return resolved.Path, resolved.Provenance
|
||||
}
|
||||
|
||||
func resolveScriptoriumInput(
|
||||
inputName string,
|
||||
inputCfg config.ScriptoriumInputConfig,
|
||||
transcriptInputs analyzeTranscriptInputs,
|
||||
m *manifest.Manifest,
|
||||
paths artifacts.SessionPaths,
|
||||
sessionDir string,
|
||||
) (string, bool, error) {
|
||||
switch strings.TrimSpace(inputCfg.Source) {
|
||||
case "processed_transcript":
|
||||
if strings.TrimSpace(transcriptInputs.ProcessedPath) == "" {
|
||||
return "", false, nil
|
||||
}
|
||||
if err := validateProcessedTranscriptOutput(transcriptInputs.ProcessedPath); err != nil {
|
||||
return "", false, fmt.Errorf("processed transcript %q invalid: %w", transcriptInputs.ProcessedPath, err)
|
||||
}
|
||||
return transcriptInputs.ProcessedPath, true, nil
|
||||
case "normalized_transcript":
|
||||
if strings.TrimSpace(transcriptInputs.NormalizedPath) == "" {
|
||||
return "", false, fmt.Errorf("normalized transcript input is unavailable; run normalize stage first")
|
||||
}
|
||||
if err := validateProcessedTranscriptOutput(transcriptInputs.NormalizedPath); err != nil {
|
||||
return "", false, fmt.Errorf("normalized transcript %q invalid: %w", transcriptInputs.NormalizedPath, err)
|
||||
}
|
||||
return transcriptInputs.NormalizedPath, true, nil
|
||||
case "trimmed_transcript":
|
||||
if strings.TrimSpace(transcriptInputs.TrimmedPath) == "" {
|
||||
return "", false, fmt.Errorf("trimmed transcript input is unavailable; run trim stage first")
|
||||
}
|
||||
if err := validateProcessedTranscriptOutput(transcriptInputs.TrimmedPath); err != nil {
|
||||
return "", false, fmt.Errorf("trimmed transcript %q invalid: %w", transcriptInputs.TrimmedPath, err)
|
||||
}
|
||||
return transcriptInputs.TrimmedPath, true, nil
|
||||
case "previous_session_artifact":
|
||||
if strings.TrimSpace(inputCfg.Path) == "" {
|
||||
return "", false, nil
|
||||
@@ -435,7 +403,27 @@ func resolveScriptoriumInput(
|
||||
}
|
||||
return resolved, true, nil
|
||||
default:
|
||||
return "", false, fmt.Errorf("unsupported source %q", inputCfg.Source)
|
||||
resolved, err := artifacts.ResolveSessionArtifact(paths, m, inputCfg.Source)
|
||||
if err == nil {
|
||||
return resolved.Path, true, nil
|
||||
}
|
||||
if errors.Is(err, artifacts.ErrSessionArtifactNotFound) {
|
||||
normalized, normalizeErr := artifacts.NormalizeSessionArtifactSource(inputCfg.Source)
|
||||
if normalizeErr != nil {
|
||||
return "", false, normalizeErr
|
||||
}
|
||||
switch normalized {
|
||||
case artifacts.ArtifactTranscriptPolished:
|
||||
return "", false, nil
|
||||
case artifacts.ArtifactTranscriptFull:
|
||||
return "", false, fmt.Errorf("normalized transcript input is unavailable; run normalize stage first")
|
||||
case artifacts.ArtifactTranscriptTrimmed:
|
||||
return "", false, fmt.Errorf("trimmed transcript input is unavailable; run trim stage first")
|
||||
default:
|
||||
return "", false, nil
|
||||
}
|
||||
}
|
||||
return "", false, err
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
@@ -17,7 +17,7 @@ import (
|
||||
|
||||
func TestAnalyzeGeneratesSessionRecapFromTrimmedTranscript(t *testing.T) {
|
||||
env, m, fake := setupAnalyzeEnv(t)
|
||||
paths := env.ArtifactStore.SessionPaths(m.SessionID)
|
||||
paths := sessionPathsForEnv(env, m.SessionID)
|
||||
writeAnalyzeFile(t, filepath.Join(paths.TranscriptsDir, "trimmed.json"), `{"segments":[]}`)
|
||||
|
||||
result, err := (analyzeStage{}).Run(context.Background(), env, m)
|
||||
@@ -70,7 +70,7 @@ func TestAnalyzeGeneratesSessionRecapFromTrimmedTranscript(t *testing.T) {
|
||||
|
||||
func TestAnalyzeRenderDebugFalseDoesNotCallRenderArtifact(t *testing.T) {
|
||||
env, m, fake := setupAnalyzeEnv(t)
|
||||
paths := env.ArtifactStore.SessionPaths(m.SessionID)
|
||||
paths := sessionPathsForEnv(env, m.SessionID)
|
||||
writeAnalyzeFile(t, filepath.Join(paths.TranscriptsDir, "trimmed.json"), `{"segments":[]}`)
|
||||
|
||||
env.Config.Pipeline.Scriptorium.RenderDebug = false
|
||||
@@ -86,7 +86,7 @@ func TestAnalyzeRenderDebugFalseDoesNotCallRenderArtifact(t *testing.T) {
|
||||
|
||||
func TestAnalyzeRenderDebugArtifactOverrideFalseWinsOverGlobalTrue(t *testing.T) {
|
||||
env, m, fake := setupAnalyzeEnv(t)
|
||||
paths := env.ArtifactStore.SessionPaths(m.SessionID)
|
||||
paths := sessionPathsForEnv(env, m.SessionID)
|
||||
writeAnalyzeFile(t, filepath.Join(paths.TranscriptsDir, "trimmed.json"), `{"segments":[]}`)
|
||||
|
||||
env.Config.Pipeline.Scriptorium.RenderDebug = true
|
||||
@@ -109,7 +109,7 @@ func TestAnalyzeRenderDebugArtifactOverrideFalseWinsOverGlobalTrue(t *testing.T)
|
||||
|
||||
func TestAnalyzeRenderDebugTrueCallsRenderBeforeRun(t *testing.T) {
|
||||
env, m, _ := setupAnalyzeEnv(t)
|
||||
paths := env.ArtifactStore.SessionPaths(m.SessionID)
|
||||
paths := sessionPathsForEnv(env, m.SessionID)
|
||||
writeAnalyzeFile(t, filepath.Join(paths.TranscriptsDir, "trimmed.json"), `{"segments":[]}`)
|
||||
|
||||
env.Config.Pipeline.Scriptorium.RenderDebug = true
|
||||
@@ -130,7 +130,7 @@ func TestAnalyzeRenderDebugTrueCallsRenderBeforeRun(t *testing.T) {
|
||||
|
||||
func TestAnalyzeRenderOutputPathIsRecorded(t *testing.T) {
|
||||
env, m, fake := setupAnalyzeEnv(t)
|
||||
paths := env.ArtifactStore.SessionPaths(m.SessionID)
|
||||
paths := sessionPathsForEnv(env, m.SessionID)
|
||||
writeAnalyzeFile(t, filepath.Join(paths.TranscriptsDir, "trimmed.json"), `{"segments":[]}`)
|
||||
env.Config.Pipeline.Scriptorium.RenderDebug = true
|
||||
|
||||
@@ -154,7 +154,7 @@ func TestAnalyzeRenderOutputPathIsRecorded(t *testing.T) {
|
||||
|
||||
func TestAnalyzeRenderFailurePreventsRun(t *testing.T) {
|
||||
env, m, fake := setupAnalyzeEnv(t)
|
||||
paths := env.ArtifactStore.SessionPaths(m.SessionID)
|
||||
paths := sessionPathsForEnv(env, m.SessionID)
|
||||
writeAnalyzeFile(t, filepath.Join(paths.TranscriptsDir, "trimmed.json"), `{"segments":[]}`)
|
||||
env.Config.Pipeline.Scriptorium.RenderDebug = true
|
||||
fake.RenderErr = errors.New("render boom")
|
||||
@@ -176,7 +176,7 @@ func TestAnalyzeRenderFailurePreventsRun(t *testing.T) {
|
||||
|
||||
func TestAnalyzeRenderInvalidJSONFailsClearly(t *testing.T) {
|
||||
env, m, _ := setupAnalyzeEnv(t)
|
||||
paths := env.ArtifactStore.SessionPaths(m.SessionID)
|
||||
paths := sessionPathsForEnv(env, m.SessionID)
|
||||
writeAnalyzeFile(t, filepath.Join(paths.TranscriptsDir, "trimmed.json"), `{"segments":[]}`)
|
||||
env.Config.Pipeline.Scriptorium.RenderDebug = true
|
||||
runner := &orderedScriptoriumRunner{
|
||||
@@ -199,7 +199,7 @@ func TestAnalyzeRenderInvalidJSONFailsClearly(t *testing.T) {
|
||||
|
||||
func TestAnalyzeRunStillSucceedsWhenRenderSucceeds(t *testing.T) {
|
||||
env, m, fake := setupAnalyzeEnv(t)
|
||||
paths := env.ArtifactStore.SessionPaths(m.SessionID)
|
||||
paths := sessionPathsForEnv(env, m.SessionID)
|
||||
writeAnalyzeFile(t, filepath.Join(paths.TranscriptsDir, "trimmed.json"), `{"segments":[]}`)
|
||||
env.Config.Pipeline.Scriptorium.RenderDebug = true
|
||||
|
||||
@@ -220,7 +220,7 @@ func TestAnalyzeRunStillSucceedsWhenRenderSucceeds(t *testing.T) {
|
||||
|
||||
func TestAnalyzeOmitsOptionalPreviousRecapWhenUnavailable(t *testing.T) {
|
||||
env, m, fake := setupAnalyzeEnv(t)
|
||||
paths := env.ArtifactStore.SessionPaths(m.SessionID)
|
||||
paths := sessionPathsForEnv(env, m.SessionID)
|
||||
writeAnalyzeFile(t, filepath.Join(paths.TranscriptsDir, "processed.json"), `{"segments":[]}`)
|
||||
|
||||
env.Config.Pipeline.Scriptorium.Artifacts["session_recap"] = config.ScriptoriumArtifactConfig{
|
||||
@@ -256,6 +256,32 @@ func TestAnalyzeOmitsOptionalPreviousRecapWhenUnavailable(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
func TestAnalyzeUsesRunLocalPathsAndPromotesCanonical(t *testing.T) {
|
||||
env, m, fake := setupAnalyzeEnv(t)
|
||||
env.Config.Session.Campaign = "sample-campaign"
|
||||
m.Campaign = "sample-campaign"
|
||||
m.RunID = "20260518T010203Z-abcdef12"
|
||||
paths := sessionPathsForEnv(env, m.SessionID)
|
||||
writeAnalyzeFile(t, filepath.Join(paths.TranscriptsDir, "trimmed.json"), `{"segments":[]}`)
|
||||
|
||||
result, err := (analyzeStage{}).Run(context.Background(), env, m)
|
||||
if err != nil {
|
||||
t.Fatalf("Run() error = %v", err)
|
||||
}
|
||||
if len(fake.RunRequests) != 1 {
|
||||
t.Fatalf("run requests = %d, want 1", len(fake.RunRequests))
|
||||
}
|
||||
if !strings.Contains(fake.RunRequests[0].OutputPath, filepath.Join("runs", m.RunID, "analyze", "outputs")) {
|
||||
t.Fatalf("run output path = %q, want run-local path", fake.RunRequests[0].OutputPath)
|
||||
}
|
||||
if len(result.Outputs) != 1 {
|
||||
t.Fatalf("outputs len = %d, want 1", len(result.Outputs))
|
||||
}
|
||||
if strings.Contains(result.Outputs[0].AbsolutePath, string(filepath.Separator)+"runs"+string(filepath.Separator)) {
|
||||
t.Fatalf("promoted output path = %q, want canonical session path", result.Outputs[0].AbsolutePath)
|
||||
}
|
||||
}
|
||||
|
||||
type orderedScriptoriumRunner struct {
|
||||
Calls []string
|
||||
RenderErr error
|
||||
@@ -318,7 +344,7 @@ func (r *orderedScriptoriumRunner) RunArtifact(_ context.Context, req scriptoriu
|
||||
|
||||
func TestAnalyzeIncludesPreviousRecapWhenConfiguredAndAvailable(t *testing.T) {
|
||||
env, m, fake := setupAnalyzeEnv(t)
|
||||
paths := env.ArtifactStore.SessionPaths(m.SessionID)
|
||||
paths := sessionPathsForEnv(env, m.SessionID)
|
||||
writeAnalyzeFile(t, filepath.Join(paths.TranscriptsDir, "processed.json"), `{"segments":[]}`)
|
||||
|
||||
previousRecapPath := filepath.Join(filepath.Dir(env.Config.SessionPath), "previous", "session_recap.md")
|
||||
@@ -359,7 +385,7 @@ func TestAnalyzeIncludesPreviousRecapWhenConfiguredAndAvailable(t *testing.T) {
|
||||
|
||||
func TestAnalyzeFailsWhenRequiredPreviousRecapMissing(t *testing.T) {
|
||||
env, m, _ := setupAnalyzeEnv(t)
|
||||
paths := env.ArtifactStore.SessionPaths(m.SessionID)
|
||||
paths := sessionPathsForEnv(env, m.SessionID)
|
||||
writeAnalyzeFile(t, filepath.Join(paths.TranscriptsDir, "processed.json"), `{"segments":[]}`)
|
||||
|
||||
env.Config.Pipeline.Scriptorium.Artifacts["session_recap"] = config.ScriptoriumArtifactConfig{
|
||||
@@ -390,7 +416,7 @@ func TestAnalyzeFailsWhenRequiredPreviousRecapMissing(t *testing.T) {
|
||||
|
||||
func TestAnalyzeFailsWhenOutputPathMissing(t *testing.T) {
|
||||
env, m, fake := setupAnalyzeEnv(t)
|
||||
paths := env.ArtifactStore.SessionPaths(m.SessionID)
|
||||
paths := sessionPathsForEnv(env, m.SessionID)
|
||||
writeAnalyzeFile(t, filepath.Join(paths.TranscriptsDir, "trimmed.json"), `{"segments":[]}`)
|
||||
|
||||
artifact := env.Config.Pipeline.Scriptorium.Artifacts["session_recap"]
|
||||
@@ -425,7 +451,7 @@ func TestAnalyzeFailsWhenTrimmedTranscriptMissing(t *testing.T) {
|
||||
|
||||
func TestAnalyzeSupportsProcessedTranscriptSourceWhenConfigured(t *testing.T) {
|
||||
env, m, fake := setupAnalyzeEnv(t)
|
||||
paths := env.ArtifactStore.SessionPaths(m.SessionID)
|
||||
paths := sessionPathsForEnv(env, m.SessionID)
|
||||
writeAnalyzeFile(t, filepath.Join(paths.TranscriptsDir, "processed.json"), `{"segments":[]}`)
|
||||
|
||||
artifact := env.Config.Pipeline.Scriptorium.Artifacts["session_recap"]
|
||||
@@ -447,9 +473,33 @@ func TestAnalyzeSupportsProcessedTranscriptSourceWhenConfigured(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
func TestAnalyzeSupportsCanonicalTrimmedTranscriptSourceWhenConfigured(t *testing.T) {
|
||||
env, m, fake := setupAnalyzeEnv(t)
|
||||
paths := sessionPathsForEnv(env, m.SessionID)
|
||||
writeAnalyzeFile(t, filepath.Join(paths.TranscriptsDir, "trimmed.json"), `{"segments":[]}`)
|
||||
|
||||
artifact := env.Config.Pipeline.Scriptorium.Artifacts["session_recap"]
|
||||
artifact.Inputs["transcript"] = config.ScriptoriumInputConfig{
|
||||
Source: "narratio.transcript.trimmed",
|
||||
Required: true,
|
||||
}
|
||||
env.Config.Pipeline.Scriptorium.Artifacts["session_recap"] = artifact
|
||||
|
||||
_, err := (analyzeStage{}).Run(context.Background(), env, m)
|
||||
if err != nil {
|
||||
t.Fatalf("Run() error = %v", err)
|
||||
}
|
||||
if len(fake.RunRequests) != 1 {
|
||||
t.Fatalf("run requests = %d, want 1", len(fake.RunRequests))
|
||||
}
|
||||
if fake.RunRequests[0].InputPaths["transcript"] != filepath.Join(paths.TranscriptsDir, "trimmed.json") {
|
||||
t.Fatalf("transcript input = %q, want trimmed transcript path", fake.RunRequests[0].InputPaths["transcript"])
|
||||
}
|
||||
}
|
||||
|
||||
func TestAnalyzeSupportsNormalizedTranscriptSourceWhenConfigured(t *testing.T) {
|
||||
env, m, fake := setupAnalyzeEnv(t)
|
||||
paths := env.ArtifactStore.SessionPaths(m.SessionID)
|
||||
paths := sessionPathsForEnv(env, m.SessionID)
|
||||
normalizedPath := filepath.Join(paths.TranscriptsDir, "normalized.json")
|
||||
writeAnalyzeFile(t, normalizedPath, `{"segments":[{"id":1}]}`)
|
||||
|
||||
@@ -472,9 +522,39 @@ func TestAnalyzeSupportsNormalizedTranscriptSourceWhenConfigured(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
func TestAnalyzeSupportsCanonicalNormalizedTranscriptSourceFromManifestOutput(t *testing.T) {
|
||||
env, m, fake := setupAnalyzeEnv(t)
|
||||
paths := sessionPathsForEnv(env, m.SessionID)
|
||||
fallbackPath := filepath.Join(paths.TranscriptsDir, "normalized.json")
|
||||
manifestPath := filepath.Join(paths.ArtifactsDir, "normalized.from-manifest.json")
|
||||
writeAnalyzeFile(t, fallbackPath, `{"segments":[{"id":999}]}`)
|
||||
writeAnalyzeFile(t, manifestPath, `{"segments":[{"id":10}]}`)
|
||||
m.MarkStageSucceeded("normalize", time.Now().UTC(), []manifest.ArtifactRecord{
|
||||
{Kind: "transcript_normalized", LocalPath: manifestPath},
|
||||
})
|
||||
|
||||
artifact := env.Config.Pipeline.Scriptorium.Artifacts["session_recap"]
|
||||
artifact.Inputs["transcript"] = config.ScriptoriumInputConfig{
|
||||
Source: "narratio.transcript.full",
|
||||
Required: true,
|
||||
}
|
||||
env.Config.Pipeline.Scriptorium.Artifacts["session_recap"] = artifact
|
||||
|
||||
_, err := (analyzeStage{}).Run(context.Background(), env, m)
|
||||
if err != nil {
|
||||
t.Fatalf("Run() error = %v", err)
|
||||
}
|
||||
if len(fake.RunRequests) != 1 {
|
||||
t.Fatalf("run requests = %d, want 1", len(fake.RunRequests))
|
||||
}
|
||||
if fake.RunRequests[0].InputPaths["transcript"] != manifestPath {
|
||||
t.Fatalf("transcript input = %q, want manifest normalized transcript path", fake.RunRequests[0].InputPaths["transcript"])
|
||||
}
|
||||
}
|
||||
|
||||
func TestAnalyzeSupportsNormalizedTranscriptSourceFromManifestOutput(t *testing.T) {
|
||||
env, m, fake := setupAnalyzeEnv(t)
|
||||
paths := env.ArtifactStore.SessionPaths(m.SessionID)
|
||||
paths := sessionPathsForEnv(env, m.SessionID)
|
||||
fallbackPath := filepath.Join(paths.TranscriptsDir, "normalized.json")
|
||||
manifestPath := filepath.Join(paths.ArtifactsDir, "normalized.from-manifest.json")
|
||||
writeAnalyzeFile(t, fallbackPath, `{"segments":[{"id":999}]}`)
|
||||
@@ -528,7 +608,7 @@ func TestAnalyzeFailsWhenNormalizedTranscriptMissing(t *testing.T) {
|
||||
|
||||
func TestAnalyzeFailsWhenProcessedTranscriptInvalidJSON(t *testing.T) {
|
||||
env, m, _ := setupAnalyzeEnv(t)
|
||||
paths := env.ArtifactStore.SessionPaths(m.SessionID)
|
||||
paths := sessionPathsForEnv(env, m.SessionID)
|
||||
writeAnalyzeFile(t, filepath.Join(paths.TranscriptsDir, "trimmed.json"), `{not-json`)
|
||||
|
||||
_, err := (analyzeStage{}).Run(context.Background(), env, m)
|
||||
@@ -542,7 +622,7 @@ func TestAnalyzeFailsWhenProcessedTranscriptInvalidJSON(t *testing.T) {
|
||||
|
||||
func TestAnalyzeFailsWhenProcessedTranscriptMissingSegmentsArray(t *testing.T) {
|
||||
env, m, _ := setupAnalyzeEnv(t)
|
||||
paths := env.ArtifactStore.SessionPaths(m.SessionID)
|
||||
paths := sessionPathsForEnv(env, m.SessionID)
|
||||
writeAnalyzeFile(t, filepath.Join(paths.TranscriptsDir, "trimmed.json"), `{"not_segments":[]}`)
|
||||
|
||||
_, err := (analyzeStage{}).Run(context.Background(), env, m)
|
||||
@@ -556,7 +636,7 @@ func TestAnalyzeFailsWhenProcessedTranscriptMissingSegmentsArray(t *testing.T) {
|
||||
|
||||
func TestAnalyzeRecordsRefsAndMetadata(t *testing.T) {
|
||||
env, m, _ := setupAnalyzeEnv(t)
|
||||
paths := env.ArtifactStore.SessionPaths(m.SessionID)
|
||||
paths := sessionPathsForEnv(env, m.SessionID)
|
||||
writeAnalyzeFile(t, filepath.Join(paths.TranscriptsDir, "trimmed.json"), `{"segments":[]}`)
|
||||
|
||||
result, err := (analyzeStage{}).Run(context.Background(), env, m)
|
||||
@@ -585,7 +665,7 @@ func TestAnalyzeRecordsRefsAndMetadata(t *testing.T) {
|
||||
|
||||
func TestAnalyzeHandlesAdapterError(t *testing.T) {
|
||||
env, m, fake := setupAnalyzeEnv(t)
|
||||
paths := env.ArtifactStore.SessionPaths(m.SessionID)
|
||||
paths := sessionPathsForEnv(env, m.SessionID)
|
||||
writeAnalyzeFile(t, filepath.Join(paths.TranscriptsDir, "trimmed.json"), `{"segments":[]}`)
|
||||
fake.RunErr = errors.New("adapter boom")
|
||||
|
||||
@@ -600,7 +680,7 @@ func TestAnalyzeHandlesAdapterError(t *testing.T) {
|
||||
|
||||
func TestAnalyzeHandlesValidationFailedResultAsError(t *testing.T) {
|
||||
env, m, fake := setupAnalyzeEnv(t)
|
||||
paths := env.ArtifactStore.SessionPaths(m.SessionID)
|
||||
paths := sessionPathsForEnv(env, m.SessionID)
|
||||
writeAnalyzeFile(t, filepath.Join(paths.TranscriptsDir, "trimmed.json"), `{"segments":[]}`)
|
||||
fake.RunResult = scriptorium.ArtifactResult{
|
||||
ValidationFailed: true,
|
||||
@@ -619,7 +699,7 @@ func TestAnalyzeHandlesValidationFailedResultAsError(t *testing.T) {
|
||||
|
||||
func TestAnalyzeSkipsWhenNoEnabledScriptoriumArtifactsConfigured(t *testing.T) {
|
||||
env, m, _ := setupAnalyzeEnv(t)
|
||||
paths := env.ArtifactStore.SessionPaths(m.SessionID)
|
||||
paths := sessionPathsForEnv(env, m.SessionID)
|
||||
writeAnalyzeFile(t, filepath.Join(paths.TranscriptsDir, "trimmed.json"), `{"segments":[]}`)
|
||||
|
||||
env.Config.Pipeline.Scriptorium.Artifacts["session_recap"] = config.ScriptoriumArtifactConfig{
|
||||
@@ -692,7 +772,7 @@ func setupAnalyzeEnv(t *testing.T) (*Env, *manifest.Manifest, *scriptorium.FakeR
|
||||
}
|
||||
|
||||
store := artifacts.NewLocalStore(workspace)
|
||||
if _, err := store.EnsureLayout("2026-05-03"); err != nil {
|
||||
if _, err := store.EnsureLayoutFor("sample-campaign", "2026-05-03"); err != nil {
|
||||
t.Fatalf("EnsureLayout() error = %v", err)
|
||||
}
|
||||
|
||||
|
||||
@@ -18,6 +18,10 @@ import (
|
||||
)
|
||||
|
||||
type archiveStage struct{}
|
||||
type archiveUploadFile struct {
|
||||
RelativePath string
|
||||
LocalPath string
|
||||
}
|
||||
|
||||
var archivePrerequisiteStages = []string{
|
||||
"prepare",
|
||||
@@ -29,15 +33,6 @@ var archivePrerequisiteStages = []string{
|
||||
"analyze",
|
||||
}
|
||||
|
||||
var archiveRunUploadDirs = []string{
|
||||
"inputs",
|
||||
"transcripts",
|
||||
"artifacts",
|
||||
"reports",
|
||||
"config",
|
||||
"logs",
|
||||
}
|
||||
|
||||
func (archiveStage) Name() string { return "archive" }
|
||||
|
||||
func (archiveStage) Declares() IODecl {
|
||||
@@ -83,16 +78,16 @@ func (archiveStage) Run(ctx context.Context, env *Env, m *manifest.Manifest) (*S
|
||||
return nil, fmt.Errorf("archive: remote object store backend is required when archive run upload is enabled")
|
||||
}
|
||||
|
||||
workDir, err := archiveWorkDir(env, m)
|
||||
runRoot, err := resolveArchiveRunRoot(env, m)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("archive: resolve local workdir: %w", err)
|
||||
return nil, fmt.Errorf("archive: resolve run root: %w", err)
|
||||
}
|
||||
workDirInfo, err := os.Stat(workDir)
|
||||
runRootInfo, err := os.Stat(runRoot)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("archive: local workdir %q: %w", workDir, err)
|
||||
return nil, fmt.Errorf("archive: run root %q: %w", runRoot, err)
|
||||
}
|
||||
if !workDirInfo.IsDir() {
|
||||
return nil, fmt.Errorf("archive: local workdir %q is not a directory", workDir)
|
||||
if !runRootInfo.IsDir() {
|
||||
return nil, fmt.Errorf("archive: run root %q is not a directory", runRoot)
|
||||
}
|
||||
|
||||
runPrefix, err := archiveRunPrefix(env, m)
|
||||
@@ -112,29 +107,30 @@ func (archiveStage) Run(ctx context.Context, env *Env, m *manifest.Manifest) (*S
|
||||
return nil, fmt.Errorf("archive: run id is required")
|
||||
}
|
||||
|
||||
runFiles, err := collectArchiveRunFiles(workDir)
|
||||
manifestSource, err := resolveArchiveRunManifestSource(runRoot)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("archive: resolve run manifest source: %w", err)
|
||||
}
|
||||
|
||||
runFiles, err := collectArchiveRunFiles(runRoot, manifestSource)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("archive: collect run files: %w", err)
|
||||
}
|
||||
promotions, err := resolveArchivePromotions(workDir, env.Config.Pipeline.Archive.PromoteArtifacts)
|
||||
sessionRoot, err := resolveArchiveSessionRoot(env, m)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("archive: resolve session root for promotions: %w", err)
|
||||
}
|
||||
promotions, err := resolveArchivePromotions(sessionRoot, env.Config.Pipeline.Archive.PromoteArtifacts)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("archive: resolve promotion rules: %w", err)
|
||||
}
|
||||
currentManifestSource := filepath.Join(workDir, "manifest.json")
|
||||
if info, err := os.Stat(currentManifestSource); err != nil {
|
||||
return nil, fmt.Errorf("archive: current manifest source %q: %w", currentManifestSource, err)
|
||||
} else if info.IsDir() {
|
||||
return nil, fmt.Errorf("archive: current manifest source %q is a directory", currentManifestSource)
|
||||
}
|
||||
|
||||
runUploaded := make([]string, 0, len(runFiles))
|
||||
for _, rel := range runFiles {
|
||||
localPath := filepath.Join(workDir, filepath.FromSlash(rel))
|
||||
key := artifacts.S3RunRelativeDestinationKey(runPrefix, rel)
|
||||
if _, err := env.ObjectStore.Upload(ctx, localPath, key, storage.UploadOptions{}); err != nil {
|
||||
return nil, fmt.Errorf("archive: upload run file %q to %q: %w", rel, key, err)
|
||||
for _, file := range runFiles {
|
||||
key := artifacts.S3RunRelativeDestinationKey(runPrefix, file.RelativePath)
|
||||
if _, err := env.ObjectStore.Upload(ctx, file.LocalPath, key, storage.UploadOptions{}); err != nil {
|
||||
return nil, fmt.Errorf("archive: upload run file %q to %q: %w", file.RelativePath, key, err)
|
||||
}
|
||||
runUploaded = append(runUploaded, rel)
|
||||
runUploaded = append(runUploaded, file.RelativePath)
|
||||
}
|
||||
|
||||
promotedUploaded := make([]string, 0, len(promotions))
|
||||
@@ -247,36 +243,53 @@ func validateArchivePrerequisites(m *manifest.Manifest) error {
|
||||
return nil
|
||||
}
|
||||
|
||||
func archiveWorkDir(env *Env, m *manifest.Manifest) (string, error) {
|
||||
workDir := strings.TrimSpace(m.LocalWorkDir)
|
||||
if workDir != "" {
|
||||
cleaned := filepath.Clean(workDir)
|
||||
if info, err := os.Stat(cleaned); err == nil && info.IsDir() {
|
||||
return cleaned, nil
|
||||
}
|
||||
}
|
||||
|
||||
func resolveArchiveRunRoot(env *Env, m *manifest.Manifest) (string, error) {
|
||||
sessionID := strings.TrimSpace(env.Config.Session.SessionID)
|
||||
if sessionID == "" {
|
||||
if sessionID == "" && m != nil {
|
||||
sessionID = strings.TrimSpace(m.SessionID)
|
||||
}
|
||||
campaign := strings.TrimSpace(env.Config.Session.Campaign)
|
||||
if campaign == "" {
|
||||
if campaign == "" && m != nil {
|
||||
campaign = strings.TrimSpace(m.Campaign)
|
||||
}
|
||||
runID := strings.TrimSpace(m.RunID)
|
||||
runID := ""
|
||||
if m != nil {
|
||||
runID = strings.TrimSpace(m.RunID)
|
||||
}
|
||||
if sessionID == "" || campaign == "" {
|
||||
return "", fmt.Errorf("campaign and session id are required")
|
||||
}
|
||||
if runID == "" {
|
||||
return "", fmt.Errorf("run id is required")
|
||||
}
|
||||
if campaign == "" || sessionID == "" {
|
||||
return "", fmt.Errorf("campaign and session id are required")
|
||||
|
||||
canonical := filepath.Clean(artifacts.SessionRunRootForCampaign(env.Config.Pipeline.Workspace.Root, campaign, sessionID, runID))
|
||||
canonicalExists, err := directoryExists(canonical)
|
||||
if err != nil {
|
||||
return "", fmt.Errorf("check canonical run root %q: %w", canonical, err)
|
||||
}
|
||||
runScoped := artifacts.SessionRunWorkDir(env.Config.Pipeline.Workspace.Root, campaign, sessionID, runID)
|
||||
if info, err := os.Stat(runScoped); err == nil && info.IsDir() {
|
||||
return runScoped, nil
|
||||
if !canonicalExists {
|
||||
return "", fmt.Errorf("run root not found for campaign %q session %q run %q at canonical path %q", campaign, sessionID, runID, canonical)
|
||||
}
|
||||
legacy := artifacts.SessionWorkDir(env.Config.Pipeline.Workspace.Root, sessionID)
|
||||
return legacy, nil
|
||||
return canonical, nil
|
||||
}
|
||||
|
||||
func resolveArchiveSessionRoot(env *Env, m *manifest.Manifest) (string, error) {
|
||||
sessionID := strings.TrimSpace(env.Config.Session.SessionID)
|
||||
if sessionID == "" && m != nil {
|
||||
sessionID = strings.TrimSpace(m.SessionID)
|
||||
}
|
||||
campaign := strings.TrimSpace(env.Config.Session.Campaign)
|
||||
if campaign == "" && m != nil {
|
||||
campaign = strings.TrimSpace(m.Campaign)
|
||||
}
|
||||
if sessionID == "" {
|
||||
return "", fmt.Errorf("session id is required")
|
||||
}
|
||||
if campaign == "" {
|
||||
return "", fmt.Errorf("campaign is required")
|
||||
}
|
||||
return filepath.Clean(artifacts.SessionWorkDirForCampaign(env.Config.Pipeline.Workspace.Root, campaign, sessionID)), nil
|
||||
}
|
||||
|
||||
func archiveRunPrefix(env *Env, m *manifest.Manifest) (string, error) {
|
||||
@@ -329,22 +342,27 @@ func archiveBucket(env *Env, m *manifest.Manifest) string {
|
||||
return strings.TrimSpace(env.Config.Pipeline.Storage.S3.Bucket)
|
||||
}
|
||||
|
||||
func resolveArchivePromotions(workDir string, rules []config.ArchivePromotionRule) ([]archivePromotion, error) {
|
||||
func resolveArchivePromotions(sessionRoot string, rules []config.ArchivePromotionRule) ([]archivePromotion, error) {
|
||||
sessionRoot = filepath.Clean(strings.TrimSpace(sessionRoot))
|
||||
if sessionRoot == "" {
|
||||
return nil, fmt.Errorf("session root is required")
|
||||
}
|
||||
out := make([]archivePromotion, 0, len(rules))
|
||||
for _, rule := range rules {
|
||||
from := strings.TrimSpace(rule.From)
|
||||
to := strings.TrimSpace(rule.To)
|
||||
required := rule.Required == nil || *rule.Required
|
||||
|
||||
localPath, err := resolveWorkDirRelativePath(workDir, from)
|
||||
resolvedPath, err := resolveWorkDirRelativePath(sessionRoot, from)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("promotion from %q: %w", from, err)
|
||||
}
|
||||
info, err := os.Stat(localPath)
|
||||
info, err := os.Stat(resolvedPath)
|
||||
exists := err == nil && !info.IsDir()
|
||||
if err != nil && !os.IsNotExist(err) {
|
||||
return nil, fmt.Errorf("promotion source %q: %w", from, err)
|
||||
}
|
||||
localPath := resolvedPath
|
||||
|
||||
out = append(out, archivePromotion{
|
||||
From: from,
|
||||
@@ -375,58 +393,97 @@ func resolveWorkDirRelativePath(workDir, rel string) (string, error) {
|
||||
return cleanedFull, nil
|
||||
}
|
||||
|
||||
func collectArchiveRunFiles(workDir string) ([]string, error) {
|
||||
files := make([]string, 0, 64)
|
||||
|
||||
for _, dirName := range archiveRunUploadDirs {
|
||||
fullDir := filepath.Join(workDir, dirName)
|
||||
info, err := os.Stat(fullDir)
|
||||
if err != nil {
|
||||
if os.IsNotExist(err) {
|
||||
continue
|
||||
}
|
||||
return nil, fmt.Errorf("stat %q: %w", fullDir, err)
|
||||
func collectArchiveRunFiles(runRoot, manifestPath string) ([]archiveUploadFile, error) {
|
||||
files := make([]archiveUploadFile, 0, 64)
|
||||
err := filepath.WalkDir(runRoot, func(path string, d fs.DirEntry, walkErr error) error {
|
||||
if walkErr != nil {
|
||||
return walkErr
|
||||
}
|
||||
if !info.IsDir() {
|
||||
continue
|
||||
}
|
||||
|
||||
if err := filepath.WalkDir(fullDir, func(path string, d fs.DirEntry, walkErr error) error {
|
||||
if walkErr != nil {
|
||||
return walkErr
|
||||
}
|
||||
if d.IsDir() {
|
||||
if d.IsDir() {
|
||||
if path == runRoot {
|
||||
return nil
|
||||
}
|
||||
rel, err := filepath.Rel(workDir, path)
|
||||
relDir, err := filepath.Rel(runRoot, path)
|
||||
if err != nil {
|
||||
return fmt.Errorf("relative path from %q to %q: %w", workDir, path, err)
|
||||
return fmt.Errorf("relative dir from %q to %q: %w", runRoot, path, err)
|
||||
}
|
||||
relDir = filepath.ToSlash(relDir)
|
||||
// Preserve existing behavior: audio is not uploaded in archive run record.
|
||||
if relDir == "audio" || strings.HasPrefix(relDir, "audio/") {
|
||||
return filepath.SkipDir
|
||||
}
|
||||
rel = filepath.ToSlash(rel)
|
||||
files = append(files, rel)
|
||||
return nil
|
||||
}); err != nil {
|
||||
return nil, fmt.Errorf("walk %q: %w", fullDir, err)
|
||||
}
|
||||
rel, err := filepath.Rel(runRoot, path)
|
||||
if err != nil {
|
||||
return fmt.Errorf("relative path from %q to %q: %w", runRoot, path, err)
|
||||
}
|
||||
rel = filepath.ToSlash(rel)
|
||||
files = append(files, archiveUploadFile{
|
||||
RelativePath: rel,
|
||||
LocalPath: path,
|
||||
})
|
||||
return nil
|
||||
})
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("walk %q: %w", runRoot, err)
|
||||
}
|
||||
|
||||
manifestPath := filepath.Join(workDir, "manifest.json")
|
||||
manifestInfo, err := os.Stat(manifestPath)
|
||||
if err != nil {
|
||||
if os.IsNotExist(err) {
|
||||
return nil, fmt.Errorf("manifest.json not found in workdir %q", workDir)
|
||||
return nil, fmt.Errorf("manifest.json not found (checked path %q)", manifestPath)
|
||||
}
|
||||
return nil, fmt.Errorf("stat %q: %w", manifestPath, err)
|
||||
}
|
||||
if manifestInfo.IsDir() {
|
||||
return nil, fmt.Errorf("manifest path %q is a directory", manifestPath)
|
||||
}
|
||||
files = append(files, "manifest.json")
|
||||
files = append(files, archiveUploadFile{
|
||||
RelativePath: "manifest.json",
|
||||
LocalPath: manifestPath,
|
||||
})
|
||||
seen := map[string]archiveUploadFile{}
|
||||
for _, file := range files {
|
||||
seen[file.RelativePath] = file
|
||||
}
|
||||
files = files[:0]
|
||||
for _, file := range seen {
|
||||
files = append(files, file)
|
||||
}
|
||||
|
||||
sort.Strings(files)
|
||||
sort.Slice(files, func(i, j int) bool {
|
||||
return files[i].RelativePath < files[j].RelativePath
|
||||
})
|
||||
return files, nil
|
||||
}
|
||||
|
||||
func resolveArchiveRunManifestSource(runRoot string) (string, error) {
|
||||
path := filepath.Join(filepath.Clean(runRoot), "manifest.json")
|
||||
info, err := os.Stat(path)
|
||||
if err != nil {
|
||||
if os.IsNotExist(err) {
|
||||
return "", fmt.Errorf("manifest.json not found in run root %q", runRoot)
|
||||
}
|
||||
return "", fmt.Errorf("stat %q: %w", path, err)
|
||||
}
|
||||
if info.IsDir() {
|
||||
return "", fmt.Errorf("manifest path %q is a directory", path)
|
||||
}
|
||||
return path, nil
|
||||
}
|
||||
|
||||
func directoryExists(path string) (bool, error) {
|
||||
info, err := os.Stat(path)
|
||||
if err == nil {
|
||||
return info.IsDir(), nil
|
||||
}
|
||||
if os.IsNotExist(err) {
|
||||
return false, nil
|
||||
}
|
||||
return false, err
|
||||
}
|
||||
|
||||
func writeCurrentManifestSnapshot(m *manifest.Manifest, archiveMetadata map[string]any) (string, error) {
|
||||
if m == nil {
|
||||
return "", fmt.Errorf("manifest is required")
|
||||
|
||||
@@ -3,6 +3,7 @@ package stage
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"reflect"
|
||||
"strings"
|
||||
@@ -72,14 +73,15 @@ func TestArchiveUploadsRunRecordPromotionsAndCurrentPointer(t *testing.T) {
|
||||
runPrefix := m.S3RunPrefix
|
||||
sessionPrefix := m.S3SessionPrefix
|
||||
wantRunUploads := []string{
|
||||
"artifacts/session_recap.md",
|
||||
"config/audita.generated.yml",
|
||||
"inputs/session.yml",
|
||||
"analyze/outputs/artifacts/session_recap.md",
|
||||
"merge/config/seriatim.generated.yml",
|
||||
"prepare/inputs/session.yml",
|
||||
"prepare/outputs/audio/speaker.flac",
|
||||
"logs/audita.stderr.log",
|
||||
"manifest.json",
|
||||
"reports/audita.report.json",
|
||||
"transcripts/raw/speaker.json",
|
||||
"transcripts/trimmed.json",
|
||||
"polish/reports/audita.report.json",
|
||||
"transcribe/outputs/transcripts/raw/speaker.json",
|
||||
"trim/outputs/transcripts/trimmed.json",
|
||||
}
|
||||
for _, rel := range wantRunUploads {
|
||||
key := runPrefix + rel
|
||||
@@ -131,12 +133,11 @@ func TestArchiveUploadsRunRecordPromotionsAndCurrentPointer(t *testing.T) {
|
||||
}
|
||||
|
||||
func TestArchiveUsesCustomPromotionRules(t *testing.T) {
|
||||
env, m, workDir := archiveFixture(t)
|
||||
env, m, _ := archiveFixture(t)
|
||||
env.Config.Pipeline.Archive.PromoteArtifacts = []config.ArchivePromotionRule{
|
||||
{From: "transcripts/trimmed.json", To: "published/trimmed.json", Required: boolPtr(true)},
|
||||
{From: "artifacts/session_recap.md", To: "published/recap.md", Required: boolPtr(true)},
|
||||
}
|
||||
writeStageTestFile(t, filepath.Join(workDir, "published", "ignored.txt"), "ignore\n")
|
||||
|
||||
_, err := archiveStage{}.Run(context.Background(), env, m)
|
||||
if err != nil {
|
||||
@@ -182,6 +183,21 @@ func TestArchiveFailsWhenRequiredPromotionMissing(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
func TestArchiveFailsWhenCanonicalRunRootMissing(t *testing.T) {
|
||||
env, m, runRoot := archiveFixture(t)
|
||||
if err := os.RemoveAll(runRoot); err != nil {
|
||||
t.Fatalf("remove run root: %v", err)
|
||||
}
|
||||
|
||||
_, err := archiveStage{}.Run(context.Background(), env, m)
|
||||
if err == nil {
|
||||
t.Fatal("expected missing run-root error, got nil")
|
||||
}
|
||||
if !strings.Contains(err.Error(), "run root not found") {
|
||||
t.Fatalf("error = %v, want missing run-root error", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestArchiveDoesNotWriteCurrentPointerWhenPromotionUploadFails(t *testing.T) {
|
||||
env, m, _ := archiveFixture(t)
|
||||
fake := env.ObjectStore.(*storage.FakeBackend)
|
||||
@@ -240,22 +256,27 @@ func archiveFixture(t *testing.T) (*Env, *manifest.Manifest, string) {
|
||||
runID := "20260516T010203Z-1a2b3c4d"
|
||||
campaign := "forsaken"
|
||||
sessionID := "2026-04-19"
|
||||
workDir := filepath.Join(root, "work", campaign, sessionID, runID)
|
||||
sessionRoot := artifacts.SessionWorkDirForCampaign(root, campaign, sessionID)
|
||||
runRoot := artifacts.SessionRunRootForCampaign(root, campaign, sessionID, runID)
|
||||
|
||||
writeStageTestFile(t, filepath.Join(workDir, "inputs", "session.yml"), "session_id: 2026-04-19\n")
|
||||
writeStageTestFile(t, filepath.Join(workDir, "transcripts", "raw", "speaker.json"), "{}\n")
|
||||
writeStageTestFile(t, filepath.Join(workDir, "transcripts", "trimmed.json"), "{}\n")
|
||||
writeStageTestFile(t, filepath.Join(workDir, "artifacts", "session_recap.md"), "# recap\n")
|
||||
writeStageTestFile(t, filepath.Join(workDir, "reports", "audita.report.json"), "{}\n")
|
||||
writeStageTestFile(t, filepath.Join(workDir, "config", "audita.generated.yml"), "key: value\n")
|
||||
writeStageTestFile(t, filepath.Join(workDir, "logs", "audita.stderr.log"), "stderr\n")
|
||||
writeStageTestFile(t, filepath.Join(workDir, "audio", "speaker.flac"), "flac")
|
||||
writeStageTestFile(t, filepath.Join(workDir, "manifest.json"), "{}\n")
|
||||
writeStageTestFile(t, filepath.Join(sessionRoot, "transcripts", "trimmed.json"), "{}\n")
|
||||
writeStageTestFile(t, filepath.Join(sessionRoot, "artifacts", "session_recap.md"), "# recap\n")
|
||||
|
||||
writeStageTestFile(t, filepath.Join(runRoot, "prepare", "inputs", "session.yml"), "session_id: 2026-04-19\n")
|
||||
writeStageTestFile(t, filepath.Join(runRoot, "prepare", "outputs", "audio", "speaker.flac"), "flac\n")
|
||||
writeStageTestFile(t, filepath.Join(runRoot, "transcribe", "outputs", "transcripts", "raw", "speaker.json"), "{}\n")
|
||||
writeStageTestFile(t, filepath.Join(runRoot, "trim", "outputs", "transcripts", "trimmed.json"), "{}\n")
|
||||
writeStageTestFile(t, filepath.Join(runRoot, "analyze", "outputs", "artifacts", "session_recap.md"), "# recap\n")
|
||||
writeStageTestFile(t, filepath.Join(runRoot, "polish", "reports", "audita.report.json"), "{}\n")
|
||||
writeStageTestFile(t, filepath.Join(runRoot, "merge", "config", "seriatim.generated.yml"), "key: value\n")
|
||||
writeStageTestFile(t, filepath.Join(runRoot, "logs", "audita.stderr.log"), "stderr\n")
|
||||
writeStageTestFile(t, filepath.Join(runRoot, "audio", "speaker.flac"), "flac")
|
||||
writeStageTestFile(t, filepath.Join(runRoot, "manifest.json"), "{}\n")
|
||||
|
||||
m := manifest.New(sessionID, time.Date(2026, 5, 16, 1, 2, 3, 0, time.UTC))
|
||||
m.Campaign = campaign
|
||||
m.RunID = runID
|
||||
m.LocalWorkDir = workDir
|
||||
m.LocalWorkDir = runRoot
|
||||
m.S3Bucket = "my-dnd-archive"
|
||||
m.S3SessionPrefix = artifacts.S3SessionPrefix("dnd", campaign, sessionID)
|
||||
m.S3RunPrefix = artifacts.S3RunPrefix(m.S3SessionPrefix, runID)
|
||||
@@ -289,7 +310,7 @@ func archiveFixture(t *testing.T) (*Env, *manifest.Manifest, string) {
|
||||
},
|
||||
ObjectStore: &storage.FakeBackend{},
|
||||
}
|
||||
return env, m, workDir
|
||||
return env, m, runRoot
|
||||
}
|
||||
|
||||
type promotionFailingStore struct {
|
||||
|
||||
@@ -7,6 +7,7 @@ import (
|
||||
"path/filepath"
|
||||
"sort"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"gitea.maximumdirect.net/eric/narratio/internal/adapters/seriatim"
|
||||
"gitea.maximumdirect.net/eric/narratio/internal/artifacts"
|
||||
@@ -56,7 +57,11 @@ func (mergeStage) Run(ctx context.Context, env *Env, m *manifest.Manifest) (*Sta
|
||||
return nil, fmt.Errorf("merge: session id is required")
|
||||
}
|
||||
|
||||
paths := env.ArtifactStore.SessionPaths(sessionID)
|
||||
paths := sessionPathsForEnv(env, sessionID)
|
||||
runLayout, err := resolveRunStageLayout(env, m, paths, sessionID, "merge")
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("merge: resolve run-stage layout: %w", err)
|
||||
}
|
||||
|
||||
inputs, err := discoverRawTranscripts(m, paths)
|
||||
if err != nil {
|
||||
@@ -80,16 +85,37 @@ func (mergeStage) Run(ctx context.Context, env *Env, m *manifest.Manifest) (*Sta
|
||||
return nil, fmt.Errorf("merge: %w", err)
|
||||
}
|
||||
|
||||
mergedPath := filepath.Join(paths.TranscriptsDir, "merged.json")
|
||||
reportPath := filepath.Join(paths.ArtifactsDir, "seriatim.report.json")
|
||||
canonicalMergedPath := filepath.Join(paths.TranscriptsDir, "merged.json")
|
||||
mergedPath, err := runLocalPathForCanonical(runLayout, paths, canonicalMergedPath)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("merge: resolve run-local merged transcript path: %w", err)
|
||||
}
|
||||
canonicalReportPath := filepath.Join(paths.ArtifactsDir, "seriatim.report.json")
|
||||
reportPath := canonicalReportPath
|
||||
if runLayout.Enabled {
|
||||
reportPath, err = runLocalPathForCanonical(runLayout, paths, canonicalReportPath)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("merge: resolve run-local report path: %w", err)
|
||||
}
|
||||
}
|
||||
stdoutPath := filepath.Join(paths.LogsDir, "seriatim.stdout.log")
|
||||
stderrPath := filepath.Join(paths.LogsDir, "seriatim.stderr.log")
|
||||
genCfgPath := filepath.Join(paths.ConfigDir, "seriatim.generated.yml")
|
||||
if runLayout.Enabled {
|
||||
stdoutPath = filepath.Join(runLayout.LogsDir, "seriatim.stdout.log")
|
||||
stderrPath = filepath.Join(runLayout.LogsDir, "seriatim.stderr.log")
|
||||
genCfgPath = filepath.Join(runLayout.ConfigDir, "seriatim.generated.yml")
|
||||
}
|
||||
|
||||
normalizedInputs, normalizeLogs, normalizeConfigs, normalizeMeta, err := normalizeMergeInputs(ctx, env, inputs, paths, runLayout)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
reportEnabled := env.Config.Pipeline.Seriatim.Report != nil && *env.Config.Pipeline.Seriatim.Report
|
||||
req := seriatim.MergeRequest{
|
||||
GeneratedConfigPath: genCfgPath,
|
||||
InputTranscriptPaths: inputs,
|
||||
InputTranscriptPaths: normalizedInputs,
|
||||
OutputMergedTranscriptPath: mergedPath,
|
||||
ReportPath: "",
|
||||
SpeakersPath: speakersPath,
|
||||
@@ -124,35 +150,50 @@ func (mergeStage) Run(ctx context.Context, env *Env, m *manifest.Manifest) (*Sta
|
||||
}
|
||||
}
|
||||
|
||||
outputs := []artifacts.Ref{{
|
||||
Kind: "transcript_merged",
|
||||
Category: "transcripts",
|
||||
SessionID: sessionID,
|
||||
AbsolutePath: finalMergedPath,
|
||||
}}
|
||||
promotedMerged, err := promoteRunLocalOutput(env.ArtifactStore, finalMergedPath, canonicalMergedPath, artifacts.Ref{
|
||||
Kind: "transcript_merged",
|
||||
Category: "transcripts",
|
||||
SessionID: sessionID,
|
||||
})
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("merge: promote merged transcript: %w", err)
|
||||
}
|
||||
outputs := []artifacts.Ref{promotedMerged}
|
||||
if reportEnabled {
|
||||
outputs = append(outputs, artifacts.Ref{
|
||||
Kind: "seriatim_report",
|
||||
Category: "artifacts",
|
||||
SessionID: sessionID,
|
||||
AbsolutePath: finalReportPath,
|
||||
promotedReport, err := promoteRunLocalOutput(env.ArtifactStore, finalReportPath, canonicalReportPath, artifacts.Ref{
|
||||
Kind: "seriatim_report",
|
||||
Category: "artifacts",
|
||||
SessionID: sessionID,
|
||||
})
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("merge: promote report: %w", err)
|
||||
}
|
||||
outputs = append(outputs, promotedReport)
|
||||
}
|
||||
|
||||
coalesceGap := any(nil)
|
||||
if env.Config.Pipeline.Seriatim.CoalesceGap != nil {
|
||||
coalesceGap = *env.Config.Pipeline.Seriatim.CoalesceGap
|
||||
}
|
||||
reportCanonicalPath := ""
|
||||
if reportEnabled {
|
||||
reportCanonicalPath = canonicalReportPath
|
||||
}
|
||||
|
||||
meta := map[string]any{
|
||||
"stage": "merge",
|
||||
"input_transcripts_count": len(inputs),
|
||||
"input_transcripts_count": len(normalizedInputs),
|
||||
"input_transcript_paths": inputs,
|
||||
"normalized_inputs_count": len(normalizedInputs),
|
||||
"normalized_input_paths": normalizedInputs,
|
||||
"normalize_inputs": normalizeMeta,
|
||||
"output_schema": env.Config.Pipeline.Seriatim.OutputSchema,
|
||||
"coalesce_gap": coalesceGap,
|
||||
"report_enabled": reportEnabled,
|
||||
"output_path": finalMergedPath,
|
||||
"report_path": finalReportPath,
|
||||
"run_output_path": finalMergedPath,
|
||||
"output_path": canonicalMergedPath,
|
||||
"run_report_path": finalReportPath,
|
||||
"report_path": reportCanonicalPath,
|
||||
"timeout": env.Config.Pipeline.Seriatim.Timeout,
|
||||
"binary": env.Config.Pipeline.Seriatim.Binary,
|
||||
"adapter_duration_ms": res.Duration.Milliseconds(),
|
||||
@@ -172,12 +213,111 @@ func (mergeStage) Run(ctx context.Context, env *Env, m *manifest.Manifest) (*Sta
|
||||
|
||||
return &StageResult{
|
||||
Outputs: outputs,
|
||||
Logs: []string{stdoutPath, stderrPath},
|
||||
GeneratedConfigs: []string{genCfgPath},
|
||||
Logs: append(normalizeLogs, stdoutPath, stderrPath),
|
||||
GeneratedConfigs: append(normalizeConfigs, genCfgPath),
|
||||
Metadata: meta,
|
||||
}, nil
|
||||
}
|
||||
|
||||
type normalizeMergeInputMeta struct {
|
||||
InputPath string `json:"input_path"`
|
||||
OutputPath string `json:"output_path"`
|
||||
StdoutLogPath string `json:"stdout_log_path"`
|
||||
StderrLogPath string `json:"stderr_log_path"`
|
||||
GeneratedConfig string `json:"generated_config_path"`
|
||||
DurationMs int64 `json:"duration_ms"`
|
||||
ExitCode int `json:"exit_code"`
|
||||
InvokedBinary string `json:"invoked_binary"`
|
||||
OutputSchema string `json:"output_schema"`
|
||||
AdapterReportPath string `json:"adapter_report_path,omitempty"`
|
||||
AdapterOutputPath string `json:"adapter_output_path,omitempty"`
|
||||
}
|
||||
|
||||
func normalizeMergeInputs(
|
||||
ctx context.Context,
|
||||
env *Env,
|
||||
rawInputs []string,
|
||||
paths artifacts.SessionPaths,
|
||||
runLayout runStageLayout,
|
||||
) ([]string, []string, []string, []normalizeMergeInputMeta, error) {
|
||||
normalizedInputs := make([]string, 0, len(rawInputs))
|
||||
logs := make([]string, 0, len(rawInputs)*2)
|
||||
configs := make([]string, 0, len(rawInputs))
|
||||
meta := make([]normalizeMergeInputMeta, 0, len(rawInputs))
|
||||
normalizedDir := filepath.Join(paths.TranscriptsRawDir, "normalized")
|
||||
if runLayout.Enabled {
|
||||
normalizedDir = filepath.Join(runLayout.ScratchDir, "normalized")
|
||||
}
|
||||
if err := os.MkdirAll(normalizedDir, 0o755); err != nil {
|
||||
return nil, nil, nil, nil, fmt.Errorf("merge: ensure normalized transcripts directory %q: %w", normalizedDir, err)
|
||||
}
|
||||
|
||||
var timeout time.Duration
|
||||
timeoutRaw := strings.TrimSpace(env.Config.Pipeline.Seriatim.Timeout)
|
||||
if timeoutRaw != "" {
|
||||
parsed, err := time.ParseDuration(timeoutRaw)
|
||||
if err != nil {
|
||||
return nil, nil, nil, nil, fmt.Errorf("merge: parse seriatim timeout %q: %w", env.Config.Pipeline.Seriatim.Timeout, err)
|
||||
}
|
||||
timeout = parsed
|
||||
}
|
||||
for _, input := range rawInputs {
|
||||
base := strings.TrimSuffix(filepath.Base(input), filepath.Ext(input))
|
||||
outPath := filepath.Join(normalizedDir, base+".normalized.json")
|
||||
stdoutPath := filepath.Join(paths.LogsDir, "seriatim.normalize."+base+".stdout.log")
|
||||
stderrPath := filepath.Join(paths.LogsDir, "seriatim.normalize."+base+".stderr.log")
|
||||
cfgPath := filepath.Join(paths.ConfigDir, "seriatim.normalize."+base+".generated.yml")
|
||||
if runLayout.Enabled {
|
||||
stdoutPath = filepath.Join(runLayout.LogsDir, "seriatim.normalize."+base+".stdout.log")
|
||||
stderrPath = filepath.Join(runLayout.LogsDir, "seriatim.normalize."+base+".stderr.log")
|
||||
cfgPath = filepath.Join(runLayout.ConfigDir, "seriatim.normalize."+base+".generated.yml")
|
||||
}
|
||||
|
||||
req := seriatim.NormalizeRequest{
|
||||
Binary: env.Config.Pipeline.Seriatim.Binary,
|
||||
InputTranscriptPath: input,
|
||||
OutputNormalizedPath: outPath,
|
||||
OutputSchema: env.Config.Pipeline.Seriatim.OutputSchema,
|
||||
ReportPath: "",
|
||||
StdoutLogPath: stdoutPath,
|
||||
StderrLogPath: stderrPath,
|
||||
GeneratedConfigPath: cfgPath,
|
||||
Timeout: timeout,
|
||||
}
|
||||
res, err := env.Seriatim.Normalize(ctx, req)
|
||||
if err != nil {
|
||||
return nil, nil, nil, nil, fmt.Errorf("merge: normalize input %q failed: %w", input, err)
|
||||
}
|
||||
|
||||
finalOutputPath := outPath
|
||||
if strings.TrimSpace(res.OutputNormalizedPath) != "" {
|
||||
finalOutputPath = strings.TrimSpace(res.OutputNormalizedPath)
|
||||
}
|
||||
if err := validateTranscriptJSONFile(finalOutputPath); err != nil {
|
||||
return nil, nil, nil, nil, fmt.Errorf("merge: normalized transcript %q invalid (input %q): %w", finalOutputPath, input, err)
|
||||
}
|
||||
|
||||
normalizedInputs = append(normalizedInputs, finalOutputPath)
|
||||
logs = append(logs, stdoutPath, stderrPath)
|
||||
configs = append(configs, cfgPath)
|
||||
meta = append(meta, normalizeMergeInputMeta{
|
||||
InputPath: input,
|
||||
OutputPath: finalOutputPath,
|
||||
StdoutLogPath: stdoutPath,
|
||||
StderrLogPath: stderrPath,
|
||||
GeneratedConfig: cfgPath,
|
||||
DurationMs: res.Duration.Milliseconds(),
|
||||
ExitCode: res.ExitCode,
|
||||
InvokedBinary: res.InvokedBinary,
|
||||
OutputSchema: res.OutputSchema,
|
||||
AdapterReportPath: res.ReportPath,
|
||||
AdapterOutputPath: res.OutputNormalizedPath,
|
||||
})
|
||||
}
|
||||
|
||||
return normalizedInputs, logs, configs, meta, nil
|
||||
}
|
||||
|
||||
func discoverRawTranscripts(m *manifest.Manifest, paths artifacts.SessionPaths) ([]string, error) {
|
||||
fromManifest := make([]string, 0)
|
||||
if m != nil && m.Stages != nil {
|
||||
|
||||
@@ -2,6 +2,7 @@ package stage
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
@@ -14,9 +15,25 @@ import (
|
||||
"gitea.maximumdirect.net/eric/narratio/internal/manifest"
|
||||
)
|
||||
|
||||
type normalizeDirAssertingRunner struct {
|
||||
*seriatim.FakeRunner
|
||||
ExpectedDir string
|
||||
}
|
||||
|
||||
func (r *normalizeDirAssertingRunner) Normalize(ctx context.Context, req seriatim.NormalizeRequest) (seriatim.NormalizeResult, error) {
|
||||
info, err := os.Stat(r.ExpectedDir)
|
||||
if err != nil {
|
||||
return seriatim.NormalizeResult{}, fmt.Errorf("normalized directory check failed for %q: %w", r.ExpectedDir, err)
|
||||
}
|
||||
if !info.IsDir() {
|
||||
return seriatim.NormalizeResult{}, fmt.Errorf("normalized path %q exists but is not a directory", r.ExpectedDir)
|
||||
}
|
||||
return r.FakeRunner.Normalize(ctx, req)
|
||||
}
|
||||
|
||||
func TestMergeStageMergesRawTranscriptsAndRecordsMetadata(t *testing.T) {
|
||||
env, m := setupMergeEnv(t)
|
||||
paths := env.ArtifactStore.SessionPaths(m.SessionID)
|
||||
paths := sessionPathsForEnv(env, m.SessionID)
|
||||
|
||||
inA := filepath.Join(paths.TranscriptsRawDir, "alice.json")
|
||||
inB := filepath.Join(paths.TranscriptsRawDir, "bob.json")
|
||||
@@ -54,6 +71,17 @@ func TestMergeStageMergesRawTranscriptsAndRecordsMetadata(t *testing.T) {
|
||||
if len(req.InputTranscriptPaths) != 2 {
|
||||
t.Fatalf("input transcripts = %#v, want 2", req.InputTranscriptPaths)
|
||||
}
|
||||
if len(fake.NormalizeRequests) != 2 {
|
||||
t.Fatalf("normalize requests = %#v, want 2", fake.NormalizeRequests)
|
||||
}
|
||||
if fake.NormalizeRequests[0].InputTranscriptPath != inA || fake.NormalizeRequests[1].InputTranscriptPath != inB {
|
||||
t.Fatalf("normalize request inputs = %#v", fake.NormalizeRequests)
|
||||
}
|
||||
for _, mergeIn := range req.InputTranscriptPaths {
|
||||
if !strings.Contains(mergeIn, filepath.Join("transcripts", "raw", "normalized")) {
|
||||
t.Fatalf("merge input path = %q, want normalized input path", mergeIn)
|
||||
}
|
||||
}
|
||||
|
||||
if len(result.Outputs) != 2 {
|
||||
t.Fatalf("outputs len = %d, want 2", len(result.Outputs))
|
||||
@@ -64,11 +92,11 @@ func TestMergeStageMergesRawTranscriptsAndRecordsMetadata(t *testing.T) {
|
||||
if result.Outputs[1].Kind != "seriatim_report" {
|
||||
t.Fatalf("output[1] kind = %q, want seriatim_report", result.Outputs[1].Kind)
|
||||
}
|
||||
if len(result.Logs) != 2 {
|
||||
t.Fatalf("logs = %#v, want 2 paths", result.Logs)
|
||||
if len(result.Logs) != 6 {
|
||||
t.Fatalf("logs = %#v, want 6 paths (4 normalize + 2 merge)", result.Logs)
|
||||
}
|
||||
if len(result.GeneratedConfigs) != 1 {
|
||||
t.Fatalf("generated configs = %#v, want 1 path", result.GeneratedConfigs)
|
||||
if len(result.GeneratedConfigs) != 3 {
|
||||
t.Fatalf("generated configs = %#v, want 3 paths (2 normalize + 1 merge)", result.GeneratedConfigs)
|
||||
}
|
||||
|
||||
meta := result.Metadata
|
||||
@@ -84,11 +112,17 @@ func TestMergeStageMergesRawTranscriptsAndRecordsMetadata(t *testing.T) {
|
||||
if meta["input_transcripts_count"] != 2 {
|
||||
t.Fatalf("metadata input_transcripts_count = %#v, want 2", meta["input_transcripts_count"])
|
||||
}
|
||||
if meta["normalized_inputs_count"] != 2 {
|
||||
t.Fatalf("metadata normalized_inputs_count = %#v, want 2", meta["normalized_inputs_count"])
|
||||
}
|
||||
if _, ok := meta["normalize_inputs"]; !ok {
|
||||
t.Fatalf("metadata normalize_inputs missing: %#v", meta)
|
||||
}
|
||||
}
|
||||
|
||||
func TestMergeStageFailsWhenNoRawTranscripts(t *testing.T) {
|
||||
env, m := setupMergeEnv(t)
|
||||
paths := env.ArtifactStore.SessionPaths(m.SessionID)
|
||||
paths := sessionPathsForEnv(env, m.SessionID)
|
||||
writeFile(t, filepath.Join(paths.InputsDir, "speakers.yml"), "match: []\n")
|
||||
writeFile(t, filepath.Join(paths.InputsDir, "autocorrect.yml"), "rules: []\n")
|
||||
env.Seriatim = &seriatim.FakeRunner{}
|
||||
@@ -104,7 +138,7 @@ func TestMergeStageFailsWhenNoRawTranscripts(t *testing.T) {
|
||||
|
||||
func TestMergeStageFailsOnInvalidInputJSON(t *testing.T) {
|
||||
env, m := setupMergeEnv(t)
|
||||
paths := env.ArtifactStore.SessionPaths(m.SessionID)
|
||||
paths := sessionPathsForEnv(env, m.SessionID)
|
||||
writeFile(t, filepath.Join(paths.TranscriptsRawDir, "alice.json"), "not-json")
|
||||
writeFile(t, filepath.Join(paths.InputsDir, "speakers.yml"), "match: []\n")
|
||||
writeFile(t, filepath.Join(paths.InputsDir, "autocorrect.yml"), "rules: []\n")
|
||||
@@ -121,7 +155,7 @@ func TestMergeStageFailsOnInvalidInputJSON(t *testing.T) {
|
||||
|
||||
func TestMergeStageFailsWhenAdapterFails(t *testing.T) {
|
||||
env, m := setupMergeEnv(t)
|
||||
paths := env.ArtifactStore.SessionPaths(m.SessionID)
|
||||
paths := sessionPathsForEnv(env, m.SessionID)
|
||||
writeFile(t, filepath.Join(paths.TranscriptsRawDir, "alice.json"), `{"segments":[]}`)
|
||||
writeFile(t, filepath.Join(paths.InputsDir, "speakers.yml"), "match: []\n")
|
||||
writeFile(t, filepath.Join(paths.InputsDir, "autocorrect.yml"), "rules: []\n")
|
||||
@@ -138,7 +172,7 @@ func TestMergeStageFailsWhenAdapterFails(t *testing.T) {
|
||||
|
||||
func TestMergeStageFallsBackToRawDirectoryWhenTranscribeOutputsMissing(t *testing.T) {
|
||||
env, m := setupMergeEnv(t)
|
||||
paths := env.ArtifactStore.SessionPaths(m.SessionID)
|
||||
paths := sessionPathsForEnv(env, m.SessionID)
|
||||
writeFile(t, filepath.Join(paths.TranscriptsRawDir, "alice.json"), `{"segments":[]}`)
|
||||
writeFile(t, filepath.Join(paths.InputsDir, "speakers.yml"), "match: []\n")
|
||||
writeFile(t, filepath.Join(paths.InputsDir, "autocorrect.yml"), "rules: []\n")
|
||||
@@ -152,11 +186,14 @@ func TestMergeStageFallsBackToRawDirectoryWhenTranscribeOutputsMissing(t *testin
|
||||
if len(fake.Requests) != 1 || len(fake.Requests[0].InputTranscriptPaths) != 1 {
|
||||
t.Fatalf("fallback inputs = %#v", fake.Requests)
|
||||
}
|
||||
if len(fake.NormalizeRequests) != 1 {
|
||||
t.Fatalf("normalize requests = %#v, want 1", fake.NormalizeRequests)
|
||||
}
|
||||
}
|
||||
|
||||
func TestMergeStageResolvesWorkspaceQualifiedManifestOutputsWithoutDuplication(t *testing.T) {
|
||||
env, m := setupMergeEnvWithRelativeWorkspaceRoot(t)
|
||||
paths := env.ArtifactStore.SessionPaths(m.SessionID)
|
||||
paths := sessionPathsForEnv(env, m.SessionID)
|
||||
|
||||
rawPath := filepath.Join(paths.TranscriptsRawDir, "alice.json")
|
||||
writeFile(t, rawPath, `{"segments":[]}`)
|
||||
@@ -180,14 +217,114 @@ func TestMergeStageResolvesWorkspaceQualifiedManifestOutputsWithoutDuplication(t
|
||||
if len(got) != 1 {
|
||||
t.Fatalf("input transcript paths = %#v, want len 1", got)
|
||||
}
|
||||
if got[0] != filepath.Clean(rawPath) {
|
||||
t.Fatalf("resolved transcript path = %q, want %q", got[0], filepath.Clean(rawPath))
|
||||
if !strings.Contains(got[0], filepath.Join("transcripts", "raw", "normalized")) {
|
||||
t.Fatalf("resolved transcript path = %q, want normalized path under transcripts/raw/normalized", got[0])
|
||||
}
|
||||
}
|
||||
|
||||
func TestMergeStageCreatesNormalizedRawDirectoryBeforeNormalize(t *testing.T) {
|
||||
env, m := setupMergeEnv(t)
|
||||
paths := sessionPathsForEnv(env, m.SessionID)
|
||||
|
||||
rawPath := filepath.Join(paths.TranscriptsRawDir, "alice.json")
|
||||
writeFile(t, rawPath, `{"segments":[]}`)
|
||||
writeFile(t, filepath.Join(paths.InputsDir, "speakers.yml"), "match: []\n")
|
||||
writeFile(t, filepath.Join(paths.InputsDir, "autocorrect.yml"), "rules: []\n")
|
||||
|
||||
normalizedDir := filepath.Join(paths.TranscriptsRawDir, "normalized")
|
||||
if err := os.RemoveAll(normalizedDir); err != nil {
|
||||
t.Fatalf("remove normalized dir: %v", err)
|
||||
}
|
||||
|
||||
env.Seriatim = &normalizeDirAssertingRunner{
|
||||
FakeRunner: &seriatim.FakeRunner{},
|
||||
ExpectedDir: normalizedDir,
|
||||
}
|
||||
|
||||
if _, err := (mergeStage{}).Run(context.Background(), env, m); err != nil {
|
||||
t.Fatalf("merge.Run() error = %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestMergeStageFailsWhenNormalizeAdapterFails(t *testing.T) {
|
||||
env, m := setupMergeEnv(t)
|
||||
paths := sessionPathsForEnv(env, m.SessionID)
|
||||
in := filepath.Join(paths.TranscriptsRawDir, "alice.json")
|
||||
writeFile(t, in, `{"segments":[]}`)
|
||||
writeFile(t, filepath.Join(paths.InputsDir, "speakers.yml"), "match: []\n")
|
||||
writeFile(t, filepath.Join(paths.InputsDir, "autocorrect.yml"), "rules: []\n")
|
||||
env.Seriatim = &seriatim.FakeRunner{NormalizeErr: context.DeadlineExceeded}
|
||||
|
||||
_, err := (mergeStage{}).Run(context.Background(), env, m)
|
||||
if err == nil {
|
||||
t.Fatal("expected error, got nil")
|
||||
}
|
||||
if !strings.Contains(err.Error(), "normalize input") {
|
||||
t.Fatalf("error = %q", err.Error())
|
||||
}
|
||||
}
|
||||
|
||||
func TestMergeStageFailsWhenNormalizedOutputInvalid(t *testing.T) {
|
||||
env, m := setupMergeEnv(t)
|
||||
paths := sessionPathsForEnv(env, m.SessionID)
|
||||
in := filepath.Join(paths.TranscriptsRawDir, "alice.json")
|
||||
writeFile(t, in, `{"segments":[]}`)
|
||||
writeFile(t, filepath.Join(paths.InputsDir, "speakers.yml"), "match: []\n")
|
||||
writeFile(t, filepath.Join(paths.InputsDir, "autocorrect.yml"), "rules: []\n")
|
||||
|
||||
badNormalized := filepath.Join(paths.ArtifactsDir, "bad.normalized.json")
|
||||
writeFile(t, badNormalized, "not-json")
|
||||
env.Seriatim = &seriatim.FakeRunner{
|
||||
NormalizeResult: seriatim.NormalizeResult{
|
||||
OutputNormalizedPath: badNormalized,
|
||||
},
|
||||
}
|
||||
|
||||
_, err := (mergeStage{}).Run(context.Background(), env, m)
|
||||
if err == nil {
|
||||
t.Fatal("expected error, got nil")
|
||||
}
|
||||
if !strings.Contains(err.Error(), "normalized transcript") {
|
||||
t.Fatalf("error = %q", err.Error())
|
||||
}
|
||||
}
|
||||
|
||||
func TestMergeStageUsesRunLocalPathsAndPromotesCanonical(t *testing.T) {
|
||||
env, m := setupMergeEnv(t)
|
||||
env.Config.Session.Campaign = "sample-campaign"
|
||||
m.Campaign = "sample-campaign"
|
||||
m.RunID = "20260518T010203Z-abcdef12"
|
||||
paths := sessionPathsForEnv(env, m.SessionID)
|
||||
|
||||
in := filepath.Join(paths.TranscriptsRawDir, "alice.json")
|
||||
writeFile(t, in, `{"segments":[]}`)
|
||||
writeFile(t, filepath.Join(paths.InputsDir, "speakers.yml"), "match: []\n")
|
||||
writeFile(t, filepath.Join(paths.InputsDir, "autocorrect.yml"), "rules: []\n")
|
||||
|
||||
fake := &seriatim.FakeRunner{}
|
||||
env.Seriatim = fake
|
||||
result, err := (mergeStage{}).Run(context.Background(), env, m)
|
||||
if err != nil {
|
||||
t.Fatalf("merge.Run() error = %v", err)
|
||||
}
|
||||
if len(fake.Requests) != 1 {
|
||||
t.Fatalf("requests = %d, want 1", len(fake.Requests))
|
||||
}
|
||||
req := fake.Requests[0]
|
||||
if !strings.Contains(req.OutputMergedTranscriptPath, filepath.Join("runs", m.RunID, "merge", "outputs")) {
|
||||
t.Fatalf("run output path = %q, want run-local path", req.OutputMergedTranscriptPath)
|
||||
}
|
||||
if len(result.Outputs) == 0 {
|
||||
t.Fatalf("outputs = %#v, want promoted outputs", result.Outputs)
|
||||
}
|
||||
if strings.Contains(result.Outputs[0].AbsolutePath, string(filepath.Separator)+"runs"+string(filepath.Separator)) {
|
||||
t.Fatalf("promoted output path = %q, want canonical session path", result.Outputs[0].AbsolutePath)
|
||||
}
|
||||
}
|
||||
|
||||
func TestMergeStageResolvesSessionRelativeManifestOutputs(t *testing.T) {
|
||||
env, m := setupMergeEnv(t)
|
||||
paths := env.ArtifactStore.SessionPaths(m.SessionID)
|
||||
paths := sessionPathsForEnv(env, m.SessionID)
|
||||
|
||||
rawPath := filepath.Join(paths.TranscriptsRawDir, "alice.json")
|
||||
writeFile(t, rawPath, `{"segments":[]}`)
|
||||
@@ -211,8 +348,8 @@ func TestMergeStageResolvesSessionRelativeManifestOutputs(t *testing.T) {
|
||||
if len(got) != 1 {
|
||||
t.Fatalf("input transcript paths = %#v, want len 1", got)
|
||||
}
|
||||
if got[0] != filepath.Clean(rawPath) {
|
||||
t.Fatalf("resolved transcript path = %q, want %q", got[0], filepath.Clean(rawPath))
|
||||
if !strings.Contains(got[0], filepath.Join("transcripts", "raw", "normalized")) {
|
||||
t.Fatalf("resolved transcript path = %q, want normalized path under transcripts/raw/normalized", got[0])
|
||||
}
|
||||
}
|
||||
|
||||
@@ -269,7 +406,7 @@ func setupMergeEnvWithWorkspace(t *testing.T, workspace string) (*Env, *manifest
|
||||
}
|
||||
|
||||
store := artifacts.NewLocalStore(workspace)
|
||||
if _, err := store.EnsureLayout("2026-05-03"); err != nil {
|
||||
if _, err := store.EnsureLayoutFor("sample-campaign", "2026-05-03"); err != nil {
|
||||
t.Fatalf("EnsureLayout() error = %v", err)
|
||||
}
|
||||
return &Env{
|
||||
|
||||
@@ -54,7 +54,11 @@ func (normalizeStage) Run(ctx context.Context, env *Env, m *manifest.Manifest) (
|
||||
return nil, fmt.Errorf("normalize: session id is required")
|
||||
}
|
||||
|
||||
paths := env.ArtifactStore.SessionPaths(sessionID)
|
||||
paths := sessionPathsForEnv(env, sessionID)
|
||||
runLayout, err := resolveRunStageLayout(env, m, paths, sessionID, "normalize")
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("normalize: resolve run-stage layout: %w", err)
|
||||
}
|
||||
processedPath, processedSource, err := discoverProcessedTranscript(m, paths)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("normalize: resolve processed transcript: %w", err)
|
||||
@@ -67,18 +71,34 @@ func (normalizeStage) Run(ctx context.Context, env *Env, m *manifest.Manifest) (
|
||||
}
|
||||
|
||||
normalizeCfg := normalizeConfigOrDefault(env.Config.Pipeline.Normalize)
|
||||
normalizedPath, err := resolveScriptoriumOutputPath(paths, normalizeCfg.OutputPath)
|
||||
canonicalNormalizedPath, err := resolveScriptoriumOutputPath(paths, normalizeCfg.OutputPath)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("normalize: resolve normalized output path: %w", err)
|
||||
}
|
||||
normalizedPath, err := runLocalPathForCanonical(runLayout, paths, canonicalNormalizedPath)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("normalize: resolve run-local normalized output path: %w", err)
|
||||
}
|
||||
reportEnabled := normalizeCfg.Report != nil && *normalizeCfg.Report
|
||||
reportPath := ""
|
||||
canonicalReportPath := filepath.Join(paths.ArtifactsDir, "seriatim.normalize.report.json")
|
||||
if reportEnabled {
|
||||
reportPath = filepath.Join(paths.ArtifactsDir, "seriatim.normalize.report.json")
|
||||
reportPath = canonicalReportPath
|
||||
if runLayout.Enabled {
|
||||
reportPath, err = runLocalPathForCanonical(runLayout, paths, canonicalReportPath)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("normalize: resolve run-local report path: %w", err)
|
||||
}
|
||||
}
|
||||
}
|
||||
stdoutPath := filepath.Join(paths.LogsDir, "seriatim.normalize.stdout.log")
|
||||
stderrPath := filepath.Join(paths.LogsDir, "seriatim.normalize.stderr.log")
|
||||
generatedConfigPath := filepath.Join(paths.ConfigDir, "seriatim.normalize.generated.yml")
|
||||
if runLayout.Enabled {
|
||||
stdoutPath = filepath.Join(runLayout.LogsDir, "seriatim.normalize.stdout.log")
|
||||
stderrPath = filepath.Join(runLayout.LogsDir, "seriatim.normalize.stderr.log")
|
||||
generatedConfigPath = filepath.Join(runLayout.ConfigDir, "seriatim.normalize.generated.yml")
|
||||
}
|
||||
timeout, err := resolveTrimSeriatimTimeout(env.Config.Pipeline.Seriatim.Timeout)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("normalize: resolve seriatim timeout: %w", err)
|
||||
@@ -113,44 +133,56 @@ func (normalizeStage) Run(ctx context.Context, env *Env, m *manifest.Manifest) (
|
||||
}
|
||||
}
|
||||
|
||||
outputs := []artifacts.Ref{{
|
||||
Kind: "transcript_normalized",
|
||||
Category: "transcripts",
|
||||
SessionID: sessionID,
|
||||
AbsolutePath: finalNormalizedPath,
|
||||
}}
|
||||
promotedNormalized, err := promoteRunLocalOutput(env.ArtifactStore, finalNormalizedPath, canonicalNormalizedPath, artifacts.Ref{
|
||||
Kind: "transcript_normalized",
|
||||
Category: "transcripts",
|
||||
SessionID: sessionID,
|
||||
})
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("normalize: promote normalized transcript: %w", err)
|
||||
}
|
||||
outputs := []artifacts.Ref{promotedNormalized}
|
||||
if reportEnabled {
|
||||
outputs = append(outputs, artifacts.Ref{
|
||||
Kind: "seriatim_normalize_report",
|
||||
Category: "artifacts",
|
||||
SessionID: sessionID,
|
||||
AbsolutePath: finalReportPath,
|
||||
promotedReport, err := promoteRunLocalOutput(env.ArtifactStore, finalReportPath, canonicalReportPath, artifacts.Ref{
|
||||
Kind: "seriatim_normalize_report",
|
||||
Category: "artifacts",
|
||||
SessionID: sessionID,
|
||||
})
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("normalize: promote report: %w", err)
|
||||
}
|
||||
outputs = append(outputs, promotedReport)
|
||||
}
|
||||
reportCanonicalPath := ""
|
||||
if reportEnabled {
|
||||
reportCanonicalPath = canonicalReportPath
|
||||
}
|
||||
|
||||
meta := map[string]any{
|
||||
"stage": "normalize",
|
||||
"processed_transcript_path": processedPath,
|
||||
"processed_transcript_source": processedSource,
|
||||
"normalized_transcript_path": finalNormalizedPath,
|
||||
"normalized_transcript_source": "stage.normalize.output",
|
||||
"output_schema": normalizeCfg.OutputSchema,
|
||||
"report_enabled": reportEnabled,
|
||||
"report_path": finalReportPath,
|
||||
"timeout": env.Config.Pipeline.Seriatim.Timeout,
|
||||
"binary": env.Config.Pipeline.Seriatim.Binary,
|
||||
"stdout_log_path": stdoutPath,
|
||||
"stderr_log_path": stderrPath,
|
||||
"generated_config_path": generatedConfigPath,
|
||||
"adapter_duration_ms": res.Duration.Milliseconds(),
|
||||
"adapter_exit_code": res.ExitCode,
|
||||
"adapter_invoked_binary": res.InvokedBinary,
|
||||
"adapter_output_schema": res.OutputSchema,
|
||||
"adapter_output_path": res.OutputNormalizedPath,
|
||||
"adapter_report_path": res.ReportPath,
|
||||
"adapter_generated_config": res.GeneratedConfigPath,
|
||||
"adapter_stdout_log_path": res.StdoutLogPath,
|
||||
"adapter_stderr_log_path": res.StderrLogPath,
|
||||
"stage": "normalize",
|
||||
"processed_transcript_path": processedPath,
|
||||
"processed_transcript_source": processedSource,
|
||||
"run_normalized_transcript_path": finalNormalizedPath,
|
||||
"normalized_transcript_path": canonicalNormalizedPath,
|
||||
"normalized_transcript_source": "stage.normalize.output",
|
||||
"output_schema": normalizeCfg.OutputSchema,
|
||||
"report_enabled": reportEnabled,
|
||||
"run_report_path": finalReportPath,
|
||||
"report_path": reportCanonicalPath,
|
||||
"timeout": env.Config.Pipeline.Seriatim.Timeout,
|
||||
"binary": env.Config.Pipeline.Seriatim.Binary,
|
||||
"stdout_log_path": stdoutPath,
|
||||
"stderr_log_path": stderrPath,
|
||||
"generated_config_path": generatedConfigPath,
|
||||
"adapter_duration_ms": res.Duration.Milliseconds(),
|
||||
"adapter_exit_code": res.ExitCode,
|
||||
"adapter_invoked_binary": res.InvokedBinary,
|
||||
"adapter_output_schema": res.OutputSchema,
|
||||
"adapter_output_path": res.OutputNormalizedPath,
|
||||
"adapter_report_path": res.ReportPath,
|
||||
"adapter_generated_config": res.GeneratedConfigPath,
|
||||
"adapter_stdout_log_path": res.StdoutLogPath,
|
||||
"adapter_stderr_log_path": res.StderrLogPath,
|
||||
}
|
||||
if res.Metadata != nil {
|
||||
meta["adapter_metadata"] = res.Metadata
|
||||
|
||||
@@ -16,7 +16,7 @@ import (
|
||||
|
||||
func TestNormalizeStageConsumesProcessedTranscriptFromManifest(t *testing.T) {
|
||||
env, m, ser := setupNormalizeEnv(t)
|
||||
paths := env.ArtifactStore.SessionPaths(m.SessionID)
|
||||
paths := sessionPathsForEnv(env, m.SessionID)
|
||||
manifestProcessed := filepath.Join(paths.ArtifactsDir, "processed.from-manifest.json")
|
||||
writeFile(t, manifestProcessed, `{"segments":[{"id":10}]}`)
|
||||
writeFile(t, filepath.Join(paths.TranscriptsDir, "processed.json"), `{"segments":[{"id":99}]}`)
|
||||
@@ -43,7 +43,7 @@ func TestNormalizeStageConsumesProcessedTranscriptFromManifest(t *testing.T) {
|
||||
|
||||
func TestNormalizeStageFallsBackToProcessedTranscriptPath(t *testing.T) {
|
||||
env, m, ser := setupNormalizeEnv(t)
|
||||
paths := env.ArtifactStore.SessionPaths(m.SessionID)
|
||||
paths := sessionPathsForEnv(env, m.SessionID)
|
||||
fallback := filepath.Join(paths.TranscriptsDir, "processed.json")
|
||||
writeFile(t, fallback, `{"segments":[{"id":1}]}`)
|
||||
|
||||
@@ -72,7 +72,7 @@ func TestNormalizeStageFailsWhenProcessedTranscriptMissing(t *testing.T) {
|
||||
|
||||
func TestNormalizeStageFailsWhenProcessedTranscriptInvalidJSON(t *testing.T) {
|
||||
env, m, _ := setupNormalizeEnv(t)
|
||||
paths := env.ArtifactStore.SessionPaths(m.SessionID)
|
||||
paths := sessionPathsForEnv(env, m.SessionID)
|
||||
writeFile(t, filepath.Join(paths.TranscriptsDir, "processed.json"), "not-json")
|
||||
|
||||
_, err := (normalizeStage{}).Run(context.Background(), env, m)
|
||||
@@ -86,7 +86,7 @@ func TestNormalizeStageFailsWhenProcessedTranscriptInvalidJSON(t *testing.T) {
|
||||
|
||||
func TestNormalizeStageFailsWhenProcessedTranscriptMissingSegments(t *testing.T) {
|
||||
env, m, _ := setupNormalizeEnv(t)
|
||||
paths := env.ArtifactStore.SessionPaths(m.SessionID)
|
||||
paths := sessionPathsForEnv(env, m.SessionID)
|
||||
writeFile(t, filepath.Join(paths.TranscriptsDir, "processed.json"), `{"schema":"audita.processed.v1"}`)
|
||||
|
||||
_, err := (normalizeStage{}).Run(context.Background(), env, m)
|
||||
@@ -100,7 +100,7 @@ func TestNormalizeStageFailsWhenProcessedTranscriptMissingSegments(t *testing.T)
|
||||
|
||||
func TestNormalizeStagePassesConfiguredOutputSchemaToAdapter(t *testing.T) {
|
||||
env, m, ser := setupNormalizeEnv(t)
|
||||
paths := env.ArtifactStore.SessionPaths(m.SessionID)
|
||||
paths := sessionPathsForEnv(env, m.SessionID)
|
||||
writeFile(t, filepath.Join(paths.TranscriptsDir, "processed.json"), `{"segments":[{"id":1}]}`)
|
||||
env.Config.Pipeline.Normalize.OutputSchema = "seriatim-full"
|
||||
|
||||
@@ -121,7 +121,7 @@ func TestNormalizeStagePassesConfiguredOutputSchemaToAdapter(t *testing.T) {
|
||||
|
||||
func TestNormalizeStageRecordsNormalizedTranscriptOutputKind(t *testing.T) {
|
||||
env, m, _ := setupNormalizeEnv(t)
|
||||
paths := env.ArtifactStore.SessionPaths(m.SessionID)
|
||||
paths := sessionPathsForEnv(env, m.SessionID)
|
||||
writeFile(t, filepath.Join(paths.TranscriptsDir, "processed.json"), `{"segments":[{"id":1}]}`)
|
||||
|
||||
result, err := (normalizeStage{}).Run(context.Background(), env, m)
|
||||
@@ -138,7 +138,7 @@ func TestNormalizeStageRecordsNormalizedTranscriptOutputKind(t *testing.T) {
|
||||
|
||||
func TestNormalizeStageRecordsReportLogAndGeneratedConfigRefs(t *testing.T) {
|
||||
env, m, _ := setupNormalizeEnv(t)
|
||||
paths := env.ArtifactStore.SessionPaths(m.SessionID)
|
||||
paths := sessionPathsForEnv(env, m.SessionID)
|
||||
writeFile(t, filepath.Join(paths.TranscriptsDir, "processed.json"), `{"segments":[{"id":1}]}`)
|
||||
report := true
|
||||
env.Config.Pipeline.Normalize.Report = &report
|
||||
@@ -164,7 +164,7 @@ func TestNormalizeStageRecordsReportLogAndGeneratedConfigRefs(t *testing.T) {
|
||||
|
||||
func TestNormalizeStageFailsWhenAdapterReturnsError(t *testing.T) {
|
||||
env, m, ser := setupNormalizeEnv(t)
|
||||
paths := env.ArtifactStore.SessionPaths(m.SessionID)
|
||||
paths := sessionPathsForEnv(env, m.SessionID)
|
||||
writeFile(t, filepath.Join(paths.TranscriptsDir, "processed.json"), `{"segments":[{"id":1}]}`)
|
||||
ser.NormalizeErr = errors.New("normalize failed")
|
||||
|
||||
@@ -179,7 +179,7 @@ func TestNormalizeStageFailsWhenAdapterReturnsError(t *testing.T) {
|
||||
|
||||
func TestNormalizeStageFailsWhenNormalizedOutputInvalid(t *testing.T) {
|
||||
env, m, ser := setupNormalizeEnv(t)
|
||||
paths := env.ArtifactStore.SessionPaths(m.SessionID)
|
||||
paths := sessionPathsForEnv(env, m.SessionID)
|
||||
writeFile(t, filepath.Join(paths.TranscriptsDir, "processed.json"), `{"segments":[{"id":1}]}`)
|
||||
badOutput := filepath.Join(paths.TranscriptsDir, "normalized.bad.json")
|
||||
writeFile(t, badOutput, "not-json")
|
||||
@@ -196,7 +196,7 @@ func TestNormalizeStageFailsWhenNormalizedOutputInvalid(t *testing.T) {
|
||||
|
||||
func TestNormalizeStageReportEnabledFailsWhenReportMissing(t *testing.T) {
|
||||
env, m, ser := setupNormalizeEnv(t)
|
||||
paths := env.ArtifactStore.SessionPaths(m.SessionID)
|
||||
paths := sessionPathsForEnv(env, m.SessionID)
|
||||
writeFile(t, filepath.Join(paths.TranscriptsDir, "processed.json"), `{"segments":[{"id":1}]}`)
|
||||
report := true
|
||||
env.Config.Pipeline.Normalize.Report = &report
|
||||
@@ -211,6 +211,33 @@ func TestNormalizeStageReportEnabledFailsWhenReportMissing(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
func TestNormalizeStageUsesRunLocalPathsAndPromotesCanonical(t *testing.T) {
|
||||
env, m, ser := setupNormalizeEnv(t)
|
||||
env.Config.Session.Campaign = "sample-campaign"
|
||||
m.Campaign = "sample-campaign"
|
||||
m.RunID = "20260518T010203Z-abcdef12"
|
||||
paths := sessionPathsForEnv(env, m.SessionID)
|
||||
writeFile(t, filepath.Join(paths.TranscriptsDir, "processed.json"), `{"segments":[{"id":1}]}`)
|
||||
|
||||
result, err := (normalizeStage{}).Run(context.Background(), env, m)
|
||||
if err != nil {
|
||||
t.Fatalf("normalize.Run() error = %v", err)
|
||||
}
|
||||
if len(ser.NormalizeRequests) != 1 {
|
||||
t.Fatalf("normalize requests = %d, want 1", len(ser.NormalizeRequests))
|
||||
}
|
||||
req := ser.NormalizeRequests[0]
|
||||
if !strings.Contains(req.OutputNormalizedPath, filepath.Join("runs", m.RunID, "normalize", "outputs")) {
|
||||
t.Fatalf("run output path = %q, want run-local path", req.OutputNormalizedPath)
|
||||
}
|
||||
if len(result.Outputs) == 0 {
|
||||
t.Fatalf("outputs = %#v, want promoted outputs", result.Outputs)
|
||||
}
|
||||
if strings.Contains(result.Outputs[0].AbsolutePath, string(filepath.Separator)+"runs"+string(filepath.Separator)) {
|
||||
t.Fatalf("promoted output path = %q, want canonical session path", result.Outputs[0].AbsolutePath)
|
||||
}
|
||||
}
|
||||
|
||||
func setupNormalizeEnv(t *testing.T) (*Env, *manifest.Manifest, *seriatim.FakeRunner) {
|
||||
t.Helper()
|
||||
workspace := t.TempDir()
|
||||
@@ -243,7 +270,7 @@ func setupNormalizeEnv(t *testing.T) (*Env, *manifest.Manifest, *seriatim.FakeRu
|
||||
}
|
||||
|
||||
store := artifacts.NewLocalStore(workspace)
|
||||
if _, err := store.EnsureLayout("2026-05-03"); err != nil {
|
||||
if _, err := store.EnsureLayoutFor("sample-campaign", "2026-05-03"); err != nil {
|
||||
t.Fatalf("EnsureLayout() error = %v", err)
|
||||
}
|
||||
|
||||
|
||||
@@ -64,6 +64,7 @@ func TestStagesReturnExpectedMetadata(t *testing.T) {
|
||||
},
|
||||
Session: &config.SessionConfig{
|
||||
SessionID: "2026-05-03",
|
||||
Campaign: "sample-campaign",
|
||||
Inputs: config.SessionInputsConfig{
|
||||
AudioDir: "./audio",
|
||||
SpeakersFile: "./speakers.yml",
|
||||
@@ -88,7 +89,7 @@ func TestStagesReturnExpectedMetadata(t *testing.T) {
|
||||
}
|
||||
m.RunID = "20260516T000000Z-abcdef12"
|
||||
m.Campaign = "sample-campaign"
|
||||
m.LocalWorkDir = filepath.Join(root, "work", "sample-campaign", "2026-05-03", m.RunID)
|
||||
m.LocalWorkDir = artifacts.SessionRunRootForCampaign(root, "sample-campaign", "2026-05-03", m.RunID)
|
||||
m.S3RunPrefix = "dnd/campaigns/sample-campaign/sessions/2026-05-03/runs/" + m.RunID + "/"
|
||||
m.S3Bucket = "my-dnd-archive"
|
||||
if err := os.MkdirAll(filepath.Join(m.LocalWorkDir, "inputs"), 0o755); err != nil {
|
||||
@@ -224,7 +225,7 @@ func TestPlaceholderAdapterErrorPropagation(t *testing.T) {
|
||||
|
||||
root := t.TempDir()
|
||||
store := artifacts.NewLocalStore(root)
|
||||
_, err := store.EnsureLayout("2026-05-03")
|
||||
_, err := store.EnsureLayoutFor("sample-campaign", "2026-05-03")
|
||||
if err != nil {
|
||||
t.Fatalf("EnsureLayout() error = %v", err)
|
||||
}
|
||||
|
||||
@@ -55,7 +55,11 @@ func (polishStage) Run(ctx context.Context, env *Env, m *manifest.Manifest) (*St
|
||||
return nil, fmt.Errorf("polish: session id is required")
|
||||
}
|
||||
|
||||
paths := env.ArtifactStore.SessionPaths(sessionID)
|
||||
paths := sessionPathsForEnv(env, sessionID)
|
||||
runLayout, err := resolveRunStageLayout(env, m, paths, sessionID, "polish")
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("polish: resolve run-stage layout: %w", err)
|
||||
}
|
||||
mergedPath, source, err := discoverMergedTranscript(m, paths)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("polish: resolve merged transcript: %w", err)
|
||||
@@ -72,12 +76,29 @@ func (polishStage) Run(ctx context.Context, env *Env, m *manifest.Manifest) (*St
|
||||
return nil, fmt.Errorf("polish: %w", err)
|
||||
}
|
||||
|
||||
processedPath := filepath.Join(paths.TranscriptsDir, "processed.json")
|
||||
reportPath := filepath.Join(paths.ArtifactsDir, "audita.report.json")
|
||||
canonicalProcessedPath := filepath.Join(paths.TranscriptsDir, "processed.json")
|
||||
processedPath, err := runLocalPathForCanonical(runLayout, paths, canonicalProcessedPath)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("polish: resolve run-local processed transcript path: %w", err)
|
||||
}
|
||||
canonicalReportPath := filepath.Join(paths.ArtifactsDir, "audita.report.json")
|
||||
reportPath := canonicalReportPath
|
||||
if runLayout.Enabled {
|
||||
reportPath, err = runLocalPathForCanonical(runLayout, paths, canonicalReportPath)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("polish: resolve run-local report path: %w", err)
|
||||
}
|
||||
}
|
||||
workDir := filepath.Join(paths.ArtifactsDir, "audita-work")
|
||||
stdoutPath := filepath.Join(paths.LogsDir, "audita.stdout.log")
|
||||
stderrPath := filepath.Join(paths.LogsDir, "audita.stderr.log")
|
||||
generatedConfigPath := filepath.Join(paths.ConfigDir, "audita.generated.yml")
|
||||
if runLayout.Enabled {
|
||||
workDir = filepath.Join(runLayout.ScratchDir, "audita-work")
|
||||
stdoutPath = filepath.Join(runLayout.LogsDir, "audita.stdout.log")
|
||||
stderrPath = filepath.Join(runLayout.LogsDir, "audita.stderr.log")
|
||||
generatedConfigPath = filepath.Join(runLayout.ConfigDir, "audita.generated.yml")
|
||||
}
|
||||
|
||||
reportEnabled := env.Config.Pipeline.Audita.Report != nil && *env.Config.Pipeline.Audita.Report
|
||||
req := audita.PolishRequest{
|
||||
@@ -128,19 +149,25 @@ func (polishStage) Run(ctx context.Context, env *Env, m *manifest.Manifest) (*St
|
||||
}
|
||||
}
|
||||
|
||||
outputs := []artifacts.Ref{{
|
||||
Kind: "transcript_processed",
|
||||
Category: "transcripts",
|
||||
SessionID: sessionID,
|
||||
AbsolutePath: finalProcessedPath,
|
||||
}}
|
||||
promotedProcessed, err := promoteRunLocalOutput(env.ArtifactStore, finalProcessedPath, canonicalProcessedPath, artifacts.Ref{
|
||||
Kind: "transcript_processed",
|
||||
Category: "transcripts",
|
||||
SessionID: sessionID,
|
||||
})
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("polish: promote processed transcript: %w", err)
|
||||
}
|
||||
outputs := []artifacts.Ref{promotedProcessed}
|
||||
if reportEnabled {
|
||||
outputs = append(outputs, artifacts.Ref{
|
||||
Kind: "audita_report",
|
||||
Category: "artifacts",
|
||||
SessionID: sessionID,
|
||||
AbsolutePath: finalReportPath,
|
||||
promotedReport, err := promoteRunLocalOutput(env.ArtifactStore, finalReportPath, canonicalReportPath, artifacts.Ref{
|
||||
Kind: "audita_report",
|
||||
Category: "artifacts",
|
||||
SessionID: sessionID,
|
||||
})
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("polish: promote report: %w", err)
|
||||
}
|
||||
outputs = append(outputs, promotedReport)
|
||||
}
|
||||
|
||||
var validationConcurrency any
|
||||
@@ -155,14 +182,20 @@ func (polishStage) Run(ctx context.Context, env *Env, m *manifest.Manifest) (*St
|
||||
if env.Config.Pipeline.Audita.ProposalLLMConcurrency != nil {
|
||||
proposalLLMConcurrency = *env.Config.Pipeline.Audita.ProposalLLMConcurrency
|
||||
}
|
||||
reportCanonicalPath := ""
|
||||
if reportEnabled {
|
||||
reportCanonicalPath = canonicalReportPath
|
||||
}
|
||||
|
||||
meta := map[string]any{
|
||||
"stage": "polish",
|
||||
"merged_transcript_path": mergedPath,
|
||||
"merged_transcript_source": source,
|
||||
"glossary_path": glossaryPath,
|
||||
"output_path": finalProcessedPath,
|
||||
"report_path": finalReportPath,
|
||||
"run_output_path": finalProcessedPath,
|
||||
"output_path": canonicalProcessedPath,
|
||||
"run_report_path": finalReportPath,
|
||||
"report_path": reportCanonicalPath,
|
||||
"audita_work_dir": workDir,
|
||||
"report_enabled": reportEnabled,
|
||||
"modules": append([]string(nil), req.Modules...),
|
||||
|
||||
@@ -16,7 +16,7 @@ import (
|
||||
|
||||
func TestPolishStagePolishesMergedTranscriptAndRecordsMetadata(t *testing.T) {
|
||||
env, m := setupPolishEnv(t)
|
||||
paths := env.ArtifactStore.SessionPaths(m.SessionID)
|
||||
paths := sessionPathsForEnv(env, m.SessionID)
|
||||
|
||||
mergedPath := filepath.Join(paths.TranscriptsDir, "merged.json")
|
||||
writeFile(t, mergedPath, `{"segments":[]}`)
|
||||
@@ -120,7 +120,7 @@ func TestPolishStagePolishesMergedTranscriptAndRecordsMetadata(t *testing.T) {
|
||||
|
||||
func TestPolishStageFallsBackToMergedTranscriptPath(t *testing.T) {
|
||||
env, m := setupPolishEnv(t)
|
||||
paths := env.ArtifactStore.SessionPaths(m.SessionID)
|
||||
paths := sessionPathsForEnv(env, m.SessionID)
|
||||
mergedPath := filepath.Join(paths.TranscriptsDir, "merged.json")
|
||||
writeFile(t, mergedPath, `{"segments":[]}`)
|
||||
writeFile(t, filepath.Join(paths.InputsDir, "glossary.yml"), "terms: []\n")
|
||||
@@ -142,7 +142,7 @@ func TestPolishStageFallsBackToMergedTranscriptPath(t *testing.T) {
|
||||
|
||||
func TestPolishStageFailsWhenMergedTranscriptMissing(t *testing.T) {
|
||||
env, m := setupPolishEnv(t)
|
||||
paths := env.ArtifactStore.SessionPaths(m.SessionID)
|
||||
paths := sessionPathsForEnv(env, m.SessionID)
|
||||
writeFile(t, filepath.Join(paths.InputsDir, "glossary.yml"), "terms: []\n")
|
||||
env.Audita = &audita.FakeRunner{}
|
||||
|
||||
@@ -157,7 +157,7 @@ func TestPolishStageFailsWhenMergedTranscriptMissing(t *testing.T) {
|
||||
|
||||
func TestPolishStageFailsWhenMergedTranscriptInvalidJSON(t *testing.T) {
|
||||
env, m := setupPolishEnv(t)
|
||||
paths := env.ArtifactStore.SessionPaths(m.SessionID)
|
||||
paths := sessionPathsForEnv(env, m.SessionID)
|
||||
writeFile(t, filepath.Join(paths.TranscriptsDir, "merged.json"), "not-json")
|
||||
writeFile(t, filepath.Join(paths.InputsDir, "glossary.yml"), "terms: []\n")
|
||||
env.Audita = &audita.FakeRunner{}
|
||||
@@ -173,7 +173,7 @@ func TestPolishStageFailsWhenMergedTranscriptInvalidJSON(t *testing.T) {
|
||||
|
||||
func TestPolishStageFailsWhenGlossaryMissing(t *testing.T) {
|
||||
env, m := setupPolishEnv(t)
|
||||
paths := env.ArtifactStore.SessionPaths(m.SessionID)
|
||||
paths := sessionPathsForEnv(env, m.SessionID)
|
||||
writeFile(t, filepath.Join(paths.TranscriptsDir, "merged.json"), `{"segments":[]}`)
|
||||
env.Audita = &audita.FakeRunner{}
|
||||
|
||||
@@ -188,7 +188,7 @@ func TestPolishStageFailsWhenGlossaryMissing(t *testing.T) {
|
||||
|
||||
func TestPolishStageFailsWhenAdapterFails(t *testing.T) {
|
||||
env, m := setupPolishEnv(t)
|
||||
paths := env.ArtifactStore.SessionPaths(m.SessionID)
|
||||
paths := sessionPathsForEnv(env, m.SessionID)
|
||||
writeFile(t, filepath.Join(paths.TranscriptsDir, "merged.json"), `{"segments":[]}`)
|
||||
writeFile(t, filepath.Join(paths.InputsDir, "glossary.yml"), "terms: []\n")
|
||||
env.Audita = &audita.FakeRunner{Err: errors.New("audita failed")}
|
||||
@@ -204,7 +204,7 @@ func TestPolishStageFailsWhenAdapterFails(t *testing.T) {
|
||||
|
||||
func TestPolishStageFailsWhenProcessedOutputInvalid(t *testing.T) {
|
||||
env, m := setupPolishEnv(t)
|
||||
paths := env.ArtifactStore.SessionPaths(m.SessionID)
|
||||
paths := sessionPathsForEnv(env, m.SessionID)
|
||||
writeFile(t, filepath.Join(paths.TranscriptsDir, "merged.json"), `{"segments":[]}`)
|
||||
writeFile(t, filepath.Join(paths.InputsDir, "glossary.yml"), "terms: []\n")
|
||||
badPath := filepath.Join(paths.TranscriptsDir, "processed.invalid.json")
|
||||
@@ -222,7 +222,7 @@ func TestPolishStageFailsWhenProcessedOutputInvalid(t *testing.T) {
|
||||
|
||||
func TestPolishStageFailsWhenReportInvalid(t *testing.T) {
|
||||
env, m := setupPolishEnv(t)
|
||||
paths := env.ArtifactStore.SessionPaths(m.SessionID)
|
||||
paths := sessionPathsForEnv(env, m.SessionID)
|
||||
writeFile(t, filepath.Join(paths.TranscriptsDir, "merged.json"), `{"segments":[]}`)
|
||||
writeFile(t, filepath.Join(paths.InputsDir, "glossary.yml"), "terms: []\n")
|
||||
badReport := filepath.Join(paths.ArtifactsDir, "bad.report.json")
|
||||
@@ -238,6 +238,36 @@ func TestPolishStageFailsWhenReportInvalid(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
func TestPolishStageUsesRunLocalPathsAndPromotesCanonical(t *testing.T) {
|
||||
env, m := setupPolishEnv(t)
|
||||
env.Config.Session.Campaign = "sample-campaign"
|
||||
m.Campaign = "sample-campaign"
|
||||
m.RunID = "20260518T010203Z-abcdef12"
|
||||
paths := sessionPathsForEnv(env, m.SessionID)
|
||||
writeFile(t, filepath.Join(paths.TranscriptsDir, "merged.json"), `{"segments":[]}`)
|
||||
writeFile(t, filepath.Join(paths.InputsDir, "glossary.yml"), "terms: []\n")
|
||||
|
||||
fake := &audita.FakeRunner{}
|
||||
env.Audita = fake
|
||||
result, err := (polishStage{}).Run(context.Background(), env, m)
|
||||
if err != nil {
|
||||
t.Fatalf("polish.Run() error = %v", err)
|
||||
}
|
||||
if len(fake.Requests) != 1 {
|
||||
t.Fatalf("requests = %d, want 1", len(fake.Requests))
|
||||
}
|
||||
req := fake.Requests[0]
|
||||
if !strings.Contains(req.OutputProcessedPath, filepath.Join("runs", m.RunID, "polish", "outputs")) {
|
||||
t.Fatalf("run output path = %q, want run-local path", req.OutputProcessedPath)
|
||||
}
|
||||
if len(result.Outputs) == 0 {
|
||||
t.Fatalf("outputs = %#v, want promoted outputs", result.Outputs)
|
||||
}
|
||||
if strings.Contains(result.Outputs[0].AbsolutePath, string(filepath.Separator)+"runs"+string(filepath.Separator)) {
|
||||
t.Fatalf("promoted output path = %q, want canonical session path", result.Outputs[0].AbsolutePath)
|
||||
}
|
||||
}
|
||||
|
||||
func setupPolishEnv(t *testing.T) (*Env, *manifest.Manifest) {
|
||||
t.Helper()
|
||||
workspace := t.TempDir()
|
||||
@@ -284,7 +314,7 @@ func setupPolishEnv(t *testing.T) (*Env, *manifest.Manifest) {
|
||||
}
|
||||
|
||||
store := artifacts.NewLocalStore(workspace)
|
||||
if _, err := store.EnsureLayout("2026-05-03"); err != nil {
|
||||
if _, err := store.EnsureLayoutFor("sample-campaign", "2026-05-03"); err != nil {
|
||||
t.Fatalf("EnsureLayout() error = %v", err)
|
||||
}
|
||||
return &Env{
|
||||
|
||||
@@ -54,7 +54,7 @@ func (prepareStage) Run(ctx context.Context, env *Env, m *manifest.Manifest) (*S
|
||||
return nil, fmt.Errorf("prepare: session id is required")
|
||||
}
|
||||
|
||||
paths, err := env.ArtifactStore.EnsureLayout(sessionID)
|
||||
paths, err := ensureLayoutForEnv(env, sessionID)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("prepare: ensure workdir layout: %w", err)
|
||||
}
|
||||
@@ -358,6 +358,10 @@ func pathsWorkDirForManifest(env *Env, m *manifest.Manifest, sessionID string) s
|
||||
if env == nil || env.Config == nil || env.Config.Pipeline == nil || env.Config.Session == nil {
|
||||
return ""
|
||||
}
|
||||
campaign := strings.TrimSpace(env.Config.Session.Campaign)
|
||||
if campaign == "" {
|
||||
return ""
|
||||
}
|
||||
if m != nil && strings.TrimSpace(m.LocalWorkDir) != "" {
|
||||
return strings.TrimSpace(m.LocalWorkDir)
|
||||
}
|
||||
@@ -366,9 +370,9 @@ func pathsWorkDirForManifest(env *Env, m *manifest.Manifest, sessionID string) s
|
||||
runID = strings.TrimSpace(m.RunID)
|
||||
}
|
||||
if runID != "" {
|
||||
return artifacts.SessionRunWorkDir(env.Config.Pipeline.Workspace.Root, env.Config.Session.Campaign, sessionID, runID)
|
||||
return artifacts.SessionRunRootForCampaign(env.Config.Pipeline.Workspace.Root, campaign, sessionID, runID)
|
||||
}
|
||||
return artifacts.SessionWorkDir(env.Config.Pipeline.Workspace.Root, sessionID)
|
||||
return artifacts.SessionWorkDirForCampaign(env.Config.Pipeline.Workspace.Root, campaign, sessionID)
|
||||
}
|
||||
|
||||
func resolvePath(baseDir, p string) (string, error) {
|
||||
|
||||
@@ -33,7 +33,7 @@ func TestPrepareStageExplicitAudioFiles(t *testing.T) {
|
||||
t.Fatalf("result metadata = %#v, want prepared=true", result)
|
||||
}
|
||||
|
||||
paths := env.ArtifactStore.SessionPaths(m.SessionID)
|
||||
paths := sessionPathsForEnv(env, m.SessionID)
|
||||
for _, p := range []string{
|
||||
filepath.Join(paths.InputsDir, "session.yml"),
|
||||
filepath.Join(paths.InputsDir, "pipeline.resolved.yml"),
|
||||
@@ -76,7 +76,7 @@ func TestPrepareStageAudioDirEnumeration(t *testing.T) {
|
||||
t.Fatalf("audio_files_resolved = %#v, want 1", got)
|
||||
}
|
||||
|
||||
paths := env.ArtifactStore.SessionPaths(m.SessionID)
|
||||
paths := sessionPathsForEnv(env, m.SessionID)
|
||||
if _, err := os.Stat(filepath.Join(paths.AudioDir, "a.flac")); err != nil {
|
||||
t.Fatalf("expected copied flac: %v", err)
|
||||
}
|
||||
@@ -160,7 +160,7 @@ func TestPrepareStageS3AudioDownloadAndMaterialization(t *testing.T) {
|
||||
env.Config.Pipeline.Spool = config.SpoolConfig{Root: filepath.Join(t.TempDir(), "spool")}
|
||||
env.Config.Pipeline.Storage.S3 = &config.StorageS3Config{Bucket: "my-dnd-archive", RootPrefix: "dnd"}
|
||||
m.RunID = "20260515T031522Z-a1b2c3d4"
|
||||
m.LocalWorkDir = artifacts.SessionRunWorkDir(env.Config.Pipeline.Workspace.Root, "forsaken", m.SessionID, m.RunID)
|
||||
m.LocalWorkDir = artifacts.SessionRunRootForCampaign(env.Config.Pipeline.Workspace.Root, "forsaken", m.SessionID, m.RunID)
|
||||
m.LocalSpoolDir = artifacts.SessionSpoolAudioDir(env.Config.Pipeline.Spool.Root, "forsaken", m.SessionID, m.RunID)
|
||||
|
||||
fake := &storage.FakeBackend{}
|
||||
@@ -256,7 +256,7 @@ func TestPrepareStageS3AudioFailures(t *testing.T) {
|
||||
env.Config.Pipeline.Spool = config.SpoolConfig{Root: filepath.Join(t.TempDir(), "spool")}
|
||||
env.Config.Pipeline.Storage.S3 = &config.StorageS3Config{Bucket: "my-dnd-archive", RootPrefix: "dnd"}
|
||||
m.RunID = "20260515T031522Z-a1b2c3d4"
|
||||
m.LocalWorkDir = artifacts.SessionRunWorkDir(env.Config.Pipeline.Workspace.Root, "forsaken", m.SessionID, m.RunID)
|
||||
m.LocalWorkDir = artifacts.SessionRunRootForCampaign(env.Config.Pipeline.Workspace.Root, "forsaken", m.SessionID, m.RunID)
|
||||
m.LocalSpoolDir = artifacts.SessionSpoolAudioDir(env.Config.Pipeline.Spool.Root, "forsaken", m.SessionID, m.RunID)
|
||||
|
||||
fake := &storage.FakeBackend{}
|
||||
|
||||
136
internal/stage/run_local.go
Normal file
136
internal/stage/run_local.go
Normal file
@@ -0,0 +1,136 @@
|
||||
package stage
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
|
||||
"gitea.maximumdirect.net/eric/narratio/internal/artifacts"
|
||||
"gitea.maximumdirect.net/eric/narratio/internal/manifest"
|
||||
)
|
||||
|
||||
type runStageLayout struct {
|
||||
Enabled bool
|
||||
Root string
|
||||
OutputsDir string
|
||||
LogsDir string
|
||||
ReportsDir string
|
||||
ConfigDir string
|
||||
ScratchDir string
|
||||
}
|
||||
|
||||
func resolveRunStageLayout(
|
||||
env *Env,
|
||||
m *manifest.Manifest,
|
||||
sessionPaths artifacts.SessionPaths,
|
||||
sessionID, stageName string,
|
||||
) (runStageLayout, error) {
|
||||
if env == nil || env.Config == nil || env.Config.Pipeline == nil {
|
||||
return runStageLayout{}, fmt.Errorf("stage environment pipeline config is required")
|
||||
}
|
||||
if strings.TrimSpace(sessionID) == "" {
|
||||
return runStageLayout{}, fmt.Errorf("session id is required")
|
||||
}
|
||||
stageName = strings.TrimSpace(stageName)
|
||||
if stageName == "" {
|
||||
return runStageLayout{}, fmt.Errorf("stage name is required")
|
||||
}
|
||||
|
||||
runID := ""
|
||||
if m != nil {
|
||||
runID = strings.TrimSpace(m.RunID)
|
||||
}
|
||||
campaign := strings.TrimSpace(env.Config.Session.Campaign)
|
||||
if campaign == "" && m != nil {
|
||||
campaign = strings.TrimSpace(m.Campaign)
|
||||
}
|
||||
|
||||
// Compatibility fallback for direct stage tests and older call paths
|
||||
// that execute a stage without a run id.
|
||||
if runID == "" || campaign == "" {
|
||||
return runStageLayout{}, nil
|
||||
}
|
||||
|
||||
root := artifacts.SessionRunStageDirForCampaign(
|
||||
env.Config.Pipeline.Workspace.Root,
|
||||
campaign,
|
||||
sessionID,
|
||||
runID,
|
||||
stageName,
|
||||
)
|
||||
layout := runStageLayout{
|
||||
Enabled: true,
|
||||
Root: root,
|
||||
OutputsDir: filepath.Join(root, "outputs"),
|
||||
LogsDir: filepath.Join(root, "logs"),
|
||||
ReportsDir: filepath.Join(root, "reports"),
|
||||
ConfigDir: filepath.Join(root, "config"),
|
||||
ScratchDir: filepath.Join(root, "scratch"),
|
||||
}
|
||||
for _, dir := range []string{
|
||||
layout.Root,
|
||||
layout.OutputsDir,
|
||||
layout.LogsDir,
|
||||
layout.ReportsDir,
|
||||
layout.ConfigDir,
|
||||
layout.ScratchDir,
|
||||
} {
|
||||
if err := os.MkdirAll(dir, 0o755); err != nil {
|
||||
return runStageLayout{}, fmt.Errorf("create run-stage directory %q: %w", dir, err)
|
||||
}
|
||||
}
|
||||
return layout, nil
|
||||
}
|
||||
|
||||
func runLocalPathForCanonical(layout runStageLayout, sessionPaths artifacts.SessionPaths, canonicalPath string) (string, error) {
|
||||
if !layout.Enabled {
|
||||
return filepath.Clean(canonicalPath), nil
|
||||
}
|
||||
cleanCanonical := filepath.Clean(strings.TrimSpace(canonicalPath))
|
||||
if cleanCanonical == "" {
|
||||
return "", fmt.Errorf("canonical path is required")
|
||||
}
|
||||
rel, err := filepath.Rel(filepath.Clean(sessionPaths.Root), cleanCanonical)
|
||||
if err != nil {
|
||||
return "", fmt.Errorf("derive session-relative path for %q: %w", cleanCanonical, err)
|
||||
}
|
||||
rel = filepath.Clean(rel)
|
||||
if rel == "." || rel == ".." || strings.HasPrefix(rel, ".."+string(filepath.Separator)) {
|
||||
return "", fmt.Errorf("canonical path %q is outside session root %q", cleanCanonical, sessionPaths.Root)
|
||||
}
|
||||
localPath := filepath.Join(layout.OutputsDir, rel)
|
||||
if err := os.MkdirAll(filepath.Dir(localPath), 0o755); err != nil {
|
||||
return "", fmt.Errorf("create run-local output parent for %q: %w", localPath, err)
|
||||
}
|
||||
return localPath, nil
|
||||
}
|
||||
|
||||
func promoteRunLocalOutput(
|
||||
store artifacts.Store,
|
||||
srcPath, canonicalPath string,
|
||||
ref artifacts.Ref,
|
||||
) (artifacts.Ref, error) {
|
||||
srcPath = filepath.Clean(strings.TrimSpace(srcPath))
|
||||
canonicalPath = filepath.Clean(strings.TrimSpace(canonicalPath))
|
||||
if srcPath == "" {
|
||||
return artifacts.Ref{}, fmt.Errorf("source path is required")
|
||||
}
|
||||
if canonicalPath == "" {
|
||||
return artifacts.Ref{}, fmt.Errorf("canonical destination path is required")
|
||||
}
|
||||
data, err := os.ReadFile(srcPath)
|
||||
if err != nil {
|
||||
return artifacts.Ref{}, fmt.Errorf("read run-local output %q: %w", srcPath, err)
|
||||
}
|
||||
if err := store.WriteFileAtomic(canonicalPath, data, 0o644); err != nil {
|
||||
return artifacts.Ref{}, fmt.Errorf("promote output to %q: %w", canonicalPath, err)
|
||||
}
|
||||
checksum, err := store.Checksum(canonicalPath)
|
||||
if err != nil {
|
||||
return artifacts.Ref{}, fmt.Errorf("checksum promoted output %q: %w", canonicalPath, err)
|
||||
}
|
||||
ref.AbsolutePath = canonicalPath
|
||||
ref.Checksum = checksum
|
||||
return ref, nil
|
||||
}
|
||||
35
internal/stage/run_local_test.go
Normal file
35
internal/stage/run_local_test.go
Normal file
@@ -0,0 +1,35 @@
|
||||
package stage
|
||||
|
||||
import (
|
||||
"os"
|
||||
"path/filepath"
|
||||
"testing"
|
||||
|
||||
"gitea.maximumdirect.net/eric/narratio/internal/artifacts"
|
||||
)
|
||||
|
||||
func TestRunLocalPathForCanonicalCreatesParentDirectories(t *testing.T) {
|
||||
root := t.TempDir()
|
||||
sessionRoot := filepath.Join(root, "work", "dilfs", "2026-05-17")
|
||||
layout := runStageLayout{
|
||||
Enabled: true,
|
||||
OutputsDir: filepath.Join(sessionRoot, "runs", "run-1", "merge", "outputs"),
|
||||
}
|
||||
if err := os.MkdirAll(layout.OutputsDir, 0o755); err != nil {
|
||||
t.Fatalf("mkdir outputs dir: %v", err)
|
||||
}
|
||||
|
||||
canonical := filepath.Join(sessionRoot, "transcripts", "merged.json")
|
||||
got, err := runLocalPathForCanonical(layout, artifacts.SessionPaths{Root: sessionRoot}, canonical)
|
||||
if err != nil {
|
||||
t.Fatalf("runLocalPathForCanonical() error = %v", err)
|
||||
}
|
||||
|
||||
want := filepath.Join(layout.OutputsDir, "transcripts", "merged.json")
|
||||
if got != want {
|
||||
t.Fatalf("runLocalPathForCanonical() = %q, want %q", got, want)
|
||||
}
|
||||
if _, err := os.Stat(filepath.Dir(got)); err != nil {
|
||||
t.Fatalf("expected run-local parent directory to exist: %v", err)
|
||||
}
|
||||
}
|
||||
23
internal/stage/session_paths.go
Normal file
23
internal/stage/session_paths.go
Normal file
@@ -0,0 +1,23 @@
|
||||
package stage
|
||||
|
||||
import (
|
||||
"strings"
|
||||
|
||||
"gitea.maximumdirect.net/eric/narratio/internal/artifacts"
|
||||
)
|
||||
|
||||
func sessionPathsForEnv(env *Env, sessionID string) artifacts.SessionPaths {
|
||||
campaign := ""
|
||||
if env != nil && env.Config != nil && env.Config.Session != nil {
|
||||
campaign = strings.TrimSpace(env.Config.Session.Campaign)
|
||||
}
|
||||
return env.ArtifactStore.SessionPathsFor(campaign, sessionID)
|
||||
}
|
||||
|
||||
func ensureLayoutForEnv(env *Env, sessionID string) (artifacts.SessionPaths, error) {
|
||||
campaign := ""
|
||||
if env != nil && env.Config != nil && env.Config.Session != nil {
|
||||
campaign = strings.TrimSpace(env.Config.Session.Campaign)
|
||||
}
|
||||
return env.ArtifactStore.EnsureLayoutFor(campaign, sessionID)
|
||||
}
|
||||
@@ -55,7 +55,11 @@ func (transcribeStage) Run(ctx context.Context, env *Env, m *manifest.Manifest)
|
||||
return nil, fmt.Errorf("transcribe: session id is required")
|
||||
}
|
||||
|
||||
paths := env.ArtifactStore.SessionPaths(sessionID)
|
||||
paths := sessionPathsForEnv(env, sessionID)
|
||||
runLayout, err := resolveRunStageLayout(env, m, paths, sessionID, "transcribe")
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("transcribe: resolve run-stage layout: %w", err)
|
||||
}
|
||||
audioFiles, err := discoverPreparedAudio(m, paths.AudioDir)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("transcribe: resolve audio inputs: %w", err)
|
||||
@@ -88,10 +92,15 @@ func (transcribeStage) Run(ctx context.Context, env *Env, m *manifest.Manifest)
|
||||
return nil, fmt.Errorf("transcribe: duplicate speaker/audio basename %q from %q and %q", base, prev, audioPath)
|
||||
}
|
||||
seenSpeaker[base] = audioPath
|
||||
canonicalOut := filepath.Join(paths.TranscriptsRawDir, base+".json")
|
||||
runOut, err := runLocalPathForCanonical(runLayout, paths, canonicalOut)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("transcribe: resolve run-local output path for %q: %w", base, err)
|
||||
}
|
||||
jobs = append(jobs, job{
|
||||
speakerID: base,
|
||||
audioPath: audioPath,
|
||||
outPath: filepath.Join(paths.TranscriptsRawDir, base+".json"),
|
||||
outPath: runOut,
|
||||
})
|
||||
}
|
||||
|
||||
@@ -197,12 +206,19 @@ dispatch:
|
||||
sort.Strings(speakers)
|
||||
|
||||
outputs := make([]artifacts.Ref, 0, len(speakers))
|
||||
runOutputPaths := make([]string, 0, len(speakers))
|
||||
outputPaths := make([]string, 0, len(speakers))
|
||||
orderedPerFile := make(map[string]any, len(speakers))
|
||||
for _, speaker := range speakers {
|
||||
ref := outputRef[speaker]
|
||||
outputs = append(outputs, ref)
|
||||
outputPaths = append(outputPaths, ref.AbsolutePath)
|
||||
runOutputPaths = append(runOutputPaths, ref.AbsolutePath)
|
||||
canonicalOut := filepath.Join(paths.TranscriptsRawDir, speaker+".json")
|
||||
promoted, err := promoteRunLocalOutput(env.ArtifactStore, ref.AbsolutePath, canonicalOut, ref)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("transcribe: promote %q output: %w", speaker, err)
|
||||
}
|
||||
outputs = append(outputs, promoted)
|
||||
outputPaths = append(outputPaths, canonicalOut)
|
||||
orderedPerFile[speaker] = perFile[speaker]
|
||||
}
|
||||
|
||||
@@ -221,6 +237,7 @@ dispatch:
|
||||
"retries": retries,
|
||||
"retry_delay": env.Config.Pipeline.WhisperX.RetryDelay,
|
||||
"timeout": env.Config.Pipeline.WhisperX.Timeout,
|
||||
"run_output_paths": runOutputPaths,
|
||||
"output_paths": outputPaths,
|
||||
"per_file": orderedPerFile,
|
||||
},
|
||||
|
||||
@@ -53,8 +53,8 @@ func TestTranscribeStageTranscribesPreparedAudio(t *testing.T) {
|
||||
}
|
||||
sort.Strings(gotPaths)
|
||||
wantPaths := []string{
|
||||
filepath.Join(env.ArtifactStore.SessionPaths(m.SessionID).TranscriptsRawDir, "alice.json"),
|
||||
filepath.Join(env.ArtifactStore.SessionPaths(m.SessionID).TranscriptsRawDir, "bob.json"),
|
||||
filepath.Join(sessionPathsForEnv(env, m.SessionID).TranscriptsRawDir, "alice.json"),
|
||||
filepath.Join(sessionPathsForEnv(env, m.SessionID).TranscriptsRawDir, "bob.json"),
|
||||
}
|
||||
sort.Strings(wantPaths)
|
||||
if strings.Join(gotPaths, "|") != strings.Join(wantPaths, "|") {
|
||||
@@ -191,6 +191,36 @@ func TestTranscribeStageInvalidJSONFails(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
func TestTranscribeStageUsesRunLocalOutputAndPromotesCanonical(t *testing.T) {
|
||||
env, m := setupTranscribeEnv(t, []string{"alice.flac"})
|
||||
env.Config.Session.Campaign = "sample-campaign"
|
||||
m.Campaign = "sample-campaign"
|
||||
m.RunID = "20260518T010203Z-abcdef12"
|
||||
fake := &whisperx.FakeClient{}
|
||||
env.WhisperX = fake
|
||||
|
||||
if _, err := (prepareStage{}).Run(context.Background(), env, m); err != nil {
|
||||
t.Fatalf("prepare.Run() error = %v", err)
|
||||
}
|
||||
result, err := (transcribeStage{}).Run(context.Background(), env, m)
|
||||
if err != nil {
|
||||
t.Fatalf("transcribe.Run() error = %v", err)
|
||||
}
|
||||
if len(fake.Requests) != 1 {
|
||||
t.Fatalf("requests = %d, want 1", len(fake.Requests))
|
||||
}
|
||||
runOut := fake.Requests[0].OutputRawTranscriptPath
|
||||
if !strings.Contains(runOut, filepath.Join("runs", m.RunID, "transcribe", "outputs")) {
|
||||
t.Fatalf("run-local output path = %q, want runs/{run_id}/transcribe/outputs path", runOut)
|
||||
}
|
||||
if len(result.Outputs) != 1 {
|
||||
t.Fatalf("outputs = %#v, want one output", result.Outputs)
|
||||
}
|
||||
if strings.Contains(result.Outputs[0].AbsolutePath, string(filepath.Separator)+"runs"+string(filepath.Separator)) {
|
||||
t.Fatalf("promoted output path = %q, want canonical session path", result.Outputs[0].AbsolutePath)
|
||||
}
|
||||
}
|
||||
|
||||
func setupTranscribeEnv(t *testing.T, audioFiles []string) (*Env, *manifest.Manifest) {
|
||||
t.Helper()
|
||||
|
||||
@@ -203,7 +233,7 @@ func setupTranscribeEnv(t *testing.T, audioFiles []string) (*Env, *manifest.Mani
|
||||
|
||||
sessionPath := filepath.Join(cfgDir, "session.yml")
|
||||
pipelinePath := filepath.Join(cfgDir, "pipeline.yml")
|
||||
writeFile(t, sessionPath, "session_id: 2026-05-03\n")
|
||||
writeFile(t, sessionPath, "session_id: 2026-05-03\ncampaign: sample-campaign\n")
|
||||
writeFile(t, pipelinePath, "workspace:\n root: "+workspace+"\n")
|
||||
writeFile(t, filepath.Join(cfgDir, "speakers.yml"), "alice: alice.flac\n")
|
||||
writeFile(t, filepath.Join(cfgDir, "autocorrect.yml"), "[]\n")
|
||||
@@ -227,6 +257,7 @@ func setupTranscribeEnv(t *testing.T, audioFiles []string) (*Env, *manifest.Mani
|
||||
},
|
||||
Session: &config.SessionConfig{
|
||||
SessionID: "2026-05-03",
|
||||
Campaign: "sample-campaign",
|
||||
Inputs: config.SessionInputsConfig{
|
||||
AudioDir: "./audio",
|
||||
SpeakersFile: "./speakers.yml",
|
||||
|
||||
@@ -54,7 +54,11 @@ func (trimStage) Run(ctx context.Context, env *Env, m *manifest.Manifest) (*Stag
|
||||
return nil, fmt.Errorf("trim: session id is required")
|
||||
}
|
||||
|
||||
paths := env.ArtifactStore.SessionPaths(sessionID)
|
||||
paths := sessionPathsForEnv(env, sessionID)
|
||||
runLayout, err := resolveRunStageLayout(env, m, paths, sessionID, "trim")
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("trim: resolve run-stage layout: %w", err)
|
||||
}
|
||||
normalizedPath, normalizedSource, err := discoverNormalizedTranscript(m, paths)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("trim: resolve normalized transcript: %w", err)
|
||||
@@ -69,10 +73,14 @@ func (trimStage) Run(ctx context.Context, env *Env, m *manifest.Manifest) (*Stag
|
||||
trimCfg := env.Config.Pipeline.Trim
|
||||
enabled := trimCfg != nil && trimCfg.Enabled
|
||||
|
||||
trimmedPath, err := resolveTrimmedOutputPath(paths, trimCfg)
|
||||
canonicalTrimmedPath, err := resolveTrimmedOutputPath(paths, trimCfg)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("trim: resolve trimmed output path: %w", err)
|
||||
}
|
||||
trimmedPath, err := runLocalPathForCanonical(runLayout, paths, canonicalTrimmedPath)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("trim: resolve run-local trimmed output path: %w", err)
|
||||
}
|
||||
|
||||
logPaths := []string{}
|
||||
generatedConfigs := []string{}
|
||||
@@ -81,7 +89,8 @@ func (trimStage) Run(ctx context.Context, env *Env, m *manifest.Manifest) (*Stag
|
||||
"trim_enabled": enabled,
|
||||
"normalized_transcript_path": normalizedPath,
|
||||
"normalized_transcript_source": normalizedSource,
|
||||
"trimmed_output_path": trimmedPath,
|
||||
"run_trimmed_output_path": trimmedPath,
|
||||
"trimmed_output_path": canonicalTrimmedPath,
|
||||
}
|
||||
|
||||
if !enabled {
|
||||
@@ -91,14 +100,17 @@ func (trimStage) Run(ctx context.Context, env *Env, m *manifest.Manifest) (*Stag
|
||||
if err := validateProcessedTranscriptOutput(trimmedPath); err != nil {
|
||||
return nil, fmt.Errorf("trim: copied trimmed transcript %q invalid: %w", trimmedPath, err)
|
||||
}
|
||||
promotedTrimmed, err := promoteRunLocalOutput(env.ArtifactStore, trimmedPath, canonicalTrimmedPath, artifacts.Ref{
|
||||
Kind: "transcript_trimmed",
|
||||
Category: "transcripts",
|
||||
SessionID: sessionID,
|
||||
})
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("trim: promote trimmed transcript: %w", err)
|
||||
}
|
||||
metadata["trim_action"] = "copy_disabled"
|
||||
return &StageResult{
|
||||
Outputs: []artifacts.Ref{{
|
||||
Kind: "transcript_trimmed",
|
||||
Category: "transcripts",
|
||||
SessionID: sessionID,
|
||||
AbsolutePath: trimmedPath,
|
||||
}},
|
||||
Outputs: []artifacts.Ref{promotedTrimmed},
|
||||
Metadata: metadata,
|
||||
}, nil
|
||||
}
|
||||
@@ -114,13 +126,22 @@ func (trimStage) Run(ctx context.Context, env *Env, m *manifest.Manifest) (*Stag
|
||||
}
|
||||
|
||||
boundsCfg := trimCfg.Bounds
|
||||
boundsOutputPath, err := resolveScriptoriumOutputPath(paths, boundsCfg.OutputPath)
|
||||
canonicalBoundsOutputPath, err := resolveScriptoriumOutputPath(paths, boundsCfg.OutputPath)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("trim: resolve bounds output path: %w", err)
|
||||
}
|
||||
boundsOutputPath, err := runLocalPathForCanonical(runLayout, paths, canonicalBoundsOutputPath)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("trim: resolve run-local bounds output path: %w", err)
|
||||
}
|
||||
boundsStdoutLogPath := filepath.Join(paths.LogsDir, "scriptorium.bounds.stdout.log")
|
||||
boundsStderrLogPath := filepath.Join(paths.LogsDir, "scriptorium.bounds.stderr.log")
|
||||
boundsGeneratedConfigPath := filepath.Join(paths.ConfigDir, "scriptorium.bounds.generated.yml")
|
||||
if runLayout.Enabled {
|
||||
boundsStdoutLogPath = filepath.Join(runLayout.LogsDir, "scriptorium.bounds.stdout.log")
|
||||
boundsStderrLogPath = filepath.Join(runLayout.LogsDir, "scriptorium.bounds.stderr.log")
|
||||
boundsGeneratedConfigPath = filepath.Join(runLayout.ConfigDir, "scriptorium.bounds.generated.yml")
|
||||
}
|
||||
boundsTimeout, err := resolveScriptoriumTimeout(env.Config.Pipeline.Scriptorium.Timeout, boundsCfg.Timeout)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("trim: resolve bounds timeout: %w", err)
|
||||
@@ -133,7 +154,8 @@ func (trimStage) Run(ctx context.Context, env *Env, m *manifest.Manifest) (*Stag
|
||||
|
||||
metadata["bounds_prompt_id"] = boundsCfg.PromptID
|
||||
metadata["bounds_profile_id"] = boundsCfg.ProfileID
|
||||
metadata["bounds_output_path"] = boundsOutputPath
|
||||
metadata["run_bounds_output_path"] = boundsOutputPath
|
||||
metadata["bounds_output_path"] = canonicalBoundsOutputPath
|
||||
metadata["bounds_timeout"] = boundsTimeout.String()
|
||||
metadata["bounds_input_name"] = boundsCfg.TranscriptInputName
|
||||
metadata["bounds_input_path"] = normalizedPath
|
||||
@@ -141,13 +163,22 @@ func (trimStage) Run(ctx context.Context, env *Env, m *manifest.Manifest) (*Stag
|
||||
|
||||
renderOutputPath := ""
|
||||
if boundsCfg.RenderDebug {
|
||||
renderOutputPath, err = resolveScriptoriumOutputPath(paths, boundsCfg.RenderOutputPath)
|
||||
canonicalRenderOutputPath, err := resolveScriptoriumOutputPath(paths, boundsCfg.RenderOutputPath)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("trim: resolve bounds render output path: %w", err)
|
||||
}
|
||||
renderOutputPath, err = runLocalPathForCanonical(runLayout, paths, canonicalRenderOutputPath)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("trim: resolve run-local bounds render output path: %w", err)
|
||||
}
|
||||
renderStdoutLogPath := filepath.Join(paths.LogsDir, "scriptorium.bounds.render.stdout.log")
|
||||
renderStderrLogPath := filepath.Join(paths.LogsDir, "scriptorium.bounds.render.stderr.log")
|
||||
renderGeneratedConfigPath := filepath.Join(paths.ConfigDir, "scriptorium.bounds.render.generated.yml")
|
||||
if runLayout.Enabled {
|
||||
renderStdoutLogPath = filepath.Join(runLayout.LogsDir, "scriptorium.bounds.render.stdout.log")
|
||||
renderStderrLogPath = filepath.Join(runLayout.LogsDir, "scriptorium.bounds.render.stderr.log")
|
||||
renderGeneratedConfigPath = filepath.Join(runLayout.ConfigDir, "scriptorium.bounds.render.generated.yml")
|
||||
}
|
||||
|
||||
renderReq := scriptorium.RenderArtifactRequest{
|
||||
Binary: env.Config.Pipeline.Scriptorium.Binary,
|
||||
@@ -254,7 +285,7 @@ func (trimStage) Run(ctx context.Context, env *Env, m *manifest.Manifest) (*Stag
|
||||
metadata["end_segment_id"] = boundsPayload.EndSegmentID
|
||||
metadata["warnings"] = boundsPayload.Warnings
|
||||
metadata["keep_selector"] = keepSelector
|
||||
metadata["bounds_output_path"] = finalBoundsOutputPath
|
||||
metadata["run_bounds_output_path"] = finalBoundsOutputPath
|
||||
metadata["bounds_stdout_log_path"] = boundsStdoutLogPath
|
||||
metadata["bounds_stderr_log_path"] = boundsStderrLogPath
|
||||
metadata["bounds_generated_config_path"] = boundsGeneratedConfigPath
|
||||
@@ -279,6 +310,11 @@ func (trimStage) Run(ctx context.Context, env *Env, m *manifest.Manifest) (*Stag
|
||||
trimStdoutLogPath := filepath.Join(paths.LogsDir, "seriatim.trim.stdout.log")
|
||||
trimStderrLogPath := filepath.Join(paths.LogsDir, "seriatim.trim.stderr.log")
|
||||
trimGeneratedConfigPath := filepath.Join(paths.ConfigDir, "seriatim.trim.generated.yml")
|
||||
if runLayout.Enabled {
|
||||
trimStdoutLogPath = filepath.Join(runLayout.LogsDir, "seriatim.trim.stdout.log")
|
||||
trimStderrLogPath = filepath.Join(runLayout.LogsDir, "seriatim.trim.stderr.log")
|
||||
trimGeneratedConfigPath = filepath.Join(runLayout.ConfigDir, "seriatim.trim.generated.yml")
|
||||
}
|
||||
trimTimeout, err := resolveTrimSeriatimTimeout(env.Config.Pipeline.Seriatim.Timeout)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("trim: resolve seriatim timeout: %w", err)
|
||||
@@ -315,23 +351,25 @@ func (trimStage) Run(ctx context.Context, env *Env, m *manifest.Manifest) (*Stag
|
||||
return nil, fmt.Errorf("trim: trimmed transcript %q invalid: %w", trimmedPath, err)
|
||||
}
|
||||
|
||||
outputs := []artifacts.Ref{
|
||||
{
|
||||
Kind: "transcript_trimmed",
|
||||
Category: "transcripts",
|
||||
SessionID: sessionID,
|
||||
AbsolutePath: trimmedPath,
|
||||
},
|
||||
{
|
||||
Kind: "session_bounds",
|
||||
Category: "artifacts",
|
||||
SessionID: sessionID,
|
||||
AbsolutePath: finalBoundsOutputPath,
|
||||
},
|
||||
promotedTrimmed, err := promoteRunLocalOutput(env.ArtifactStore, trimmedPath, canonicalTrimmedPath, artifacts.Ref{
|
||||
Kind: "transcript_trimmed",
|
||||
Category: "transcripts",
|
||||
SessionID: sessionID,
|
||||
})
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("trim: promote trimmed transcript: %w", err)
|
||||
}
|
||||
promotedBounds, err := promoteRunLocalOutput(env.ArtifactStore, finalBoundsOutputPath, canonicalBoundsOutputPath, artifacts.Ref{
|
||||
Kind: "session_bounds",
|
||||
Category: "artifacts",
|
||||
SessionID: sessionID,
|
||||
})
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("trim: promote session bounds: %w", err)
|
||||
}
|
||||
|
||||
return &StageResult{
|
||||
Outputs: outputs,
|
||||
Outputs: []artifacts.Ref{promotedTrimmed, promotedBounds},
|
||||
Logs: logPaths,
|
||||
GeneratedConfigs: generatedConfigs,
|
||||
Metadata: metadata,
|
||||
|
||||
@@ -18,7 +18,7 @@ import (
|
||||
|
||||
func TestTrimStageConsumesNormalizedAndProducesTrimmedTranscript(t *testing.T) {
|
||||
env, m, scr, ser := setupTrimEnv(t)
|
||||
paths := env.ArtifactStore.SessionPaths(m.SessionID)
|
||||
paths := sessionPathsForEnv(env, m.SessionID)
|
||||
normalized := filepath.Join(paths.TranscriptsDir, "normalized.json")
|
||||
writeFile(t, normalized, `{"segments":[{"id":10},{"id":868}]}`)
|
||||
m.MarkStageSucceeded("normalize", time.Now().UTC(), []manifest.ArtifactRecord{
|
||||
@@ -70,7 +70,7 @@ func TestTrimStageConsumesNormalizedAndProducesTrimmedTranscript(t *testing.T) {
|
||||
|
||||
func TestTrimStageUsesConfiguredScriptoriumInputName(t *testing.T) {
|
||||
env, m, scr, _ := setupTrimEnv(t)
|
||||
paths := env.ArtifactStore.SessionPaths(m.SessionID)
|
||||
paths := sessionPathsForEnv(env, m.SessionID)
|
||||
normalized := filepath.Join(paths.TranscriptsDir, "normalized.json")
|
||||
writeFile(t, normalized, `{"segments":[{"id":10},{"id":11}]}`)
|
||||
scr.BoundsBody = `{"trim_action":"trim","start_segment_id":10,"end_segment_id":11}`
|
||||
@@ -94,7 +94,7 @@ func TestTrimStageUsesConfiguredScriptoriumInputName(t *testing.T) {
|
||||
|
||||
func TestTrimStageRecordsLogAndGeneratedConfigRefs(t *testing.T) {
|
||||
env, m, scr, _ := setupTrimEnv(t)
|
||||
paths := env.ArtifactStore.SessionPaths(m.SessionID)
|
||||
paths := sessionPathsForEnv(env, m.SessionID)
|
||||
writeFile(t, filepath.Join(paths.TranscriptsDir, "normalized.json"), `{"segments":[{"id":1},{"id":2}]}`)
|
||||
scr.BoundsBody = `{"trim_action":"trim","start_segment_id":1,"end_segment_id":2}`
|
||||
|
||||
@@ -114,7 +114,7 @@ func TestTrimStageRecordsLogAndGeneratedConfigRefs(t *testing.T) {
|
||||
|
||||
func TestTrimStageRenderDebugDiagnosticsAreNotStageOutputs(t *testing.T) {
|
||||
env, m, scr, _ := setupTrimEnv(t)
|
||||
paths := env.ArtifactStore.SessionPaths(m.SessionID)
|
||||
paths := sessionPathsForEnv(env, m.SessionID)
|
||||
writeFile(t, filepath.Join(paths.TranscriptsDir, "normalized.json"), `{"segments":[{"id":1},{"id":2}]}`)
|
||||
scr.BoundsBody = `{"trim_action":"trim","start_segment_id":1,"end_segment_id":2}`
|
||||
|
||||
@@ -152,7 +152,7 @@ func TestTrimStageFailsWhenNormalizedTranscriptMissing(t *testing.T) {
|
||||
|
||||
func TestTrimStageDoesNotFallBackToProcessedTranscript(t *testing.T) {
|
||||
env, m, scr, ser := setupTrimEnv(t)
|
||||
paths := env.ArtifactStore.SessionPaths(m.SessionID)
|
||||
paths := sessionPathsForEnv(env, m.SessionID)
|
||||
writeFile(t, filepath.Join(paths.TranscriptsDir, "processed.json"), `{"segments":[{"id":1},{"id":2}]}`)
|
||||
|
||||
_, err := (trimStage{}).Run(context.Background(), env, m)
|
||||
@@ -172,7 +172,7 @@ func TestTrimStageDoesNotFallBackToProcessedTranscript(t *testing.T) {
|
||||
|
||||
func TestTrimStageFailsWhenNormalizedTranscriptInvalidJSON(t *testing.T) {
|
||||
env, m, _, _ := setupTrimEnv(t)
|
||||
paths := env.ArtifactStore.SessionPaths(m.SessionID)
|
||||
paths := sessionPathsForEnv(env, m.SessionID)
|
||||
writeFile(t, filepath.Join(paths.TranscriptsDir, "normalized.json"), "not-json")
|
||||
_, err := (trimStage{}).Run(context.Background(), env, m)
|
||||
if err == nil {
|
||||
@@ -185,7 +185,7 @@ func TestTrimStageFailsWhenNormalizedTranscriptInvalidJSON(t *testing.T) {
|
||||
|
||||
func TestTrimStageFailsWhenNormalizedTranscriptMissingSegmentsArray(t *testing.T) {
|
||||
env, m, _, _ := setupTrimEnv(t)
|
||||
paths := env.ArtifactStore.SessionPaths(m.SessionID)
|
||||
paths := sessionPathsForEnv(env, m.SessionID)
|
||||
writeFile(t, filepath.Join(paths.TranscriptsDir, "normalized.json"), `{"schema":"audita.processed.v1"}`)
|
||||
_, err := (trimStage{}).Run(context.Background(), env, m)
|
||||
if err == nil {
|
||||
@@ -198,7 +198,7 @@ func TestTrimStageFailsWhenNormalizedTranscriptMissingSegmentsArray(t *testing.T
|
||||
|
||||
func TestTrimStageFailsWhenBoundsOutputInvalidJSON(t *testing.T) {
|
||||
env, m, scr, _ := setupTrimEnv(t)
|
||||
paths := env.ArtifactStore.SessionPaths(m.SessionID)
|
||||
paths := sessionPathsForEnv(env, m.SessionID)
|
||||
writeFile(t, filepath.Join(paths.TranscriptsDir, "normalized.json"), `{"segments":[{"id":1},{"id":2}]}`)
|
||||
scr.BoundsBody = "not-json"
|
||||
_, err := (trimStage{}).Run(context.Background(), env, m)
|
||||
@@ -212,7 +212,7 @@ func TestTrimStageFailsWhenBoundsOutputInvalidJSON(t *testing.T) {
|
||||
|
||||
func TestTrimStageFailsWhenBoundsRangeIsDescending(t *testing.T) {
|
||||
env, m, scr, _ := setupTrimEnv(t)
|
||||
paths := env.ArtifactStore.SessionPaths(m.SessionID)
|
||||
paths := sessionPathsForEnv(env, m.SessionID)
|
||||
writeFile(t, filepath.Join(paths.TranscriptsDir, "normalized.json"), `{"segments":[{"id":1},{"id":2}]}`)
|
||||
scr.BoundsBody = `{"trim_action":"trim","start_segment_id":2,"end_segment_id":1}`
|
||||
_, err := (trimStage{}).Run(context.Background(), env, m)
|
||||
@@ -226,7 +226,7 @@ func TestTrimStageFailsWhenBoundsRangeIsDescending(t *testing.T) {
|
||||
|
||||
func TestTrimStageFailsWhenBoundsIDsMissingFromTranscript(t *testing.T) {
|
||||
env, m, scr, _ := setupTrimEnv(t)
|
||||
paths := env.ArtifactStore.SessionPaths(m.SessionID)
|
||||
paths := sessionPathsForEnv(env, m.SessionID)
|
||||
writeFile(t, filepath.Join(paths.TranscriptsDir, "normalized.json"), `{"segments":[{"id":20},{"id":21}]}`)
|
||||
scr.BoundsBody = `{"trim_action":"trim","start_segment_id":10,"end_segment_id":21}`
|
||||
_, err := (trimStage{}).Run(context.Background(), env, m)
|
||||
@@ -240,7 +240,7 @@ func TestTrimStageFailsWhenBoundsIDsMissingFromTranscript(t *testing.T) {
|
||||
|
||||
func TestTrimStageFailsWhenScriptoriumAdapterFails(t *testing.T) {
|
||||
env, m, scr, _ := setupTrimEnv(t)
|
||||
paths := env.ArtifactStore.SessionPaths(m.SessionID)
|
||||
paths := sessionPathsForEnv(env, m.SessionID)
|
||||
writeFile(t, filepath.Join(paths.TranscriptsDir, "normalized.json"), `{"segments":[{"id":1},{"id":2}]}`)
|
||||
scr.RunErr = errors.New("bounds failed")
|
||||
_, err := (trimStage{}).Run(context.Background(), env, m)
|
||||
@@ -254,7 +254,7 @@ func TestTrimStageFailsWhenScriptoriumAdapterFails(t *testing.T) {
|
||||
|
||||
func TestTrimStageFailsWhenSeriatimTrimAdapterFails(t *testing.T) {
|
||||
env, m, scr, ser := setupTrimEnv(t)
|
||||
paths := env.ArtifactStore.SessionPaths(m.SessionID)
|
||||
paths := sessionPathsForEnv(env, m.SessionID)
|
||||
writeFile(t, filepath.Join(paths.TranscriptsDir, "normalized.json"), `{"segments":[{"id":1},{"id":2}]}`)
|
||||
scr.BoundsBody = `{"trim_action":"trim","start_segment_id":1,"end_segment_id":2}`
|
||||
ser.TrimErr = errors.New("trim failed")
|
||||
@@ -269,7 +269,7 @@ func TestTrimStageFailsWhenSeriatimTrimAdapterFails(t *testing.T) {
|
||||
|
||||
func TestTrimStageDisabledCopiesNormalizedTranscript(t *testing.T) {
|
||||
env, m, scr, ser := setupTrimEnv(t)
|
||||
paths := env.ArtifactStore.SessionPaths(m.SessionID)
|
||||
paths := sessionPathsForEnv(env, m.SessionID)
|
||||
normalized := filepath.Join(paths.TranscriptsDir, "normalized.json")
|
||||
normalizedBody := `{"segments":[{"id":1,"text":"alpha"},{"id":2,"text":"beta"}]}`
|
||||
writeFile(t, normalized, normalizedBody)
|
||||
@@ -298,6 +298,39 @@ func TestTrimStageDisabledCopiesNormalizedTranscript(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
func TestTrimStageUsesRunLocalPathsAndPromotesCanonical(t *testing.T) {
|
||||
env, m, scr, ser := setupTrimEnv(t)
|
||||
env.Config.Session.Campaign = "sample-campaign"
|
||||
m.Campaign = "sample-campaign"
|
||||
m.RunID = "20260518T010203Z-abcdef12"
|
||||
paths := sessionPathsForEnv(env, m.SessionID)
|
||||
normalized := filepath.Join(paths.TranscriptsDir, "normalized.json")
|
||||
writeFile(t, normalized, `{"segments":[{"id":10},{"id":11}]}`)
|
||||
scr.BoundsBody = `{"trim_action":"trim","start_segment_id":10,"end_segment_id":11,"warnings":[]}`
|
||||
|
||||
result, err := (trimStage{}).Run(context.Background(), env, m)
|
||||
if err != nil {
|
||||
t.Fatalf("trim.Run() error = %v", err)
|
||||
}
|
||||
if len(scr.RunRequests) != 1 || len(ser.TrimRequests) != 1 {
|
||||
t.Fatalf("scriptorium run=%d seriatim trim=%d, want 1/1", len(scr.RunRequests), len(ser.TrimRequests))
|
||||
}
|
||||
if !strings.Contains(scr.RunRequests[0].OutputPath, filepath.Join("runs", m.RunID, "trim", "outputs")) {
|
||||
t.Fatalf("bounds run output path = %q, want run-local path", scr.RunRequests[0].OutputPath)
|
||||
}
|
||||
if !strings.Contains(ser.TrimRequests[0].OutputTrimmedPath, filepath.Join("runs", m.RunID, "trim", "outputs")) {
|
||||
t.Fatalf("trim output path = %q, want run-local path", ser.TrimRequests[0].OutputTrimmedPath)
|
||||
}
|
||||
if len(result.Outputs) < 2 {
|
||||
t.Fatalf("outputs = %#v, want promoted trimmed+bounds outputs", result.Outputs)
|
||||
}
|
||||
for _, out := range result.Outputs {
|
||||
if strings.Contains(out.AbsolutePath, string(filepath.Separator)+"runs"+string(filepath.Separator)) {
|
||||
t.Fatalf("promoted output path = %q, want canonical session path", out.AbsolutePath)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
type boundsScriptoriumRunner struct {
|
||||
RunRequests []scriptorium.RunArtifactRequest
|
||||
RenderRequests []scriptorium.RenderArtifactRequest
|
||||
@@ -430,7 +463,7 @@ func setupTrimEnv(t *testing.T) (*Env, *manifest.Manifest, *boundsScriptoriumRun
|
||||
}
|
||||
|
||||
store := artifacts.NewLocalStore(workspace)
|
||||
if _, err := store.EnsureLayout("2026-05-03"); err != nil {
|
||||
if _, err := store.EnsureLayoutFor("sample-campaign", "2026-05-03"); err != nil {
|
||||
t.Fatalf("EnsureLayout() error = %v", err)
|
||||
}
|
||||
|
||||
|
||||
@@ -1,31 +0,0 @@
|
||||
# Narratio UX Evaluation Report
|
||||
|
||||
## 1. Executive Summary
|
||||
Narratio has a functional core pipeline with robust S3 integration for input and output, but it currently falls short of the intended "minimalist" operator UX. The primary gaps are the lack of session configuration discovery, the absence of session template support (and the `--session-id` flag), and the missing local cleanup logic. While the pipeline runs successfully, the operator must currently provide explicit session file paths for every run.
|
||||
|
||||
## 2. Feature Matrix
|
||||
|
||||
| Feature | Status | Evidence | Tests | Documentation | Notes |
|
||||
| :--- | :--- | :--- | :--- | :--- | :--- |
|
||||
| **Pipeline Config Discovery** | Implemented | `internal/app/pipeline_config_path_test.go` | Yes | Accurate | Checks `/usr/local/etc` and `/etc`. |
|
||||
| **Session Config Discovery** | Missing | `internal/app/run.go:30` | N/A | Stale | `--session` is mandatory. |
|
||||
| **Session Templates** | Missing | `internal/config/load.go` | N/A | Missing | No variable interpolation in `session.yml`. |
|
||||
| **`--session-id` CLI Flag** | Missing | `cmd/narratio` | N/A | Missing | Not implemented in CLI. |
|
||||
| **Minimal Seriatim Config** | Implemented | `internal/config/load.go` | Yes | Accurate | Defaults for timeout/schema provided. |
|
||||
| **Minimal Audita Config** | Implemented | `internal/config/load.go` | Yes | Accurate | Defaults for base_url/model provided. |
|
||||
| **S3 Audio Input** | Implemented | `internal/stage/prepare.go` | Yes | Accurate | Supports `.flac` downloads from S3. |
|
||||
| **S3 Archive & Promotion** | Implemented | `internal/stage/archive.go` | Yes | Accurate | Correct paths and commit markers. |
|
||||
| **Local Cleanup** | Missing | `architecture.md:136` | No | Stale | Config exists, logic is not implemented. |
|
||||
|
||||
## 3. Current Happy Path
|
||||
The shortest command that works today is:
|
||||
`narratio run --session <path_to_session.yml>`
|
||||
*(Assuming `pipeline.yml` is present in `/etc/narratio/` or `/usr/local/etc/narratio/`)*.
|
||||
|
||||
## 4. Gaps to Intended UX
|
||||
1. **Session Discovery & Templates (High):** The requirement to pass `--session` and the inability to use `--session-id` with a template is the largest friction point for operators.
|
||||
2. **Local Cleanup (Medium):** Spool and work directories are not cleaned up after successful archival, leading to local disk growth.
|
||||
3. **Local Pipeline Config (Low):** Narratio does not check `./pipeline.yml`, requiring users to use `--config` or move files to system directories.
|
||||
|
||||
## 5. Recommended Next Implementation Prompt
|
||||
"Implement session configuration discovery and template support. Specifically: 1) Add a search order for `session.yml` (e.g., `./session.yml`, `/etc/narratio/session.yml`) if `--session` is omitted. 2) Implement the `--session-id` CLI flag. 3) Add variable interpolation to `session.yml` so that `{{session_id}}` can be replaced by the value from the flag or the discovered session config before YAML decoding."
|
||||
54
report.md
54
report.md
@@ -1,54 +0,0 @@
|
||||
## 1. Executive Summary
|
||||
Narratio is close on S3 input/archive mechanics but not yet close on the intended minimal operator UX.
|
||||
Core S3 workflow is implemented (prepare S3 audio download, archive run upload, promotions, current pointers), but key UX items are missing: no `--session-id` flag, no session auto-discovery, and no session template variable injection. Cleanup/retention for spool/workdirs after archive is also still future work.
|
||||
|
||||
## 2. Feature Matrix
|
||||
|
||||
| Feature | Status | Evidence | Tests | Documentation status | Notes |
|
||||
|---|---|---|---|---|---|
|
||||
| Pipeline config auto-discovery when `--config` omitted | Implemented | `internal/app/pipeline_config_path.go`, `internal/config/defaults.go` | `internal/app/pipeline_config_path_test.go`, `internal/app/commands_test.go` | Accurate in `README.md`, `architecture.md` | Order: `/usr/local/etc/narratio/pipeline.yml`, then `/etc/narratio/pipeline.yml`; no `./pipeline.yml` default |
|
||||
| Session config auto-discovery when `--session` omitted | Missing | `--session` required in `internal/app/run.go`, `plan.go`, `resume.go`, `run_stage.go` | Covered by missing-flag tests in `internal/app/commands_test.go` | Accurate (docs do not claim auto-discovery) | No precedence order exists for session file search |
|
||||
| Session template variables in `session.yml` | Missing | Strict decode path in `internal/config/load.go` + strict YAML behavior | No template tests found | Not documented as implemented | No render-before-decode templating mechanism found |
|
||||
| `--session-id` CLI injection | Missing | No `--session-id` flag in command parsers (`run/plan/resume/run-stage`) | No tests for `--session-id` | Not documented as implemented | Intended minimal UX command not currently supported |
|
||||
| Campaign/run-aware work+spool paths | Implemented | `internal/artifacts/paths.go`, usage in prepare/archive | Path/helper tests in `internal/artifacts` + stage tests | Documented in README/architecture/roadmap | Layout includes `{campaign}/{session_id}/{run_id}` |
|
||||
| Run ID generation format | Implemented | `internal/artifacts/run_id.go` | Run ID tests in `internal/artifacts` | Documented | UTC timestamp + random suffix format present |
|
||||
| Storage backend abstraction | Implemented | `internal/adapters/storage/object_store.go` | Storage backend tests in `internal/adapters/storage` | Documented in README/architecture | Narrow interface (`List/Download/Upload/Exists`) |
|
||||
| S3 backend + fake backend | Implemented | `internal/adapters/storage/s3_backend.go`, `fake.go` | Adapter tests pass without live S3 | Documented | No AWS creds in config schema/examples |
|
||||
| Prepare S3 audio input (`inputs.audio_s3`) | Implemented | `internal/stage/prepare.go` | `internal/stage/prepare_test.go` | Documented in `docs/s3-audio-input.md`, README, architecture | Lists prefix, filters `.flac`, downloads/materializes, fails on none |
|
||||
| Local audio workflow | Implemented | Prepare logic still supports `audio_dir`/`audio_files` | Prepare tests cover local behavior and conflict with `audio_s3` | Documented | Local+S3 conflict is enforced |
|
||||
| Manifest provenance for S3 audio | Implemented | S3 source metadata assignment in prepare stage | Covered by S3 prepare tests | Documented | ETag recorded as metadata, not checksum |
|
||||
| Archive run upload under `runs/{run_id}` | Implemented | `internal/stage/archive.go` | `internal/stage/archive_test.go` | Documented in `docs/archive-storage.md`, README, architecture | Successful/completed runs only |
|
||||
| Archive promotion rules | Implemented | Archive stage promotion handling | Archive tests cover required/optional/mapping behavior | Documented | Default promoted outputs: `transcripts/trimmed.json`, `artifacts/session_recap.md` |
|
||||
| `current/manifest.json` + `current/run_id.txt` last | Implemented | Archive stage upload order logic | Archive tests verify ordering and pointer content | Documented | `current/run_id.txt` is commit marker; written last |
|
||||
| Avoid upload of failed/incomplete runs | Implemented | Archive prerequisite checks | Archive tests cover prerequisite failure path | Documented | Failed runs stay local |
|
||||
| Spool/workdir cleanup after successful archive | Missing | `spool.delete_audio_after_archive` exists but no cleanup behavior in stages/app | No cleanup behavior tests found | Docs accurately call cleanup future work | Gap vs intended UX item 12 |
|
||||
| Minimal Seriatim config | Partial | Validation requires `seriatim.binary`; defaults fill timeout/schema/gap | Config load/validate tests | Docs mostly accurate | “Binary-only” works after defaults, but still validated post-defaults |
|
||||
| Minimal Audita config | Partial | Validation requires `audita.binary` and `audita.model`; defaults for timeout/base_url/etc in loader | Config tests in `internal/config` | Docs currently list `timeout`/`base_url` as required in README section | UX expectation “binary + llm_api_key_env only” does not hold because model is required |
|
||||
|
||||
## 3. Current Happy Path
|
||||
Shortest realistic command today is:
|
||||
|
||||
`narratio run --session /path/to/session.yml`
|
||||
|
||||
That works only if pipeline config is discoverable at `/usr/local/etc/narratio/pipeline.yml` or `/etc/narratio/pipeline.yml`.
|
||||
Otherwise minimum is:
|
||||
|
||||
`narratio run --config /path/to/pipeline.yml --session /path/to/session.yml`
|
||||
|
||||
`narratio run --session-id 2026-04-04` does not work today (flag not implemented).
|
||||
|
||||
## 4. Gaps to Intended UX
|
||||
1. Missing `--session-id` flow with session template injection (largest UX gap).
|
||||
2. No session config auto-discovery order when `--session` is omitted.
|
||||
3. No session template rendering engine / unresolved-variable handling.
|
||||
4. Cleanup policy not implemented (`spool.delete_audio_after_archive` is modeled only).
|
||||
5. Audita minimal config UX still stricter than intended (model required).
|
||||
6. Optional doc refinement: explicitly call out that `./pipeline.yml` is not in current default search order.
|
||||
|
||||
## 5. Recommended Next Implementation Prompt
|
||||
Implement session template and `--session-id` UX only:
|
||||
|
||||
> Add session discovery and template rendering support so `narratio run --session-id <id>` works with no `--session` in normal setups.
|
||||
> Requirements: define deterministic session discovery order; support rendering template variables in `session.yml` before strict YAML decode; inject CLI `--session-id` into template variables; fail clearly on unresolved variables; preserve strict field validation after render; keep existing `--session` explicit path behavior; add tests for discovery precedence, render success/failure, and CLI integration; update README/architecture/examples accordingly; do not change archive/prepare storage behavior.
|
||||
|
||||
Validation note: `go test ./...` passes for the inspected state.
|
||||
Reference in New Issue
Block a user