Compare commits
14 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
| 37daab7857 | |||
| 2356688cb9 | |||
| 1054b64d9f | |||
| 01fb02426c | |||
| 7dc79e052f | |||
| cb525c0f72 | |||
| 622677d038 | |||
| 550288e008 | |||
| e58e545686 | |||
| 6ff54c5a0f | |||
| 924b5d15c6 | |||
| b065663180 | |||
| 3ba564b00f | |||
| a3986cf0d6 |
2
.gitignore
vendored
2
.gitignore
vendored
@@ -2,6 +2,8 @@
|
||||
.codex
|
||||
AGENTS.md
|
||||
|
||||
.DS_Store
|
||||
|
||||
# ---> Go
|
||||
# If you prefer the allow list template instead of the deny list, see community template:
|
||||
# https://github.com/github/gitignore/blob/main/community/Golang/Go.AllowList.gitignore
|
||||
|
||||
14
README.md
14
README.md
@@ -65,6 +65,8 @@ Optional secrets-from-files config:
|
||||
|
||||
YAML decoding is strict (`KnownFields(true)`), so unknown fields fail fast.
|
||||
|
||||
Maintainer note: application defaults are centralized in [`internal/config/defaults.go`](internal/config/defaults.go).
|
||||
|
||||
## Storage And Archive Foundations
|
||||
|
||||
Narratio now includes configuration and path-model foundations for archive support, plus implemented prepare-stage S3 audio input.
|
||||
@@ -367,10 +369,16 @@ For the initial implementation, only `session_recap` generation is supported.
|
||||
|
||||
Analyze-stage session recap behavior:
|
||||
|
||||
- available transcript input sources for configured artifacts: `processed_transcript`, `normalized_transcript`, `trimmed_transcript`
|
||||
- preferred transcript artifact source IDs:
|
||||
- `narratio.transcript.polished`
|
||||
- `narratio.transcript.full`
|
||||
- `narratio.transcript.trimmed`
|
||||
- backward-compatible aliases remain supported:
|
||||
- `processed_transcript`
|
||||
- `normalized_transcript`
|
||||
- `trimmed_transcript`
|
||||
- session recap should use gameplay-only transcript input (`source: trimmed_transcript`)
|
||||
- Narratio resolves `trimmed_transcript` from trim manifest output (`transcript_trimmed`) or fallback `transcripts/trimmed.json`
|
||||
- Narratio resolves `normalized_transcript` from normalize manifest output (`transcript_normalized`) or fallback `transcripts/normalized.json`
|
||||
- Narratio resolves transcript inputs from the artifact resolver (manifest producer outputs first, then canonical session paths)
|
||||
- missing trimmed transcript fails clearly and advises running trim stage first
|
||||
- `normalized_transcript` is the preferred full-transcript source for future table/meta-analysis artifacts
|
||||
- `processed_transcript` remains supported for advanced/debug use cases
|
||||
|
||||
@@ -114,6 +114,7 @@ CLI pipeline config path resolution:
|
||||
- when `--config` is omitted, Narratio searches defaults in order:
|
||||
- `/usr/local/etc/narratio/pipeline.yml`
|
||||
- `/etc/narratio/pipeline.yml`
|
||||
- default values are centralized in `internal/config/defaults.go`
|
||||
|
||||
CLI session config path resolution:
|
||||
|
||||
|
||||
130
docs/development/workspace-implementation-plan.md
Normal file
130
docs/development/workspace-implementation-plan.md
Normal file
@@ -0,0 +1,130 @@
|
||||
# Workspace Architecture Implementation Plan (Status)
|
||||
|
||||
This document tracks the implemented workspace architecture and remaining work for v1.0.
|
||||
|
||||
## Current Architecture (Implemented)
|
||||
|
||||
Narratio now uses a canonical campaign-aware local layout:
|
||||
|
||||
```text
|
||||
{workspace.root}/work/{campaign_id}/{session_id}/
|
||||
manifest.json
|
||||
current/
|
||||
manifest.json
|
||||
run_id.txt
|
||||
inputs/
|
||||
transcripts/
|
||||
artifacts/
|
||||
reports/
|
||||
logs/
|
||||
config/
|
||||
runs/
|
||||
{run_id}/
|
||||
manifest.json
|
||||
{stage}/
|
||||
outputs/
|
||||
logs/
|
||||
reports/
|
||||
config/
|
||||
scratch/
|
||||
```
|
||||
|
||||
Core behavior:
|
||||
|
||||
- Session manifest remains the skip/resume source of truth.
|
||||
- Each invocation creates a run manifest at `runs/{run_id}/manifest.json`.
|
||||
- Stage execution writes run-local artifacts and promotes durable outputs to canonical session paths.
|
||||
- Archive uploads run records under `runs/{run_id}/`, applies promotion rules, then publishes `current/manifest.json` and `current/run_id.txt`.
|
||||
- Analyze input resolution uses centralized artifact IDs with alias support.
|
||||
- Forced upstream reruns mark downstream succeeded stages `stale` so later runs do not skip stale outputs.
|
||||
|
||||
## Section 4 Sequence Status
|
||||
|
||||
### Step 1: Campaign-aware session path model
|
||||
|
||||
Status: complete.
|
||||
|
||||
Implemented:
|
||||
|
||||
- Campaign-aware session and run path helpers.
|
||||
- Campaign-aware artifact-store layout APIs.
|
||||
- Canonical session manifest pathing under `work/{campaign}/{session}`.
|
||||
|
||||
### Step 2: Session manifest + run manifest scaffolding
|
||||
|
||||
Status: complete.
|
||||
|
||||
Implemented:
|
||||
|
||||
- Invocation-scoped run manifest type and store methods.
|
||||
- Runner creates/saves run manifests per invocation.
|
||||
- Session manifest remains authoritative for idempotent stage skipping.
|
||||
|
||||
### Step 3: Run-local stage execution + promotion
|
||||
|
||||
Status: complete.
|
||||
|
||||
Implemented:
|
||||
|
||||
- Run-local stage directory layout under `runs/{run_id}/{stage}`.
|
||||
- Shared helpers for run-local output mapping and promotion to canonical durable paths.
|
||||
- Producer run provenance recorded on durable artifact outputs.
|
||||
|
||||
### Step 4: Archive alignment
|
||||
|
||||
Status: complete.
|
||||
|
||||
Implemented:
|
||||
|
||||
- Canonical run-root/session-root resolution.
|
||||
- Deterministic run-file collection and promotion source resolution.
|
||||
- Current-pointer publication ordering retained (`current/manifest.json` then `current/run_id.txt`).
|
||||
|
||||
### Step 5: Artifact registry/resolver (analyze first consumer)
|
||||
|
||||
Status: complete.
|
||||
|
||||
Implemented:
|
||||
|
||||
- Central artifact resolver with canonical IDs:
|
||||
- `narratio.transcript.merged`
|
||||
- `narratio.transcript.polished`
|
||||
- `narratio.transcript.full`
|
||||
- `narratio.transcript.trimmed`
|
||||
- `narratio.bounds.session`
|
||||
- `narratio.artifact.session_recap`
|
||||
- Backward-compatible aliases:
|
||||
- `processed_transcript`
|
||||
- `normalized_transcript`
|
||||
- `trimmed_transcript`
|
||||
- Analyze stage switched to resolver-based source resolution.
|
||||
|
||||
### Step 6: Minimal downstream invalidation for forced reruns
|
||||
|
||||
Status: complete.
|
||||
|
||||
Implemented:
|
||||
|
||||
- Deterministic downstream invalidation based on canonical stage order.
|
||||
- On forced successful rerun of stage `X`, downstream succeeded stages are marked `stale`.
|
||||
- Resume and non-forced runs naturally re-execute stale stages.
|
||||
|
||||
### Step 7: Legacy layout migration strategy
|
||||
|
||||
Status: intentionally skipped.
|
||||
|
||||
Decision:
|
||||
|
||||
- Automatic migration and legacy fallback compatibility are intentionally not implemented.
|
||||
- The codebase targets canonical-only local layout behavior.
|
||||
- Legacy local workspace state, if present, should be recreated or migrated manually outside Narratio.
|
||||
|
||||
## Remaining Work (v1.0)
|
||||
|
||||
No required workspace/run-history migration steps remain from Section 4.
|
||||
|
||||
Possible future enhancements (non-blocking):
|
||||
|
||||
- Full checksum/input-graph stale detection.
|
||||
- Optional retention-policy expansion for run-history cleanup.
|
||||
- Broader artifact-resolver adoption across additional stage consumers.
|
||||
744
docs/development/workspace.md
Normal file
744
docs/development/workspace.md
Normal file
@@ -0,0 +1,744 @@
|
||||
# Narratio Workspace, Run History, and Artifact Resolution Architecture
|
||||
|
||||
## 1. Purpose
|
||||
|
||||
This document defines the intended v1.0 architecture for Narratio's local workspace layout, run history model, durable session outputs, manifest responsibilities, and artifact resolution contract.
|
||||
|
||||
Narratio is an idempotent session orchestrator. The command:
|
||||
|
||||
```bash
|
||||
narratio run --session-id 2026-05-07
|
||||
```
|
||||
|
||||
means "bring the identified session to its desired completed state." It does **not** mean "always create an entirely new independent output tree and ignore prior session state."
|
||||
|
||||
This distinction drives the architecture:
|
||||
|
||||
* A **session** is the durable domain object and idempotency boundary.
|
||||
* A **run** is an execution attempt that may update the session's durable state.
|
||||
* Durable outputs live at the session level.
|
||||
* Run-specific outputs, logs, generated configs, scratch files, and diagnostics live under `runs/{run_id}/`.
|
||||
* Successful stage outputs are promoted from run-local locations into canonical session-level locations.
|
||||
* The session manifest records current durable state.
|
||||
* Run manifests record execution history and debugging/provenance details.
|
||||
|
||||
This model intentionally mirrors the S3 archive model: session-level current artifacts are distinct from run-record history.
|
||||
|
||||
## 2. Core Concepts
|
||||
|
||||
### 2.1 Session
|
||||
|
||||
A session is the stable unit of work identified by `campaign_id` and `session_id`.
|
||||
|
||||
Examples:
|
||||
|
||||
```text
|
||||
campaign_id = dilfs
|
||||
session_id = 2026-05-07
|
||||
```
|
||||
|
||||
The session directory represents the current durable local state for that session. Re-running Narratio for the same session should consult this state, skip already-completed stages by default, and produce no changes unless work is incomplete, stale, forced, or explicitly selected.
|
||||
|
||||
### 2.2 Run
|
||||
|
||||
A run is a particular execution attempt identified by a generated `run_id`, for example:
|
||||
|
||||
```text
|
||||
20260517T174748Z-abcd1234
|
||||
```
|
||||
|
||||
A run may execute all stages or only a sparse subset of stages. Sparse runs are expected and desirable when the user invokes `--force`, `run-stage`, or a stage-limited command.
|
||||
|
||||
Run directories are provenance/debug records. They should reflect what actually happened during that invocation, not a synthetic complete pipeline layout.
|
||||
|
||||
### 2.3 Durable Output
|
||||
|
||||
A durable output is a canonical session-level artifact intended for later stages, user consumption, archive promotion, or future idempotency decisions.
|
||||
|
||||
Examples:
|
||||
|
||||
```text
|
||||
transcripts/merged.json
|
||||
transcripts/processed.json
|
||||
transcripts/normalized.json
|
||||
transcripts/trimmed.json
|
||||
artifacts/session_recap.md
|
||||
```
|
||||
|
||||
Durable outputs live directly under the session directory, not under a particular run directory.
|
||||
|
||||
### 2.4 Run-Local Output
|
||||
|
||||
A run-local output is the file initially produced by a stage during a specific run. After validation, durable outputs are promoted from run-local paths to session-level canonical paths.
|
||||
|
||||
Run-local outputs, logs, generated configs, reports, and scratch files should remain under:
|
||||
|
||||
```text
|
||||
runs/{run_id}/{stage}/...
|
||||
```
|
||||
|
||||
## 3. Local Workspace Layout
|
||||
|
||||
The canonical local workspace layout is:
|
||||
|
||||
```text
|
||||
{workspace.root}/work/{campaign_id}/{session_id}/
|
||||
manifest.json
|
||||
current/
|
||||
manifest.json
|
||||
run_id.txt
|
||||
inputs/
|
||||
transcripts/
|
||||
artifacts/
|
||||
reports/
|
||||
logs/
|
||||
config/
|
||||
runs/
|
||||
{run_id}/
|
||||
manifest.json
|
||||
prepare/
|
||||
transcribe/
|
||||
merge/
|
||||
polish/
|
||||
normalize/
|
||||
trim/
|
||||
analyze/
|
||||
archive/
|
||||
notify/
|
||||
```
|
||||
|
||||
Not every directory must exist at all times. Directories should be created idempotently when needed.
|
||||
|
||||
### 3.1 Session Root
|
||||
|
||||
The session root is:
|
||||
|
||||
```text
|
||||
{workspace.root}/work/{campaign_id}/{session_id}/
|
||||
```
|
||||
|
||||
The session root is the stable local home for the session. It is the default base for resolving canonical artifact paths.
|
||||
|
||||
The only files that should live directly in the session root are core session-state files, primarily:
|
||||
|
||||
```text
|
||||
manifest.json
|
||||
```
|
||||
|
||||
Lock files may also be session-root scoped if the implementation uses file locks there, but transient locks should not be treated as durable artifacts.
|
||||
|
||||
### 3.2 Session-Level Canonical Directories
|
||||
|
||||
The following directories contain current durable session state:
|
||||
|
||||
```text
|
||||
inputs/
|
||||
transcripts/
|
||||
artifacts/
|
||||
reports/
|
||||
logs/
|
||||
config/
|
||||
current/
|
||||
```
|
||||
|
||||
Recommended meanings:
|
||||
|
||||
| Directory | Purpose |
|
||||
| -------------- | ----------------------------------------------------------------------------- |
|
||||
| `inputs/` | Materialized or copied input files used by the current durable session state. |
|
||||
| `transcripts/` | Canonical transcript tiers. |
|
||||
| `artifacts/` | User-facing and machine-readable generated artifacts. |
|
||||
| `reports/` | Canonical stage reports worth preserving at the session level. |
|
||||
| `logs/` | Optional session-level logs or promoted/latest logs. |
|
||||
| `config/` | Optional session-level generated config snapshots or promoted/latest configs. |
|
||||
| `current/` | Current published session pointers, mirroring the archive backend. |
|
||||
|
||||
Canonical durable outputs should use stable paths under these directories.
|
||||
|
||||
### 3.3 Run History Directory
|
||||
|
||||
Run history lives under:
|
||||
|
||||
```text
|
||||
{workspace.root}/work/{campaign_id}/{session_id}/runs/{run_id}/
|
||||
```
|
||||
|
||||
Each run directory records what happened during that invocation. A run may contain all stage directories or only a sparse subset.
|
||||
|
||||
Example full run:
|
||||
|
||||
```text
|
||||
runs/20260517T174748Z-abcd1234/
|
||||
manifest.json
|
||||
prepare/
|
||||
transcribe/
|
||||
merge/
|
||||
polish/
|
||||
normalize/
|
||||
trim/
|
||||
analyze/
|
||||
archive/
|
||||
notify/
|
||||
```
|
||||
|
||||
Example sparse forced analyze run:
|
||||
|
||||
```text
|
||||
runs/20260518T030000Z-efgh5678/
|
||||
manifest.json
|
||||
analyze/
|
||||
```
|
||||
|
||||
Example sparse polish-through-analyze rerun:
|
||||
|
||||
```text
|
||||
runs/20260518T041500Z-a1b2c3d4/
|
||||
manifest.json
|
||||
polish/
|
||||
normalize/
|
||||
trim/
|
||||
analyze/
|
||||
```
|
||||
|
||||
Run directories should not create stage folders for stages that were not selected, executed, skipped, or otherwise considered during that run unless there is a clear diagnostic reason to do so.
|
||||
|
||||
### 3.4 Stage Run-Local Directories
|
||||
|
||||
Each stage receives a run-local directory:
|
||||
|
||||
```text
|
||||
runs/{run_id}/{stage}/
|
||||
```
|
||||
|
||||
Within that stage directory, the stage may use subdirectories such as:
|
||||
|
||||
```text
|
||||
outputs/
|
||||
logs/
|
||||
reports/
|
||||
config/
|
||||
scratch/
|
||||
```
|
||||
|
||||
For example:
|
||||
|
||||
```text
|
||||
runs/{run_id}/polish/
|
||||
outputs/transcripts/processed.json
|
||||
reports/audita.polish.report.json
|
||||
logs/stdout.log
|
||||
logs/stderr.log
|
||||
config/audita.polish.generated.yml
|
||||
scratch/
|
||||
```
|
||||
|
||||
The exact internal layout of a stage directory may vary by stage, but it should be deterministic, documented, and generated through centralized path helpers rather than ad hoc path joins.
|
||||
|
||||
## 4. Promotion Model
|
||||
|
||||
Narratio uses stage-level promotion with immediate promotion after successful validation.
|
||||
|
||||
The stage lifecycle is:
|
||||
|
||||
1. Resolve required inputs from the current session state and/or run-local context.
|
||||
2. Create the run-local stage directory.
|
||||
3. Execute the stage, writing outputs under `runs/{run_id}/{stage}/...`.
|
||||
4. Validate run-local outputs.
|
||||
5. Promote durable outputs into session-level canonical paths.
|
||||
6. Update the session manifest.
|
||||
7. Update the run manifest.
|
||||
|
||||
Promotion means an atomic or effectively atomic copy/rename from a run-local path to a session-level canonical path.
|
||||
|
||||
Example:
|
||||
|
||||
```text
|
||||
runs/{run_id}/polish/outputs/transcripts/processed.json
|
||||
```
|
||||
|
||||
is promoted to:
|
||||
|
||||
```text
|
||||
transcripts/processed.json
|
||||
```
|
||||
|
||||
Promotion should be safe and deterministic:
|
||||
|
||||
* Validate before promotion.
|
||||
* Write promoted files atomically where possible.
|
||||
* Never leave partially written durable outputs.
|
||||
* Record the producing `run_id` in the session manifest.
|
||||
* Preserve run-local files for debugging unless retention policy deletes them.
|
||||
|
||||
## 5. Promotion Policy: Option A
|
||||
|
||||
Narratio uses immediate stage-level promotion.
|
||||
|
||||
If a selected stage succeeds, its durable outputs are promoted immediately, even if a later selected stage fails.
|
||||
|
||||
Example:
|
||||
|
||||
```bash
|
||||
narratio run --session-id 2026-05-07 --force --stages polish,normalize,trim,analyze
|
||||
```
|
||||
|
||||
If `polish` succeeds and `normalize` fails:
|
||||
|
||||
* `transcripts/processed.json` may be updated from the new run.
|
||||
* `normalize`, `trim`, and `analyze` should not be marked succeeded for the new input state.
|
||||
* Downstream outputs may now be stale relative to the newly promoted polished transcript.
|
||||
|
||||
This policy is simpler, transparent, and consistent with stage-level resumability. It does require explicit stale/invalidation handling.
|
||||
|
||||
## 6. Stale and Invalidation Semantics
|
||||
|
||||
Full checksum-based stale detection may be implemented later. Before that exists, Narratio should still use a simple deterministic invalidation rule for forced or explicit upstream reruns.
|
||||
|
||||
When a stage is successfully re-executed and promoted, downstream stages should be marked stale unless they are also re-executed successfully in the same command invocation.
|
||||
|
||||
Example stage order:
|
||||
|
||||
```text
|
||||
prepare -> transcribe -> merge -> polish -> normalize -> trim -> analyze -> archive -> notify
|
||||
```
|
||||
|
||||
If `polish` is forced and promoted, then the following downstream stages should be invalidated unless rerun successfully:
|
||||
|
||||
```text
|
||||
normalize
|
||||
trim
|
||||
analyze
|
||||
archive
|
||||
notify
|
||||
```
|
||||
|
||||
A stale stage is not equivalent to a failed stage. It means its current durable outputs may no longer correspond to current upstream inputs or configuration.
|
||||
|
||||
Minimum manifest state model:
|
||||
|
||||
```text
|
||||
pending
|
||||
running
|
||||
succeeded
|
||||
failed
|
||||
skipped
|
||||
stale
|
||||
```
|
||||
|
||||
If adding a new `stale` state is too invasive for v1.0, the implementation should at least record stale metadata or clear downstream success markers in a way that prevents accidental idempotent skips based on obsolete outputs.
|
||||
|
||||
## 7. Manifest Responsibilities
|
||||
|
||||
Narratio should distinguish between session manifests and run manifests.
|
||||
|
||||
The same underlying Go types may be reused where practical, but the concepts should remain separate.
|
||||
|
||||
### 7.1 Session Manifest
|
||||
|
||||
Path:
|
||||
|
||||
```text
|
||||
{workspace.root}/work/{campaign_id}/{session_id}/manifest.json
|
||||
```
|
||||
|
||||
The session manifest answers:
|
||||
|
||||
```text
|
||||
What is the current durable state of this session?
|
||||
```
|
||||
|
||||
It should record:
|
||||
|
||||
* campaign ID
|
||||
* session ID
|
||||
* current or latest run ID
|
||||
* current stage states
|
||||
* canonical durable output refs
|
||||
* artifact IDs and paths
|
||||
* producing run ID for each current stage output
|
||||
* relevant input/config checksums when available
|
||||
* stale/invalidated stage information
|
||||
* archive/current publication metadata
|
||||
|
||||
A session's durable state may be a composite of multiple runs.
|
||||
|
||||
For example:
|
||||
|
||||
```text
|
||||
transcripts/merged.json produced by run A
|
||||
transcripts/processed.json produced by run B
|
||||
transcripts/normalized.json produced by run B
|
||||
transcripts/trimmed.json produced by run B
|
||||
artifacts/session_recap.md produced by run C
|
||||
```
|
||||
|
||||
This is valid and expected.
|
||||
|
||||
### 7.2 Run Manifest
|
||||
|
||||
Path:
|
||||
|
||||
```text
|
||||
{workspace.root}/work/{campaign_id}/{session_id}/runs/{run_id}/manifest.json
|
||||
```
|
||||
|
||||
The run manifest answers:
|
||||
|
||||
```text
|
||||
What happened during this specific execution attempt?
|
||||
```
|
||||
|
||||
It should record:
|
||||
|
||||
* run ID
|
||||
* campaign ID
|
||||
* session ID
|
||||
* command mode and selected stages
|
||||
* force flags or stage selection flags
|
||||
* stages considered during this run
|
||||
* stages executed during this run
|
||||
* stages skipped during this run and reasons
|
||||
* run-local output paths
|
||||
* promoted output paths
|
||||
* logs
|
||||
* reports
|
||||
* generated configs
|
||||
* timings
|
||||
* errors
|
||||
* non-secret subprocess invocation metadata
|
||||
|
||||
Run manifests are primarily for debugging, auditability, and archive history.
|
||||
|
||||
## 8. Idempotency and Resume Behavior
|
||||
|
||||
The idempotency boundary is the session, not the run.
|
||||
|
||||
By default:
|
||||
|
||||
```bash
|
||||
narratio run --session-id 2026-05-07
|
||||
```
|
||||
|
||||
should consult the session manifest and skip stages that are already succeeded and not stale.
|
||||
|
||||
If all stages are already complete, the command should execute zero stages and report that the session is already complete.
|
||||
|
||||
Forced execution creates a new run record but updates session-level durable state only for stages that actually succeed and promote outputs.
|
||||
|
||||
Examples:
|
||||
|
||||
```bash
|
||||
narratio run --session-id 2026-05-07 --force
|
||||
```
|
||||
|
||||
Creates a new run and attempts to re-execute the selected/default stage set.
|
||||
|
||||
```bash
|
||||
narratio run-stage --session-id 2026-05-07 analyze --force
|
||||
```
|
||||
|
||||
Creates a sparse run that executes only `analyze`, then promotes updated analysis artifacts if successful.
|
||||
|
||||
```bash
|
||||
narratio resume --session-id 2026-05-07
|
||||
```
|
||||
|
||||
Uses the session manifest to determine what remains incomplete or stale. Resume does not need to resume the same `run_id` unless the implementation explicitly supports resuming an interrupted active run.
|
||||
|
||||
## 9. Artifact Resolution Contract
|
||||
|
||||
Narratio should provide a first-class artifact registry and resolver.
|
||||
|
||||
The resolver maps symbolic artifact source names to canonical session-level paths and manifest output kinds.
|
||||
|
||||
Stages and adapters should not hardcode path fragments when resolving cross-stage inputs. They should ask the artifact resolver for the current durable artifact by ID.
|
||||
|
||||
### 9.1 Canonical Artifact IDs
|
||||
|
||||
Preferred artifact IDs should be namespaced:
|
||||
|
||||
```text
|
||||
narratio.transcript.merged
|
||||
narratio.transcript.polished
|
||||
narratio.transcript.full
|
||||
narratio.transcript.trimmed
|
||||
narratio.bounds.session
|
||||
narratio.artifact.session_recap
|
||||
```
|
||||
|
||||
Recommended initial registry:
|
||||
|
||||
| Artifact ID | Canonical Path | Producer Stage | Output Kind | Meaning |
|
||||
| --------------------------------- | ------------------------------- | -------------- | ------------------------ | ------------------------------------- |
|
||||
| `narratio.transcript.merged` | `transcripts/merged.json` | `merge` | `transcript_merged` | Deterministic Seriatim merge. |
|
||||
| `narratio.transcript.polished` | `transcripts/processed.json` | `polish` | `transcript_processed` | Full Audita-polished transcript. |
|
||||
| `narratio.transcript.full` | `transcripts/normalized.json` | `normalize` | `transcript_normalized` | Preferred full normalized transcript. |
|
||||
| `narratio.transcript.trimmed` | `transcripts/trimmed.json` | `trim` | `transcript_trimmed` | Gameplay-only transcript. |
|
||||
| `narratio.bounds.session` | `artifacts/session_bounds.json` | `trim` | `session_bounds` | Trim bounds selected for the session. |
|
||||
| `narratio.artifact.session_recap` | `artifacts/session_recap.md` | `analyze` | `artifact_session_recap` | Generated session recap. |
|
||||
|
||||
### 9.2 Backward-Compatible Aliases
|
||||
|
||||
Existing source names should remain supported:
|
||||
|
||||
| Legacy Source | Preferred Artifact ID |
|
||||
| ----------------------- | ------------------------------ |
|
||||
| `processed_transcript` | `narratio.transcript.polished` |
|
||||
| `normalized_transcript` | `narratio.transcript.full` |
|
||||
| `trimmed_transcript` | `narratio.transcript.trimmed` |
|
||||
|
||||
These aliases may be supported silently for v1.0. Documentation should prefer namespaced IDs.
|
||||
|
||||
### 9.3 Resolver Behavior
|
||||
|
||||
Artifact resolution should follow this order:
|
||||
|
||||
1. Normalize aliases to canonical artifact IDs.
|
||||
2. Look for a current output reference in the session manifest.
|
||||
3. Fall back to the canonical session-level path.
|
||||
4. If the artifact is required, fail clearly if missing.
|
||||
5. If the artifact is optional and missing, omit it from the downstream invocation.
|
||||
6. Validate the artifact using the expected content validator.
|
||||
7. Return a resolved artifact record containing ID, path, producer stage, output kind, and provenance.
|
||||
|
||||
Example conceptual result:
|
||||
|
||||
```json
|
||||
{
|
||||
"id": "narratio.transcript.trimmed",
|
||||
"path": "/var/lib/narratio/work/dilfs/2026-05-07/transcripts/trimmed.json",
|
||||
"producer_stage": "trim",
|
||||
"producer_run_id": "20260517T174748Z-abcd1234",
|
||||
"output_kind": "transcript_trimmed",
|
||||
"content_type": "application/json"
|
||||
}
|
||||
```
|
||||
|
||||
### 9.4 Artifact Validation
|
||||
|
||||
Transcript artifacts must be valid JSON with a top-level `segments` array.
|
||||
|
||||
Markdown/text artifacts must exist and be non-empty when required.
|
||||
|
||||
Bounds artifacts must match the expected bounds schema and refer to segment IDs in the same transcript ID space used by the trim stage.
|
||||
|
||||
Validation should happen before a resolved artifact is passed to another stage or external subprocess.
|
||||
|
||||
## 10. Analyze Stage Implications
|
||||
|
||||
The analyze stage should consume artifacts through the artifact resolver.
|
||||
|
||||
Preferred Scriptorium config shape:
|
||||
|
||||
```yaml
|
||||
scriptorium:
|
||||
artifacts:
|
||||
session_recap:
|
||||
enabled: true
|
||||
prompt_id: "dnd.session_recap"
|
||||
output_path: "artifacts/session_recap.md"
|
||||
inputs:
|
||||
transcript:
|
||||
source: "narratio.transcript.trimmed"
|
||||
required: true
|
||||
```
|
||||
|
||||
Additional artifacts can choose different transcript tiers:
|
||||
|
||||
```yaml
|
||||
scriptorium:
|
||||
artifacts:
|
||||
table_summary:
|
||||
enabled: true
|
||||
prompt_id: "dnd.table_summary"
|
||||
output_path: "artifacts/table_summary.md"
|
||||
inputs:
|
||||
transcript:
|
||||
source: "narratio.transcript.full"
|
||||
required: true
|
||||
```
|
||||
|
||||
For v1.0, Narratio does not need a generic DAG engine. It may execute configured analyze artifacts in deterministic order and allow later artifacts to consume earlier artifacts only when that relationship is explicit and unambiguous.
|
||||
|
||||
Rules:
|
||||
|
||||
* Artifact inputs resolve from current session-level durable state.
|
||||
* Outputs are first written run-locally.
|
||||
* Successful analyze outputs are promoted to session-level `artifacts/` paths.
|
||||
* Manifest output refs record the producing run ID.
|
||||
* Optional inputs are omitted when unavailable.
|
||||
* Required missing inputs fail before invoking Scriptorium.
|
||||
|
||||
## 11. Archive Alignment
|
||||
|
||||
Local workspace semantics should mirror archive semantics.
|
||||
|
||||
Local session-level durable paths:
|
||||
|
||||
```text
|
||||
work/{campaign}/{session}/transcripts/trimmed.json
|
||||
work/{campaign}/{session}/artifacts/session_recap.md
|
||||
work/{campaign}/{session}/current/manifest.json
|
||||
work/{campaign}/{session}/current/run_id.txt
|
||||
work/{campaign}/{session}/runs/{run_id}/...
|
||||
```
|
||||
|
||||
should map naturally to remote archive paths:
|
||||
|
||||
```text
|
||||
{root_prefix}/campaigns/{campaign}/sessions/{session}/transcripts/trimmed.json
|
||||
{root_prefix}/campaigns/{campaign}/sessions/{session}/artifacts/session_recap.md
|
||||
{root_prefix}/campaigns/{campaign}/sessions/{session}/current/manifest.json
|
||||
{root_prefix}/campaigns/{campaign}/sessions/{session}/current/run_id.txt
|
||||
{root_prefix}/campaigns/{campaign}/sessions/{session}/runs/{run_id}/...
|
||||
```
|
||||
|
||||
The archive stage should publish run records and promoted current artifacts consistently with the local model.
|
||||
|
||||
`current/run_id.txt` remains the effective commit marker for the archived current session state.
|
||||
|
||||
## 12. Path Helper Requirements
|
||||
|
||||
All code should use centralized path helpers for workspace paths.
|
||||
|
||||
Stage code should not manually assemble durable cross-stage paths using raw string joins except through the path model.
|
||||
|
||||
Recommended helper surface:
|
||||
|
||||
```text
|
||||
SessionRoot(campaignID, sessionID)
|
||||
SessionManifestPath(campaignID, sessionID)
|
||||
SessionCurrentDir(campaignID, sessionID)
|
||||
SessionTranscriptsDir(campaignID, sessionID)
|
||||
SessionArtifactsDir(campaignID, sessionID)
|
||||
SessionReportsDir(campaignID, sessionID)
|
||||
SessionLogsDir(campaignID, sessionID)
|
||||
SessionConfigDir(campaignID, sessionID)
|
||||
RunsDir(campaignID, sessionID)
|
||||
RunRoot(campaignID, sessionID, runID)
|
||||
RunManifestPath(campaignID, sessionID, runID)
|
||||
RunStageDir(campaignID, sessionID, runID, stage)
|
||||
RunStageOutputsDir(campaignID, sessionID, runID, stage)
|
||||
RunStageLogsDir(campaignID, sessionID, runID, stage)
|
||||
RunStageReportsDir(campaignID, sessionID, runID, stage)
|
||||
RunStageConfigDir(campaignID, sessionID, runID, stage)
|
||||
CanonicalArtifactPath(campaignID, sessionID, artifactID)
|
||||
```
|
||||
|
||||
Path helpers should enforce safe relative paths for configured output paths:
|
||||
|
||||
* reject absolute paths unless explicitly allowed for a particular config field
|
||||
* reject `..` traversal
|
||||
* normalize separators
|
||||
* preserve deterministic output paths
|
||||
|
||||
## 13. Directory Creation Policy
|
||||
|
||||
Directory creation should be centralized and idempotent.
|
||||
|
||||
Recommended policy:
|
||||
|
||||
* `prepare` ensures the baseline session directory structure exists.
|
||||
* Every stage also calls shared layout helpers to ensure its required run-local directories exist before writing.
|
||||
* `run-stage` should not depend on a prior `prepare` invocation merely to create folders.
|
||||
* Missing directories should be created with appropriate permissions.
|
||||
* Directory creation should not imply stage success.
|
||||
|
||||
This provides consistent layout while keeping direct stage execution robust.
|
||||
|
||||
## 14. Cleanup and Retention
|
||||
|
||||
Cleanup must preserve the distinction between durable session state and run history.
|
||||
|
||||
Workspace cleanup after successful archive may remove selected local directories only according to explicit configuration.
|
||||
|
||||
Potential retention policies:
|
||||
|
||||
```text
|
||||
keep_all_runs
|
||||
keep_failed_runs
|
||||
keep_last_n_runs
|
||||
delete_run_after_success
|
||||
```
|
||||
|
||||
For v1.0, conservative retention is preferred:
|
||||
|
||||
* Do not delete durable session-level outputs unless explicitly requested.
|
||||
* Do not delete failed run directories by default.
|
||||
* If cleanup is enabled, remove only documented run-scoped or spool-scoped paths.
|
||||
* Local development audio inputs must never be deleted by workspace cleanup.
|
||||
|
||||
## 15. Canonical-Only Layout Policy
|
||||
|
||||
Narratio now supports only the canonical campaign-aware layout:
|
||||
|
||||
```text
|
||||
{workspace.root}/work/{campaign_id}/{session_id}/manifest.json
|
||||
{workspace.root}/work/{campaign_id}/{session_id}/runs/{run_id}/...
|
||||
```
|
||||
|
||||
Legacy session-only layout compatibility is intentionally not implemented.
|
||||
|
||||
If legacy workspace data exists, operators should recreate or manually migrate that data outside Narratio before running v1.0 commands.
|
||||
|
||||
## 16. Documentation Updates Required
|
||||
|
||||
The following documentation should be updated to reflect this architecture:
|
||||
|
||||
* `README.md`
|
||||
* `docs/architecture.md`
|
||||
* a dedicated workspace/run-history document, such as this file
|
||||
* S3/archive documentation
|
||||
* analyze/artifact configuration documentation
|
||||
* example pipeline files
|
||||
|
||||
Documentation should consistently use the following terms:
|
||||
|
||||
| Term | Meaning |
|
||||
| ---------------- | ---------------------------------------------------------------------- |
|
||||
| Session | Durable domain object and idempotency boundary. |
|
||||
| Run | Execution attempt that may update session state. |
|
||||
| Durable output | Canonical current session-level output. |
|
||||
| Run-local output | Output produced inside a specific run directory before promotion. |
|
||||
| Promotion | Validated copy/rename from run-local output to durable session output. |
|
||||
| Session manifest | Current durable state of the session. |
|
||||
| Run manifest | Execution record for a particular run. |
|
||||
| Artifact ID | Symbolic source name resolved by the artifact registry. |
|
||||
|
||||
## 17. Architectural Invariants
|
||||
|
||||
The following invariants should hold after implementation:
|
||||
|
||||
1. `session_id` remains the idempotency boundary for normal operator commands.
|
||||
2. `run_id` identifies an execution attempt, not the primary durable workspace.
|
||||
3. Session-level canonical artifacts are the default inputs for downstream stages.
|
||||
4. Run-local outputs are promoted only after validation.
|
||||
5. A session's current durable state may be composed of outputs from multiple runs.
|
||||
6. Sparse run directories are valid and expected.
|
||||
7. The session manifest records current stage/artifact state and producer run IDs.
|
||||
8. The run manifest records what happened during one invocation.
|
||||
9. Artifact consumers resolve symbolic artifact IDs through a registry/resolver.
|
||||
10. Local workspace semantics mirror S3 archive semantics.
|
||||
11. Directory creation is centralized and idempotent.
|
||||
12. Stage code uses path helpers rather than ad hoc path construction.
|
||||
13. Forced upstream reruns invalidate downstream stage success unless downstream stages are rerun successfully.
|
||||
14. Cleanup never removes durable session outputs or local development inputs unless explicitly configured to do so.
|
||||
|
||||
## 18. Implementation Guidance
|
||||
|
||||
A practical implementation sequence is:
|
||||
|
||||
1. Add this architecture document.
|
||||
2. Add or revise path model helpers for session roots, run roots, stage directories, and canonical artifact paths.
|
||||
3. Introduce session manifest versus run manifest concepts.
|
||||
4. Route stage outputs through run-local directories.
|
||||
5. Add promotion helpers with validation and atomic writes.
|
||||
6. Update existing stages to promote durable outputs to session-level canonical paths.
|
||||
7. Add artifact registry and resolver.
|
||||
8. Update analyze to use artifact IDs and aliases.
|
||||
9. Add simple downstream stale invalidation for forced upstream reruns.
|
||||
10. Align archive/local path behavior and documentation.
|
||||
11. Update examples and README.
|
||||
12. Add tests for idempotency, sparse forced runs, promotion, manifest provenance, and artifact resolution.
|
||||
|
||||
This sequence intentionally avoids introducing a generic DAG engine. The v1.0 goal is a clear, deterministic, stage-oriented orchestrator with stable session-level outputs and inspectable run history.
|
||||
@@ -283,9 +283,9 @@ Session recap:
|
||||
```bash
|
||||
scriptorium run \
|
||||
--prompt dnd.session_recap \
|
||||
--input transcript=/work/session-42/transcript.polished.md \
|
||||
--input glossary=/work/session-42/glossary.yml \
|
||||
--out /work/session-42/artifacts/session_recap.md
|
||||
--input transcript=/work/campaign-7/session-42/transcript.polished.md \
|
||||
--input glossary=/work/campaign-7/session-42/glossary.yml \
|
||||
--out /work/campaign-7/session-42/artifacts/session_recap.md
|
||||
```
|
||||
|
||||
Structured events:
|
||||
@@ -293,8 +293,8 @@ Structured events:
|
||||
```bash
|
||||
scriptorium run \
|
||||
--prompt dnd.structured_events \
|
||||
--input transcript=/work/session-42/transcript.polished.md \
|
||||
--out /work/session-42/artifacts/structured_events.json
|
||||
--input transcript=/work/campaign-7/session-42/transcript.polished.md \
|
||||
--out /work/campaign-7/session-42/artifacts/structured_events.json
|
||||
```
|
||||
|
||||
Glossary suggestions:
|
||||
@@ -302,9 +302,9 @@ Glossary suggestions:
|
||||
```bash
|
||||
scriptorium run \
|
||||
--prompt dnd.glossary_suggestions \
|
||||
--input transcript=/work/session-42/transcript.polished.md \
|
||||
--input previous_recap=/work/session-41/artifacts/session_recap.md \
|
||||
--out /work/session-42/artifacts/glossary_suggestions.md
|
||||
--input transcript=/work/campaign-7/session-42/transcript.polished.md \
|
||||
--input previous_recap=/work/campaign-7/session-41/artifacts/session_recap.md \
|
||||
--out /work/campaign-7/session-42/artifacts/glossary_suggestions.md
|
||||
```
|
||||
|
||||
Player-facing summary:
|
||||
@@ -312,9 +312,9 @@ Player-facing summary:
|
||||
```bash
|
||||
scriptorium run \
|
||||
--prompt dnd.player_summary \
|
||||
--input transcript=/work/session-42/transcript.polished.md \
|
||||
--input structured_events=/work/session-42/artifacts/structured_events.json \
|
||||
--out /work/session-42/artifacts/player_summary.md
|
||||
--input transcript=/work/campaign-7/session-42/transcript.polished.md \
|
||||
--input structured_events=/work/campaign-7/session-42/artifacts/structured_events.json \
|
||||
--out /work/campaign-7/session-42/artifacts/player_summary.md
|
||||
```
|
||||
|
||||
## 21. Non-Goals
|
||||
|
||||
File diff suppressed because it is too large
Load Diff
@@ -1,35 +0,0 @@
|
||||
# S3 Archive Foundations Runbook
|
||||
|
||||
This runbook documents the currently implemented storage/archive foundations and the boundaries of current behavior.
|
||||
|
||||
## Implemented Now
|
||||
|
||||
- config modeling for:
|
||||
- `pipeline.storage.s3`
|
||||
- `pipeline.spool`
|
||||
- `pipeline.archive`
|
||||
- `session.inputs.audio_s3`
|
||||
- promotion rule validation for safe relative paths
|
||||
- run ID generation and path/key helper functions
|
||||
- manifest run/path identity fields
|
||||
- remote storage backend layer:
|
||||
- object-store interface (`List`, `Download`, `Upload`, `Exists`)
|
||||
- fake backend for deterministic tests
|
||||
- S3-compatible backend using AWS SDK v2
|
||||
- config-based backend construction helper
|
||||
|
||||
## Not Implemented Yet
|
||||
|
||||
- prepare-stage S3 object listing or download
|
||||
- archive-stage S3 upload or promotion writes
|
||||
- writing `current/manifest.json` or `current/run_id.txt` in S3
|
||||
|
||||
## Operational Notes
|
||||
|
||||
- local audio workflows remain the active development path (`audio_dir` or `audio_files`)
|
||||
- `audio_s3` and local audio config are mutually exclusive
|
||||
- do not place AWS credentials in Narratio config files
|
||||
|
||||
## Next Implementation Target
|
||||
|
||||
Use the storage backend layer in prepare-stage session audio discovery/download flow, while preserving local audio input support.
|
||||
@@ -56,7 +56,7 @@ When `inputs.audio_s3.prefix` is configured, `prepare`:
|
||||
4. downloads selected objects to spool audio:
|
||||
- `{spool.root}/{campaign}/{session_id}/{run_id}/audio/`
|
||||
5. materializes audio into workdir audio:
|
||||
- `{workspace.root}/work/{campaign}/{session_id}/{run_id}/audio/`
|
||||
- `{workspace.root}/work/{campaign}/{session_id}/runs/{run_id}/audio/`
|
||||
6. records input provenance in the manifest (bucket, key, metadata, local paths, checksum)
|
||||
|
||||
Notes:
|
||||
@@ -1,62 +0,0 @@
|
||||
# Storage Backends
|
||||
|
||||
This document describes the currently implemented remote object storage backend layer used by Narratio, and its intended role in later prepare/archive work.
|
||||
|
||||
## Implemented
|
||||
|
||||
Remote object store abstraction:
|
||||
|
||||
- `List(ctx, prefix)`
|
||||
- `Download(ctx, key, localPath)`
|
||||
- `Upload(ctx, localPath, key, opts)`
|
||||
- `Exists(ctx, key)`
|
||||
|
||||
Object metadata model includes:
|
||||
|
||||
- key
|
||||
- size
|
||||
- ETag (provider metadata only)
|
||||
- last modified time when available
|
||||
|
||||
Backends:
|
||||
|
||||
- fake storage backend for deterministic tests
|
||||
- S3-compatible backend implemented with AWS SDK for Go v2
|
||||
|
||||
Construction:
|
||||
|
||||
- config-based constructor builds S3 backend from `pipeline.storage.s3` values:
|
||||
- bucket
|
||||
- region
|
||||
- endpoint
|
||||
- force_path_style
|
||||
- access_key_id_env
|
||||
- secret_access_key_env
|
||||
|
||||
## Key Invariant
|
||||
|
||||
- callers pass full bucket-relative object keys
|
||||
- storage backends do not prepend `root_prefix`
|
||||
- storage backends do not infer campaign/session/run paths
|
||||
|
||||
S3 session/run key builders remain separate and continue to live outside backend implementations.
|
||||
|
||||
## Security Boundary
|
||||
|
||||
- do not store AWS credentials in Narratio config
|
||||
- Narratio first checks configured env-var names (`access_key_id_env`, `secret_access_key_env`);
|
||||
when both are present and non-empty, it uses static credentials from those values
|
||||
- when either configured credential value is missing, Narratio falls back to the standard AWS SDK credential chain
|
||||
- AWS SDK-specific types remain isolated to the storage adapter package
|
||||
|
||||
## Testing
|
||||
|
||||
- fake storage tests cover list/download/upload/exists and error paths
|
||||
- S3 backend tests use injected fake S3 API clients
|
||||
- tests do not require live S3 services, AWS credentials, or network access
|
||||
|
||||
## Not Implemented Yet
|
||||
|
||||
- prepare-stage S3 object listing or downloads
|
||||
- archive-stage S3 uploads or promotion writes
|
||||
- writing `current/manifest.json` or `current/run_id.txt` to S3
|
||||
@@ -111,7 +111,7 @@ func TestExecuteRunStageUnknownFails(t *testing.T) {
|
||||
func TestExecuteRunStageNormalizeIsAccepted(t *testing.T) {
|
||||
workspaceRoot := t.TempDir()
|
||||
pipelinePath, sessionPath := writeValidConfigFiles(t, workspaceRoot, "https://example.com/transcribe")
|
||||
workRoot := filepath.Join(workspaceRoot, "work", "2026-05-03")
|
||||
workRoot := filepath.Join(workspaceRoot, "work", "sample-campaign", "2026-05-03")
|
||||
mustWriteTestFile(t, filepath.Join(workRoot, "transcripts", "processed.json"), `{"segments":[{"id":1}]}`)
|
||||
|
||||
var stdout bytes.Buffer
|
||||
@@ -156,7 +156,7 @@ func TestExecuteRunStageTranscribeUsesConfiguredWhisperXServer(t *testing.T) {
|
||||
t.Fatal("expected whisperx server to be called at least once")
|
||||
}
|
||||
|
||||
outPath := filepath.Join(workspaceRoot, "work", "2026-05-03", "transcripts", "raw", "alice.json")
|
||||
outPath := filepath.Join(workspaceRoot, "work", "sample-campaign", "2026-05-03", "transcripts", "raw", "alice.json")
|
||||
data, err := os.ReadFile(outPath)
|
||||
if err != nil {
|
||||
t.Fatalf("ReadFile(%q): %v", outPath, err)
|
||||
@@ -233,7 +233,7 @@ inputs:
|
||||
_ = os.Chdir(originalWD)
|
||||
})
|
||||
|
||||
workRoot := filepath.Join(workspaceRoot, "work", sessionID)
|
||||
workRoot := filepath.Join(workspaceRoot, "work", "sample-campaign", sessionID)
|
||||
mustWriteTestFile(t, filepath.Join(workRoot, "transcripts", "merged.json"), `{"schema":"seriatim-intermediate","segments":[]}`)
|
||||
mustWriteTestFile(t, filepath.Join(workRoot, "inputs", "glossary.yml"), "[]\n")
|
||||
|
||||
|
||||
@@ -57,7 +57,7 @@ func Plan(ctx context.Context, args []string, out io.Writer) error {
|
||||
}
|
||||
|
||||
store := artifacts.NewLocalStore(cfg.Pipeline.Workspace.Root)
|
||||
paths, err := store.EnsureLayout(cfg.Session.SessionID)
|
||||
paths, err := store.EnsureLayoutFor(cfg.Session.Campaign, cfg.Session.SessionID)
|
||||
if err != nil {
|
||||
return fmt.Errorf("plan: prepare workdir: %w", err)
|
||||
}
|
||||
|
||||
@@ -36,7 +36,7 @@ func TestPlanCreatesAndReusesWorkdir(t *testing.T) {
|
||||
t.Fatalf("first output = %q, want totals", got)
|
||||
}
|
||||
|
||||
sessionWorkdir := artifacts.SessionWorkDir(workspaceRoot, "2026-05-03")
|
||||
sessionWorkdir := artifacts.SessionWorkDirForCampaign(workspaceRoot, "sample-campaign", "2026-05-03")
|
||||
expectedDirs := []string{
|
||||
sessionWorkdir,
|
||||
filepath.Join(sessionWorkdir, "inputs"),
|
||||
@@ -63,7 +63,7 @@ func TestPlanCreatesAndReusesWorkdir(t *testing.T) {
|
||||
func TestPlanShowsRunAndSkipFromManifest(t *testing.T) {
|
||||
workspaceRoot := t.TempDir()
|
||||
pipelinePath, sessionPath := writeValidConfigFiles(t, workspaceRoot)
|
||||
manifestPath := filepath.Join(workspaceRoot, "work", "2026-05-03", "manifest.json")
|
||||
manifestPath := filepath.Join(workspaceRoot, "work", "sample-campaign", "2026-05-03", "manifest.json")
|
||||
|
||||
store := &manifest.LocalStore{}
|
||||
m := manifest.New("2026-05-03", time.Date(2026, 5, 3, 10, 0, 0, 0, time.UTC))
|
||||
|
||||
@@ -54,7 +54,7 @@ func runPostArchiveCleanup(ctx context.Context, env *Env, manifestPath string, m
|
||||
}
|
||||
workDir := strings.TrimSpace(m.LocalWorkDir)
|
||||
if workDir == "" {
|
||||
workDir = artifacts.SessionRunWorkDir(
|
||||
workDir = artifacts.SessionRunRootForCampaign(
|
||||
env.Config.Pipeline.Workspace.Root,
|
||||
strings.TrimSpace(env.Config.Session.Campaign),
|
||||
strings.TrimSpace(env.Config.Session.SessionID),
|
||||
|
||||
@@ -210,7 +210,7 @@ func TestPostArchiveCleanupNotRunWhenPromotionIsMissing(t *testing.T) {
|
||||
assertExists(t, seed.spoolAudioDir)
|
||||
assertExists(t, seed.runWorkDir)
|
||||
assertExists(t, filepath.Join(seed.runWorkDir, "manifest.json"))
|
||||
assertExists(t, artifacts.SessionRunWorkDir(cfg.Pipeline.Workspace.Root, cfg.Session.Campaign, cfg.Session.SessionID, runID))
|
||||
assertExists(t, artifacts.SessionRunRootForCampaign(cfg.Pipeline.Workspace.Root, cfg.Session.Campaign, cfg.Session.SessionID, runID))
|
||||
}
|
||||
|
||||
func TestPostArchiveCleanupNotRunWhenCurrentManifestUploadFails(t *testing.T) {
|
||||
@@ -271,8 +271,8 @@ func cleanupFixtureConfig(t *testing.T) (*config.Config, cleanupSeed) {
|
||||
cfg.Pipeline.Spool.Root = filepath.Join(t.TempDir(), "spool")
|
||||
|
||||
runID := "20260516T010203Z-1a2b3c4d"
|
||||
runWorkDir := artifacts.SessionRunWorkDir(cfg.Pipeline.Workspace.Root, cfg.Session.Campaign, cfg.Session.SessionID, runID)
|
||||
otherRunDir := artifacts.SessionRunWorkDir(cfg.Pipeline.Workspace.Root, cfg.Session.Campaign, cfg.Session.SessionID, "20260516T010204Z-5e6f7a8b")
|
||||
runWorkDir := artifacts.SessionRunRootForCampaign(cfg.Pipeline.Workspace.Root, cfg.Session.Campaign, cfg.Session.SessionID, runID)
|
||||
otherRunDir := artifacts.SessionRunRootForCampaign(cfg.Pipeline.Workspace.Root, cfg.Session.Campaign, cfg.Session.SessionID, "20260516T010204Z-5e6f7a8b")
|
||||
spoolAudioDir := artifacts.SessionSpoolAudioDir(cfg.Pipeline.Spool.Root, cfg.Session.Campaign, cfg.Session.SessionID, runID)
|
||||
|
||||
mustWriteFile(t, filepath.Join(runWorkDir, "manifest.json"), "{}\n")
|
||||
@@ -326,7 +326,11 @@ func archiveStageCleanupFixture(t *testing.T) (*config.Config, cleanupSeed, stri
|
||||
{From: "artifacts/session_recap.md", To: "artifacts/session_recap.md", Required: boolPtr(true)},
|
||||
},
|
||||
}
|
||||
writeArchiveFixtureRunFiles(t, seed.runWorkDir)
|
||||
writeArchiveFixtureRunFiles(
|
||||
t,
|
||||
seed.runWorkDir,
|
||||
artifacts.SessionWorkDirForCampaign(cfg.Pipeline.Workspace.Root, cfg.Session.Campaign, cfg.Session.SessionID),
|
||||
)
|
||||
|
||||
store := &manifest.LocalStore{}
|
||||
seedManifest, err := store.Load(context.Background(), manifestPathFor(cfg))
|
||||
@@ -345,16 +349,19 @@ func archiveStageCleanupFixture(t *testing.T) (*config.Config, cleanupSeed, stri
|
||||
return cfg, seed, runID
|
||||
}
|
||||
|
||||
func writeArchiveFixtureRunFiles(t *testing.T, runWorkDir string) {
|
||||
func writeArchiveFixtureRunFiles(t *testing.T, runWorkDir, sessionRoot string) {
|
||||
t.Helper()
|
||||
mustWriteFile(t, filepath.Join(runWorkDir, "inputs", "session.yml"), "session_id: 2026-05-03\n")
|
||||
mustWriteFile(t, filepath.Join(runWorkDir, "transcripts", "raw", "speaker.json"), "{}\n")
|
||||
mustWriteFile(t, filepath.Join(runWorkDir, "transcripts", "trimmed.json"), "{}\n")
|
||||
mustWriteFile(t, filepath.Join(runWorkDir, "artifacts", "session_recap.md"), "# recap\n")
|
||||
mustWriteFile(t, filepath.Join(runWorkDir, "reports", "audita.report.json"), "{}\n")
|
||||
mustWriteFile(t, filepath.Join(runWorkDir, "config", "audita.generated.yml"), "key: value\n")
|
||||
mustWriteFile(t, filepath.Join(runWorkDir, "prepare", "inputs", "session.yml"), "session_id: 2026-05-03\n")
|
||||
mustWriteFile(t, filepath.Join(runWorkDir, "transcribe", "outputs", "transcripts", "raw", "speaker.json"), "{}\n")
|
||||
mustWriteFile(t, filepath.Join(runWorkDir, "trim", "outputs", "transcripts", "trimmed.json"), "{}\n")
|
||||
mustWriteFile(t, filepath.Join(runWorkDir, "analyze", "outputs", "artifacts", "session_recap.md"), "# recap\n")
|
||||
mustWriteFile(t, filepath.Join(runWorkDir, "polish", "reports", "audita.report.json"), "{}\n")
|
||||
mustWriteFile(t, filepath.Join(runWorkDir, "merge", "config", "seriatim.generated.yml"), "key: value\n")
|
||||
mustWriteFile(t, filepath.Join(runWorkDir, "logs", "audita.stderr.log"), "stderr\n")
|
||||
mustWriteFile(t, filepath.Join(runWorkDir, "manifest.json"), "{}\n")
|
||||
|
||||
mustWriteFile(t, filepath.Join(sessionRoot, "transcripts", "trimmed.json"), "{}\n")
|
||||
mustWriteFile(t, filepath.Join(sessionRoot, "artifacts", "session_recap.md"), "# recap\n")
|
||||
}
|
||||
|
||||
type failKeyStore struct {
|
||||
|
||||
@@ -6,6 +6,7 @@ import (
|
||||
"fmt"
|
||||
"io"
|
||||
|
||||
"gitea.maximumdirect.net/eric/narratio/internal/artifacts"
|
||||
"gitea.maximumdirect.net/eric/narratio/internal/config"
|
||||
"gitea.maximumdirect.net/eric/narratio/internal/manifest"
|
||||
)
|
||||
@@ -83,7 +84,11 @@ func Resume(ctx context.Context, args []string, out io.Writer) error {
|
||||
}
|
||||
|
||||
func loadManifestIfPresent(ctx context.Context, cfg *config.Config) (*manifest.Manifest, error) {
|
||||
path := manifestPathFor(cfg)
|
||||
path := artifacts.SessionManifestPathForCampaign(
|
||||
cfg.Pipeline.Workspace.Root,
|
||||
cfg.Session.Campaign,
|
||||
cfg.Session.SessionID,
|
||||
)
|
||||
exists, err := fileExists(path)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("check manifest %q: %w", path, err)
|
||||
|
||||
@@ -16,7 +16,7 @@ import (
|
||||
func TestResumeStartsAfterCompletedStages(t *testing.T) {
|
||||
workspaceRoot := t.TempDir()
|
||||
pipelinePath, sessionPath := writeValidConfigFiles(t, workspaceRoot)
|
||||
manifestPath := filepath.Join(workspaceRoot, "work", "2026-05-03", "manifest.json")
|
||||
manifestPath := filepath.Join(workspaceRoot, "work", "sample-campaign", "2026-05-03", "manifest.json")
|
||||
|
||||
store := &manifest.LocalStore{}
|
||||
m := manifest.New("2026-05-03", time.Date(2026, 5, 3, 10, 0, 0, 0, time.UTC))
|
||||
@@ -25,7 +25,7 @@ func TestResumeStartsAfterCompletedStages(t *testing.T) {
|
||||
if err := store.Save(context.Background(), manifestPath, m); err != nil {
|
||||
t.Fatalf("save manifest: %v", err)
|
||||
}
|
||||
workRoot := filepath.Join(workspaceRoot, "work", "2026-05-03")
|
||||
workRoot := filepath.Join(workspaceRoot, "work", "sample-campaign", "2026-05-03")
|
||||
mustWriteTestFile(t, filepath.Join(workRoot, "transcripts", "raw", "alice.json"), `{"segments":[]}`)
|
||||
mustWriteTestFile(t, filepath.Join(workRoot, "inputs", "speakers.yml"), "match:\n - speaker: Alice\n match: [\"alice\"]\n")
|
||||
mustWriteTestFile(t, filepath.Join(workRoot, "inputs", "autocorrect.yml"), "[]\n")
|
||||
@@ -52,7 +52,7 @@ func TestResumeStartsAfterCompletedStages(t *testing.T) {
|
||||
func TestResumeNoRemainingStages(t *testing.T) {
|
||||
workspaceRoot := t.TempDir()
|
||||
pipelinePath, sessionPath := writeValidConfigFiles(t, workspaceRoot)
|
||||
manifestPath := filepath.Join(workspaceRoot, "work", "2026-05-03", "manifest.json")
|
||||
manifestPath := filepath.Join(workspaceRoot, "work", "sample-campaign", "2026-05-03", "manifest.json")
|
||||
|
||||
store := &manifest.LocalStore{}
|
||||
m := manifest.New("2026-05-03", time.Date(2026, 5, 3, 10, 0, 0, 0, time.UTC))
|
||||
@@ -81,7 +81,7 @@ func TestResumeForceRerunsSucceeded(t *testing.T) {
|
||||
}))
|
||||
defer srv.Close()
|
||||
pipelinePath, sessionPath := writeValidConfigFiles(t, workspaceRoot, srv.URL)
|
||||
manifestPath := filepath.Join(workspaceRoot, "work", "2026-05-03", "manifest.json")
|
||||
manifestPath := filepath.Join(workspaceRoot, "work", "sample-campaign", "2026-05-03", "manifest.json")
|
||||
|
||||
store := &manifest.LocalStore{}
|
||||
m := manifest.New("2026-05-03", time.Date(2026, 5, 3, 10, 0, 0, 0, time.UTC))
|
||||
@@ -105,8 +105,8 @@ func TestResumeForceRerunsSucceeded(t *testing.T) {
|
||||
func TestRunStageExecutesOnlySelectedStage(t *testing.T) {
|
||||
workspaceRoot := t.TempDir()
|
||||
pipelinePath, sessionPath := writeValidConfigFiles(t, workspaceRoot)
|
||||
manifestPath := filepath.Join(workspaceRoot, "work", "2026-05-03", "manifest.json")
|
||||
workRoot := filepath.Join(workspaceRoot, "work", "2026-05-03")
|
||||
manifestPath := filepath.Join(workspaceRoot, "work", "sample-campaign", "2026-05-03", "manifest.json")
|
||||
workRoot := filepath.Join(workspaceRoot, "work", "sample-campaign", "2026-05-03")
|
||||
mustWriteTestFile(t, filepath.Join(workRoot, "transcripts", "merged.json"), `{"segments":[]}`)
|
||||
mustWriteTestFile(t, filepath.Join(workRoot, "inputs", "glossary.yml"), "terms: []\n")
|
||||
|
||||
@@ -135,8 +135,8 @@ func TestRunStageExecutesOnlySelectedStage(t *testing.T) {
|
||||
func TestRunStageSkipAndForce(t *testing.T) {
|
||||
workspaceRoot := t.TempDir()
|
||||
pipelinePath, sessionPath := writeValidConfigFiles(t, workspaceRoot)
|
||||
manifestPath := filepath.Join(workspaceRoot, "work", "2026-05-03", "manifest.json")
|
||||
workRoot := filepath.Join(workspaceRoot, "work", "2026-05-03")
|
||||
manifestPath := filepath.Join(workspaceRoot, "work", "sample-campaign", "2026-05-03", "manifest.json")
|
||||
workRoot := filepath.Join(workspaceRoot, "work", "sample-campaign", "2026-05-03")
|
||||
mustWriteTestFile(t, filepath.Join(workRoot, "transcripts", "merged.json"), `{"segments":[]}`)
|
||||
mustWriteTestFile(t, filepath.Join(workRoot, "inputs", "glossary.yml"), "terms: []\n")
|
||||
|
||||
@@ -166,11 +166,57 @@ func TestRunStageSkipAndForce(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
func TestRunStageForceMarksDownstreamStaleAndResumeContinuesFromStale(t *testing.T) {
|
||||
workspaceRoot := t.TempDir()
|
||||
pipelinePath, sessionPath := writeValidConfigFiles(t, workspaceRoot)
|
||||
manifestPath := filepath.Join(workspaceRoot, "work", "sample-campaign", "2026-05-03", "manifest.json")
|
||||
workRoot := filepath.Join(workspaceRoot, "work", "sample-campaign", "2026-05-03")
|
||||
mustWriteTestFile(t, filepath.Join(workRoot, "transcripts", "merged.json"), `{"segments":[]}`)
|
||||
mustWriteTestFile(t, filepath.Join(workRoot, "inputs", "glossary.yml"), "terms: []\n")
|
||||
|
||||
store := &manifest.LocalStore{}
|
||||
seed := manifest.New("2026-05-03", time.Date(2026, 5, 3, 10, 0, 0, 0, time.UTC))
|
||||
for _, name := range []string{"prepare", "transcribe", "merge", "polish", "normalize", "trim", "analyze", "archive", "notify"} {
|
||||
seed.MarkStageSucceeded(name, time.Date(2026, 5, 3, 10, 1, 0, 0, time.UTC), nil)
|
||||
}
|
||||
if err := store.Save(context.Background(), manifestPath, seed); err != nil {
|
||||
t.Fatalf("save manifest: %v", err)
|
||||
}
|
||||
|
||||
var out bytes.Buffer
|
||||
err := RunStage(context.Background(), []string{"--config", pipelinePath, "--session", sessionPath, "--force", "polish"}, &out)
|
||||
if err != nil {
|
||||
t.Fatalf("RunStage(force) error = %v", err)
|
||||
}
|
||||
if !strings.Contains(out.String(), "stage=polish executed=1 skipped=0 force=true") {
|
||||
t.Fatalf("output = %q, want forced polish rerun", out.String())
|
||||
}
|
||||
|
||||
afterForce, err := store.Load(context.Background(), manifestPath)
|
||||
if err != nil {
|
||||
t.Fatalf("load manifest after force: %v", err)
|
||||
}
|
||||
for _, name := range []string{"normalize", "trim", "analyze", "archive", "notify"} {
|
||||
if afterForce.Stages[name] == nil || afterForce.Stages[name].Status != manifest.StatusStale {
|
||||
t.Fatalf("stage %q = %#v, want stale", name, afterForce.Stages[name])
|
||||
}
|
||||
}
|
||||
|
||||
out.Reset()
|
||||
err = Resume(context.Background(), []string{"--config", pipelinePath, "--session", sessionPath}, &out)
|
||||
if err != nil {
|
||||
t.Fatalf("Resume() error = %v", err)
|
||||
}
|
||||
if !strings.Contains(out.String(), "executed=5 skipped=0") {
|
||||
t.Fatalf("output = %q, want resume to execute normalize..notify", out.String())
|
||||
}
|
||||
}
|
||||
|
||||
func TestRunStageTrimExecutes(t *testing.T) {
|
||||
workspaceRoot := t.TempDir()
|
||||
pipelinePath, sessionPath := writeValidConfigFiles(t, workspaceRoot)
|
||||
manifestPath := filepath.Join(workspaceRoot, "work", "2026-05-03", "manifest.json")
|
||||
workRoot := filepath.Join(workspaceRoot, "work", "2026-05-03")
|
||||
manifestPath := filepath.Join(workspaceRoot, "work", "sample-campaign", "2026-05-03", "manifest.json")
|
||||
workRoot := filepath.Join(workspaceRoot, "work", "sample-campaign", "2026-05-03")
|
||||
mustWriteTestFile(t, filepath.Join(workRoot, "transcripts", "normalized.json"), `{"segments":[{"id":1},{"id":2}]}`)
|
||||
|
||||
var out bytes.Buffer
|
||||
@@ -198,8 +244,8 @@ func TestRunStageTrimExecutes(t *testing.T) {
|
||||
func TestRunStageNormalizeExecutes(t *testing.T) {
|
||||
workspaceRoot := t.TempDir()
|
||||
pipelinePath, sessionPath := writeValidConfigFiles(t, workspaceRoot)
|
||||
manifestPath := filepath.Join(workspaceRoot, "work", "2026-05-03", "manifest.json")
|
||||
workRoot := filepath.Join(workspaceRoot, "work", "2026-05-03")
|
||||
manifestPath := filepath.Join(workspaceRoot, "work", "sample-campaign", "2026-05-03", "manifest.json")
|
||||
workRoot := filepath.Join(workspaceRoot, "work", "sample-campaign", "2026-05-03")
|
||||
mustWriteTestFile(t, filepath.Join(workRoot, "transcripts", "processed.json"), `{"segments":[{"id":1},{"id":2}]}`)
|
||||
|
||||
var out bytes.Buffer
|
||||
|
||||
@@ -1,6 +1,8 @@
|
||||
package app
|
||||
|
||||
import (
|
||||
"time"
|
||||
|
||||
"gitea.maximumdirect.net/eric/narratio/internal/manifest"
|
||||
"gitea.maximumdirect.net/eric/narratio/internal/stage"
|
||||
)
|
||||
@@ -49,3 +51,43 @@ func firstNonSucceededIndex(stages []stage.Stage, m *manifest.Manifest) int {
|
||||
}
|
||||
return len(stages)
|
||||
}
|
||||
|
||||
func canonicalStageNames() []string {
|
||||
all := stage.All()
|
||||
out := make([]string, 0, len(all))
|
||||
for _, s := range all {
|
||||
if s == nil {
|
||||
continue
|
||||
}
|
||||
out = append(out, s.Name())
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
func downstreamStageNames(stageName string) []string {
|
||||
names := canonicalStageNames()
|
||||
for i, name := range names {
|
||||
if name != stageName {
|
||||
continue
|
||||
}
|
||||
return append([]string(nil), names[i+1:]...)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func invalidateDownstreamSucceededStages(m *manifest.Manifest, upstreamStage string, at time.Time) []string {
|
||||
if m == nil || m.Stages == nil {
|
||||
return nil
|
||||
}
|
||||
|
||||
invalidated := make([]string, 0)
|
||||
for _, downstream := range downstreamStageNames(upstreamStage) {
|
||||
sr := m.Stages[downstream]
|
||||
if sr == nil || sr.Status != manifest.StatusSucceeded {
|
||||
continue
|
||||
}
|
||||
m.MarkStageStale(downstream, at, "upstream stage rerun with force")
|
||||
invalidated = append(invalidated, downstream)
|
||||
}
|
||||
return invalidated
|
||||
}
|
||||
|
||||
@@ -1,6 +1,7 @@
|
||||
package app
|
||||
|
||||
import (
|
||||
"reflect"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
@@ -40,3 +41,48 @@ func TestDecideStageActions(t *testing.T) {
|
||||
t.Fatalf("forced prepare action = %q, want %q", forced[0].Action, stageActionRun)
|
||||
}
|
||||
}
|
||||
|
||||
func TestDownstreamStageNames(t *testing.T) {
|
||||
got := downstreamStageNames("polish")
|
||||
want := []string{"normalize", "trim", "analyze", "archive", "notify"}
|
||||
if !reflect.DeepEqual(got, want) {
|
||||
t.Fatalf("downstreamStageNames(polish) = %#v, want %#v", got, want)
|
||||
}
|
||||
|
||||
missing := downstreamStageNames("unknown")
|
||||
if len(missing) != 0 {
|
||||
t.Fatalf("downstreamStageNames(unknown) = %#v, want empty", missing)
|
||||
}
|
||||
}
|
||||
|
||||
func TestInvalidateDownstreamSucceededStages(t *testing.T) {
|
||||
now := time.Now().UTC()
|
||||
m := manifest.New("2026-05-03", now)
|
||||
m.MarkStageSucceeded("prepare", now, nil)
|
||||
m.MarkStageSucceeded("transcribe", now, nil)
|
||||
m.MarkStageSucceeded("merge", now, nil)
|
||||
m.MarkStageSucceeded("polish", now, nil)
|
||||
m.MarkStageSucceeded("normalize", now, nil)
|
||||
m.MarkStageSucceeded("trim", now, nil)
|
||||
m.MarkStageFailed("analyze", now, "analysis failed")
|
||||
m.MarkStageSucceeded("archive", now, nil)
|
||||
m.MarkStageSucceeded("notify", now, nil)
|
||||
|
||||
got := invalidateDownstreamSucceededStages(m, "polish", now.Add(1*time.Second))
|
||||
want := []string{"normalize", "trim", "archive", "notify"}
|
||||
if !reflect.DeepEqual(got, want) {
|
||||
t.Fatalf("invalidateDownstreamSucceededStages() = %#v, want %#v", got, want)
|
||||
}
|
||||
|
||||
for _, stageName := range want {
|
||||
if m.Stages[stageName].Status != manifest.StatusStale {
|
||||
t.Fatalf("%s status = %q, want stale", stageName, m.Stages[stageName].Status)
|
||||
}
|
||||
}
|
||||
if m.Stages["analyze"].Status != manifest.StatusFailed {
|
||||
t.Fatalf("analyze status = %q, want failed", m.Stages["analyze"].Status)
|
||||
}
|
||||
if m.Stages["prepare"].Status != manifest.StatusSucceeded {
|
||||
t.Fatalf("prepare status = %q, want succeeded", m.Stages["prepare"].Status)
|
||||
}
|
||||
}
|
||||
|
||||
@@ -5,7 +5,6 @@ import (
|
||||
"fmt"
|
||||
"log/slog"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
|
||||
"gitea.maximumdirect.net/eric/narratio/internal/adapters/audita"
|
||||
@@ -27,11 +26,13 @@ type RunOptions struct {
|
||||
}
|
||||
|
||||
type RunSummary struct {
|
||||
SessionID string
|
||||
ManifestPath string
|
||||
StageNames []string
|
||||
Executed []string
|
||||
Skipped []string
|
||||
SessionID string
|
||||
RunID string
|
||||
ManifestPath string
|
||||
RunManifestPath string
|
||||
StageNames []string
|
||||
Executed []string
|
||||
Skipped []string
|
||||
}
|
||||
|
||||
func executeStages(ctx context.Context, cfg *config.Config, stages []stage.Stage, opts RunOptions) (*RunSummary, error) {
|
||||
@@ -93,12 +94,12 @@ func executeStages(ctx context.Context, cfg *config.Config, stages []stage.Stage
|
||||
}
|
||||
|
||||
artifactStore := env.ArtifactStore
|
||||
paths, err := artifactStore.EnsureLayout(cfg.Session.SessionID)
|
||||
paths, err := artifactStore.EnsureLayoutFor(cfg.Session.Campaign, cfg.Session.SessionID)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("prepare workdir: %w", err)
|
||||
}
|
||||
|
||||
lock, err := artifactStore.AcquireSessionLock(cfg.Session.SessionID)
|
||||
lock, err := artifactStore.AcquireSessionLockFor(cfg.Session.Campaign, cfg.Session.SessionID)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("acquire session lock: %w", err)
|
||||
}
|
||||
@@ -111,7 +112,11 @@ func executeStages(ctx context.Context, cfg *config.Config, stages []stage.Stage
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
identityChanged, err := ensureManifestIdentity(cfg, m)
|
||||
runID, err := artifacts.NewRunID()
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("generate run id: %w", err)
|
||||
}
|
||||
identityChanged, err := ensureManifestIdentity(cfg, m, runID)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("initialize manifest identity: %w", err)
|
||||
}
|
||||
@@ -120,6 +125,29 @@ func executeStages(ctx context.Context, cfg *config.Config, stages []stage.Stage
|
||||
return nil, fmt.Errorf("save manifest identity %q: %w", manifestPath, err)
|
||||
}
|
||||
}
|
||||
runManifestPath := artifacts.SessionRunManifestPathForCampaign(
|
||||
cfg.Pipeline.Workspace.Root,
|
||||
cfg.Session.Campaign,
|
||||
cfg.Session.SessionID,
|
||||
runID,
|
||||
)
|
||||
runManifestStore := &manifest.LocalStore{}
|
||||
runManifest, err := runManifestStore.CreateRun(
|
||||
ctx,
|
||||
cfg.Session.SessionID,
|
||||
cfg.Session.Campaign,
|
||||
runID,
|
||||
opts.Force,
|
||||
requestedStageNames(stages),
|
||||
)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("create run manifest: %w", err)
|
||||
}
|
||||
runManifest.SessionManifestPath = manifestPath
|
||||
syncRunManifestIdentityFromSession(m, runManifest)
|
||||
if err := runManifestStore.SaveRun(ctx, runManifestPath, runManifest); err != nil {
|
||||
return nil, fmt.Errorf("save initial run manifest %q: %w", runManifestPath, err)
|
||||
}
|
||||
|
||||
stageEnv := env
|
||||
|
||||
@@ -134,12 +162,23 @@ func executeStages(ctx context.Context, cfg *config.Config, stages []stage.Stage
|
||||
|
||||
if d.Action == stageActionSkip {
|
||||
skipped = append(skipped, s.Name())
|
||||
skipAt := nowUTC()
|
||||
runManifest.SetStageAction(s.Name(), manifest.RunStageActionSkip, skipAt)
|
||||
runManifest.MarkStageSkipped(s.Name(), skipAt, "already_succeeded")
|
||||
if err := runManifestStore.SaveRun(ctx, runManifestPath, runManifest); err != nil {
|
||||
return nil, fmt.Errorf("save run manifest after skip %q: %w", s.Name(), err)
|
||||
}
|
||||
env.Logger.Info("skipping stage", "stage", s.Name(), "reason", "already_succeeded", "force", opts.Force)
|
||||
continue
|
||||
}
|
||||
executed = append(executed, s.Name())
|
||||
|
||||
now := nowUTC()
|
||||
runManifest.SetStageAction(s.Name(), manifest.RunStageActionRun, now)
|
||||
runManifest.MarkStageRunning(s.Name(), now)
|
||||
if err := runManifestStore.SaveRun(ctx, runManifestPath, runManifest); err != nil {
|
||||
return nil, fmt.Errorf("save run manifest before stage %q: %w", s.Name(), err)
|
||||
}
|
||||
m.MarkStageRunning(s.Name(), now)
|
||||
env.Logger.Info("starting stage", "stage", s.Name())
|
||||
if err := env.ManifestStore.Save(ctx, manifestPath, m); err != nil {
|
||||
@@ -149,35 +188,65 @@ func executeStages(ctx context.Context, cfg *config.Config, stages []stage.Stage
|
||||
|
||||
result, err := s.Run(ctx, stageEnv, m)
|
||||
if err != nil {
|
||||
m.MarkStageFailed(s.Name(), nowUTC(), err.Error())
|
||||
failedAt := nowUTC()
|
||||
m.MarkStageFailed(s.Name(), failedAt, err.Error())
|
||||
if saveErr := env.ManifestStore.Save(ctx, manifestPath, m); saveErr != nil {
|
||||
return nil, fmt.Errorf("stage %q failed (%v) and manifest save failed (%v)", s.Name(), err, saveErr)
|
||||
}
|
||||
runManifest.MarkStageFailed(s.Name(), failedAt, err.Error())
|
||||
syncRunManifestIdentityFromSession(m, runManifest)
|
||||
if saveErr := runManifestStore.SaveRun(ctx, runManifestPath, runManifest); saveErr != nil {
|
||||
return nil, fmt.Errorf("stage %q failed (%v) and run-manifest save failed (%v)", s.Name(), err, saveErr)
|
||||
}
|
||||
env.Logger.Info("stage failed", "stage", s.Name(), "error", err)
|
||||
return nil, fmt.Errorf("stage %q failed: %w", s.Name(), err)
|
||||
}
|
||||
|
||||
outputs := mapResultOutputs(result)
|
||||
m.MarkStageSucceeded(s.Name(), nowUTC(), outputs)
|
||||
outputs := mapResultOutputs(result, runID)
|
||||
succeededAt := nowUTC()
|
||||
m.MarkStageSucceeded(s.Name(), succeededAt, outputs)
|
||||
applyStageResultToManifest(m, s.Name(), result)
|
||||
if opts.Force {
|
||||
invalidateDownstreamSucceededStages(m, s.Name(), succeededAt)
|
||||
}
|
||||
|
||||
if err := env.ManifestStore.Save(ctx, manifestPath, m); err != nil {
|
||||
return nil, fmt.Errorf("save manifest after stage %q: %w", s.Name(), err)
|
||||
}
|
||||
runManifest.MarkStageSucceeded(s.Name(), succeededAt, outputs)
|
||||
applyStageResultToRunManifest(runManifest, s.Name(), result)
|
||||
syncRunManifestIdentityFromSession(m, runManifest)
|
||||
if err := runManifestStore.SaveRun(ctx, runManifestPath, runManifest); err != nil {
|
||||
return nil, fmt.Errorf("save run manifest after stage %q: %w", s.Name(), err)
|
||||
}
|
||||
env.Logger.Debug("manifest saved", "stage", s.Name(), "transition", "succeeded", "path", manifestPath)
|
||||
env.Logger.Info("stage succeeded", "stage", s.Name())
|
||||
}
|
||||
|
||||
if err := runPostArchiveCleanup(ctx, env, manifestPath, m, executed); err != nil {
|
||||
failedAt := nowUTC()
|
||||
runManifest.MarkFailed(failedAt, err.Error())
|
||||
syncRunManifestIdentityFromSession(m, runManifest)
|
||||
if saveErr := runManifestStore.SaveRun(ctx, runManifestPath, runManifest); saveErr != nil {
|
||||
return nil, fmt.Errorf("post-archive cleanup failed (%v) and run-manifest save failed (%v)", err, saveErr)
|
||||
}
|
||||
return nil, fmt.Errorf("post-archive cleanup: %w", err)
|
||||
}
|
||||
completedAt := nowUTC()
|
||||
runManifest.MarkSucceeded(completedAt)
|
||||
syncRunManifestIdentityFromSession(m, runManifest)
|
||||
if err := runManifestStore.SaveRun(ctx, runManifestPath, runManifest); err != nil {
|
||||
return nil, fmt.Errorf("save final run manifest %q: %w", runManifestPath, err)
|
||||
}
|
||||
|
||||
return &RunSummary{
|
||||
SessionID: cfg.Session.SessionID,
|
||||
ManifestPath: manifestPath,
|
||||
StageNames: runNames,
|
||||
Executed: executed,
|
||||
Skipped: skipped,
|
||||
SessionID: cfg.Session.SessionID,
|
||||
RunID: runID,
|
||||
ManifestPath: manifestPath,
|
||||
RunManifestPath: runManifestPath,
|
||||
StageNames: runNames,
|
||||
Executed: executed,
|
||||
Skipped: skipped,
|
||||
}, nil
|
||||
}
|
||||
|
||||
@@ -317,11 +386,12 @@ func fileExists(path string) (bool, error) {
|
||||
return false, err
|
||||
}
|
||||
|
||||
func mapResultOutputs(result *stage.StageResult) []manifest.ArtifactRecord {
|
||||
func mapResultOutputs(result *stage.StageResult, runID string) []manifest.ArtifactRecord {
|
||||
if result == nil || len(result.Outputs) == 0 {
|
||||
return nil
|
||||
}
|
||||
|
||||
runID = strings.TrimSpace(runID)
|
||||
out := make([]manifest.ArtifactRecord, 0, len(result.Outputs))
|
||||
for _, ref := range result.Outputs {
|
||||
localPath := ref.AbsolutePath
|
||||
@@ -329,10 +399,11 @@ func mapResultOutputs(result *stage.StageResult) []manifest.ArtifactRecord {
|
||||
localPath = ref.RelativePath
|
||||
}
|
||||
out = append(out, manifest.ArtifactRecord{
|
||||
Kind: ref.Kind,
|
||||
LocalPath: localPath,
|
||||
RemoteKey: ref.RemoteKey,
|
||||
Checksum: ref.Checksum,
|
||||
Kind: ref.Kind,
|
||||
LocalPath: localPath,
|
||||
ProducerRunID: runID,
|
||||
RemoteKey: ref.RemoteKey,
|
||||
Checksum: ref.Checksum,
|
||||
})
|
||||
}
|
||||
|
||||
@@ -358,7 +429,7 @@ func applyStageResultToManifest(m *manifest.Manifest, stageName string, result *
|
||||
}
|
||||
}
|
||||
|
||||
func ensureManifestIdentity(cfg *config.Config, m *manifest.Manifest) (bool, error) {
|
||||
func ensureManifestIdentity(cfg *config.Config, m *manifest.Manifest, runID string) (bool, error) {
|
||||
if cfg == nil || cfg.Pipeline == nil || cfg.Session == nil || m == nil {
|
||||
return false, nil
|
||||
}
|
||||
@@ -374,16 +445,13 @@ func ensureManifestIdentity(cfg *config.Config, m *manifest.Manifest) (bool, err
|
||||
m.Campaign = campaign
|
||||
changed = true
|
||||
}
|
||||
if m.RunID == "" {
|
||||
runID, err := artifacts.NewRunID()
|
||||
if err != nil {
|
||||
return false, err
|
||||
}
|
||||
runID = strings.TrimSpace(runID)
|
||||
if runID != "" && m.RunID != runID {
|
||||
m.RunID = runID
|
||||
changed = true
|
||||
}
|
||||
if m.LocalWorkDir == "" && campaign != "" && sessionID != "" && m.RunID != "" {
|
||||
m.LocalWorkDir = artifacts.SessionRunWorkDir(cfg.Pipeline.Workspace.Root, campaign, sessionID, m.RunID)
|
||||
m.LocalWorkDir = artifacts.SessionRunRootForCampaign(cfg.Pipeline.Workspace.Root, campaign, sessionID, m.RunID)
|
||||
changed = true
|
||||
}
|
||||
if m.LocalSpoolDir == "" && campaign != "" && sessionID != "" && m.RunID != "" && strings.TrimSpace(cfg.Pipeline.Spool.Root) != "" {
|
||||
@@ -410,8 +478,54 @@ func ensureManifestIdentity(cfg *config.Config, m *manifest.Manifest) (bool, err
|
||||
return changed, nil
|
||||
}
|
||||
|
||||
func requestedStageNames(stages []stage.Stage) []string {
|
||||
out := make([]string, 0, len(stages))
|
||||
for _, s := range stages {
|
||||
if s == nil {
|
||||
continue
|
||||
}
|
||||
out = append(out, s.Name())
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
func applyStageResultToRunManifest(m *manifest.RunManifest, stageName string, result *stage.StageResult) {
|
||||
if m == nil || result == nil {
|
||||
return
|
||||
}
|
||||
sr := m.Stages[stageName]
|
||||
if sr == nil {
|
||||
return
|
||||
}
|
||||
if len(result.Logs) > 0 {
|
||||
sr.Logs = append([]string(nil), result.Logs...)
|
||||
}
|
||||
if len(result.GeneratedConfigs) > 0 {
|
||||
sr.GeneratedConfigs = append([]string(nil), result.GeneratedConfigs...)
|
||||
}
|
||||
if len(result.Metadata) > 0 {
|
||||
sr.Metadata = result.Metadata
|
||||
}
|
||||
}
|
||||
|
||||
func syncRunManifestIdentityFromSession(session *manifest.Manifest, run *manifest.RunManifest) {
|
||||
if session == nil || run == nil {
|
||||
return
|
||||
}
|
||||
run.Campaign = session.Campaign
|
||||
run.LocalWorkDir = session.LocalWorkDir
|
||||
run.LocalSpoolDir = session.LocalSpoolDir
|
||||
run.S3Bucket = session.S3Bucket
|
||||
run.S3SessionPrefix = session.S3SessionPrefix
|
||||
run.S3RunPrefix = session.S3RunPrefix
|
||||
}
|
||||
|
||||
func manifestPathFor(cfg *config.Config) string {
|
||||
return filepath.Join(cfg.Pipeline.Workspace.Root, "work", cfg.Session.SessionID, "manifest.json")
|
||||
return artifacts.SessionManifestPathForCampaign(
|
||||
cfg.Pipeline.Workspace.Root,
|
||||
cfg.Session.Campaign,
|
||||
cfg.Session.SessionID,
|
||||
)
|
||||
}
|
||||
|
||||
func needsObjectStoreForRun(cfg *config.Config, stages []stage.Stage) bool {
|
||||
|
||||
@@ -235,6 +235,56 @@ func TestExecuteStagesForceRerunsSucceeded(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
func TestExecuteStagesForceSuccessInvalidatesDownstreamSucceededStages(t *testing.T) {
|
||||
cfg := testConfig(t)
|
||||
manifestPath := manifestPathFor(cfg)
|
||||
store := &manifest.LocalStore{}
|
||||
|
||||
existing := manifest.New(cfg.Session.SessionID, time.Date(2026, 5, 3, 1, 0, 0, 0, time.UTC))
|
||||
for _, stageName := range []string{"prepare", "transcribe", "merge", "polish", "normalize", "trim", "archive", "notify"} {
|
||||
existing.MarkStageSucceeded(stageName, time.Date(2026, 5, 3, 1, 1, 0, 0, time.UTC), nil)
|
||||
}
|
||||
existing.MarkStageFailed("analyze", time.Date(2026, 5, 3, 1, 1, 0, 0, time.UTC), "previous analyze failure")
|
||||
if err := os.MkdirAll(filepath.Dir(manifestPath), 0o755); err != nil {
|
||||
t.Fatalf("MkdirAll() error = %v", err)
|
||||
}
|
||||
if err := store.Save(context.Background(), manifestPath, existing); err != nil {
|
||||
t.Fatalf("Save manifest error = %v", err)
|
||||
}
|
||||
|
||||
runs := 0
|
||||
stageToRun := countingStage{name: "polish", runs: &runs}
|
||||
summary, err := executeStages(context.Background(), cfg, []stage.Stage{stageToRun}, RunOptions{Force: true})
|
||||
if err != nil {
|
||||
t.Fatalf("executeStages() error = %v", err)
|
||||
}
|
||||
if runs != 1 {
|
||||
t.Fatalf("runs = %d, want 1 with force", runs)
|
||||
}
|
||||
if len(summary.Executed) != 1 || summary.Executed[0] != "polish" || len(summary.Skipped) != 0 {
|
||||
t.Fatalf("summary = %#v, want executed polish", summary)
|
||||
}
|
||||
|
||||
loaded, err := store.Load(context.Background(), manifestPath)
|
||||
if err != nil {
|
||||
t.Fatalf("Load manifest error = %v", err)
|
||||
}
|
||||
if loaded.Stages["polish"] == nil || loaded.Stages["polish"].Status != manifest.StatusSucceeded {
|
||||
t.Fatalf("polish status = %#v, want succeeded", loaded.Stages["polish"])
|
||||
}
|
||||
for _, stageName := range []string{"normalize", "trim", "archive", "notify"} {
|
||||
if loaded.Stages[stageName] == nil || loaded.Stages[stageName].Status != manifest.StatusStale {
|
||||
t.Fatalf("%s status = %#v, want stale", stageName, loaded.Stages[stageName])
|
||||
}
|
||||
}
|
||||
if loaded.Stages["analyze"] == nil || loaded.Stages["analyze"].Status != manifest.StatusFailed {
|
||||
t.Fatalf("analyze status = %#v, want preserved failed", loaded.Stages["analyze"])
|
||||
}
|
||||
if loaded.Stages["transcribe"] == nil || loaded.Stages["transcribe"].Status != manifest.StatusSucceeded {
|
||||
t.Fatalf("transcribe status = %#v, want preserved succeeded", loaded.Stages["transcribe"])
|
||||
}
|
||||
}
|
||||
|
||||
func TestExecuteStagesFailureUpdatesManifest(t *testing.T) {
|
||||
cfg := testConfig(t)
|
||||
|
||||
@@ -280,7 +330,14 @@ func TestExecuteStagesLoadsExistingManifest(t *testing.T) {
|
||||
|
||||
existing := manifest.New(cfg.Session.SessionID, time.Date(2026, 5, 3, 1, 0, 0, 0, time.UTC))
|
||||
existing.MarkStageSucceeded("prepare", time.Date(2026, 5, 3, 1, 1, 0, 0, time.UTC), nil)
|
||||
audioPath := filepath.Join(cfg.Pipeline.Workspace.Root, "work", cfg.Session.SessionID, "audio", "alice.flac")
|
||||
audioPath := filepath.Join(
|
||||
cfg.Pipeline.Workspace.Root,
|
||||
"work",
|
||||
cfg.Session.Campaign,
|
||||
cfg.Session.SessionID,
|
||||
"audio",
|
||||
"alice.flac",
|
||||
)
|
||||
if err := os.MkdirAll(filepath.Dir(audioPath), 0o755); err != nil {
|
||||
t.Fatalf("MkdirAll() error = %v", err)
|
||||
}
|
||||
@@ -315,6 +372,208 @@ func TestExecuteStagesLoadsExistingManifest(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
func TestExecuteStagesCreatesRunManifestPerInvocation(t *testing.T) {
|
||||
cfg := testConfig(t)
|
||||
run1, err := executeStages(context.Background(), cfg, []stage.Stage{BuildFullPlan()[0]}, RunOptions{})
|
||||
if err != nil {
|
||||
t.Fatalf("first executeStages() error = %v", err)
|
||||
}
|
||||
run2, err := executeStages(context.Background(), cfg, []stage.Stage{BuildFullPlan()[0]}, RunOptions{Force: true})
|
||||
if err != nil {
|
||||
t.Fatalf("second executeStages() error = %v", err)
|
||||
}
|
||||
|
||||
if run1.RunID == "" || run2.RunID == "" {
|
||||
t.Fatalf("run ids must be set, got %q and %q", run1.RunID, run2.RunID)
|
||||
}
|
||||
if run1.RunID == run2.RunID {
|
||||
t.Fatalf("expected distinct run ids, got %q", run1.RunID)
|
||||
}
|
||||
if run1.RunManifestPath == "" || run2.RunManifestPath == "" {
|
||||
t.Fatalf("run manifest paths must be set, got %q and %q", run1.RunManifestPath, run2.RunManifestPath)
|
||||
}
|
||||
if run1.RunManifestPath == run2.RunManifestPath {
|
||||
t.Fatalf("expected distinct run manifest paths, got %q", run1.RunManifestPath)
|
||||
}
|
||||
for _, path := range []string{run1.RunManifestPath, run2.RunManifestPath} {
|
||||
if _, statErr := os.Stat(path); statErr != nil {
|
||||
t.Fatalf("run manifest missing at %q: %v", path, statErr)
|
||||
}
|
||||
}
|
||||
|
||||
store := &manifest.LocalStore{}
|
||||
sessionManifest, err := store.Load(context.Background(), run2.ManifestPath)
|
||||
if err != nil {
|
||||
t.Fatalf("Load session manifest error = %v", err)
|
||||
}
|
||||
if sessionManifest.RunID != run2.RunID {
|
||||
t.Fatalf("session manifest run_id = %q, want latest run id %q", sessionManifest.RunID, run2.RunID)
|
||||
}
|
||||
}
|
||||
|
||||
func TestExecuteStagesRunManifestRecordsSkippedStage(t *testing.T) {
|
||||
cfg := testConfig(t)
|
||||
store := &manifest.LocalStore{}
|
||||
manifestPath := manifestPathFor(cfg)
|
||||
existing := manifest.New(cfg.Session.SessionID, time.Date(2026, 5, 3, 1, 0, 0, 0, time.UTC))
|
||||
existing.MarkStageSucceeded("transcribe", time.Date(2026, 5, 3, 1, 1, 0, 0, time.UTC), nil)
|
||||
if err := os.MkdirAll(filepath.Dir(manifestPath), 0o755); err != nil {
|
||||
t.Fatalf("MkdirAll() error = %v", err)
|
||||
}
|
||||
if err := store.Save(context.Background(), manifestPath, existing); err != nil {
|
||||
t.Fatalf("Save manifest error = %v", err)
|
||||
}
|
||||
|
||||
summary, err := executeStages(context.Background(), cfg, []stage.Stage{BuildFullPlan()[1]}, RunOptions{})
|
||||
if err != nil {
|
||||
t.Fatalf("executeStages() error = %v", err)
|
||||
}
|
||||
if len(summary.Skipped) != 1 || summary.Skipped[0] != "transcribe" {
|
||||
t.Fatalf("summary = %#v, want skipped transcribe", summary)
|
||||
}
|
||||
|
||||
runManifest, err := store.LoadRun(context.Background(), summary.RunManifestPath)
|
||||
if err != nil {
|
||||
t.Fatalf("LoadRun() error = %v", err)
|
||||
}
|
||||
sr := runManifest.Stages["transcribe"]
|
||||
if sr == nil {
|
||||
t.Fatal("run manifest transcribe stage missing")
|
||||
}
|
||||
if sr.Action != manifest.RunStageActionSkip {
|
||||
t.Fatalf("action = %q, want %q", sr.Action, manifest.RunStageActionSkip)
|
||||
}
|
||||
if sr.Status != manifest.StatusSkipped {
|
||||
t.Fatalf("status = %q, want %q", sr.Status, manifest.StatusSkipped)
|
||||
}
|
||||
}
|
||||
|
||||
func TestExecuteStagesRunLocalArtifactsAndCanonicalPromotion(t *testing.T) {
|
||||
cfg := testConfig(t)
|
||||
stages := []stage.Stage{
|
||||
BuildFullPlan()[0], // prepare
|
||||
BuildFullPlan()[1], // transcribe
|
||||
BuildFullPlan()[2], // merge
|
||||
BuildFullPlan()[3], // polish
|
||||
BuildFullPlan()[4], // normalize
|
||||
BuildFullPlan()[5], // trim
|
||||
}
|
||||
|
||||
summary, err := executeStages(context.Background(), cfg, stages, RunOptions{})
|
||||
if err != nil {
|
||||
t.Fatalf("executeStages() error = %v", err)
|
||||
}
|
||||
if summary.RunID == "" {
|
||||
t.Fatal("run id must be set")
|
||||
}
|
||||
|
||||
runRoot := artifacts.SessionRunRootForCampaign(
|
||||
cfg.Pipeline.Workspace.Root,
|
||||
cfg.Session.Campaign,
|
||||
cfg.Session.SessionID,
|
||||
summary.RunID,
|
||||
)
|
||||
paths := artifacts.NewLocalStore(cfg.Pipeline.Workspace.Root).SessionPathsFor(cfg.Session.Campaign, cfg.Session.SessionID)
|
||||
|
||||
runLocalChecks := []string{
|
||||
filepath.Join(runRoot, "transcribe", "outputs", "transcripts", "raw", "alice.json"),
|
||||
filepath.Join(runRoot, "merge", "logs", "seriatim.stdout.log"),
|
||||
filepath.Join(runRoot, "polish", "config", "audita.generated.yml"),
|
||||
filepath.Join(runRoot, "normalize", "logs", "seriatim.normalize.stdout.log"),
|
||||
filepath.Join(runRoot, "trim", "outputs", "transcripts", "trimmed.json"),
|
||||
}
|
||||
for _, p := range runLocalChecks {
|
||||
if _, statErr := os.Stat(p); statErr != nil {
|
||||
t.Fatalf("run-local artifact missing at %q: %v", p, statErr)
|
||||
}
|
||||
}
|
||||
|
||||
canonicalChecks := []string{
|
||||
filepath.Join(paths.TranscriptsRawDir, "alice.json"),
|
||||
filepath.Join(paths.TranscriptsDir, "merged.json"),
|
||||
filepath.Join(paths.TranscriptsDir, "processed.json"),
|
||||
filepath.Join(paths.TranscriptsDir, "normalized.json"),
|
||||
filepath.Join(paths.TranscriptsDir, "trimmed.json"),
|
||||
}
|
||||
for _, p := range canonicalChecks {
|
||||
if _, statErr := os.Stat(p); statErr != nil {
|
||||
t.Fatalf("canonical promoted artifact missing at %q: %v", p, statErr)
|
||||
}
|
||||
}
|
||||
|
||||
store := &manifest.LocalStore{}
|
||||
sessionManifest, err := store.Load(context.Background(), summary.ManifestPath)
|
||||
if err != nil {
|
||||
t.Fatalf("Load manifest error = %v", err)
|
||||
}
|
||||
if got := sessionManifest.Stages["trim"]; got == nil || len(got.Outputs) == 0 {
|
||||
t.Fatalf("trim stage outputs missing in session manifest: %#v", got)
|
||||
}
|
||||
for _, out := range sessionManifest.Stages["trim"].Outputs {
|
||||
if strings.Contains(out.LocalPath, string(filepath.Separator)+"runs"+string(filepath.Separator)) {
|
||||
t.Fatalf("session manifest output should be canonical, got run-local path %q", out.LocalPath)
|
||||
}
|
||||
if out.ProducerRunID != summary.RunID {
|
||||
t.Fatalf("producer_run_id = %q, want %q", out.ProducerRunID, summary.RunID)
|
||||
}
|
||||
}
|
||||
|
||||
runManifest, err := store.LoadRun(context.Background(), summary.RunManifestPath)
|
||||
if err != nil {
|
||||
t.Fatalf("LoadRun() error = %v", err)
|
||||
}
|
||||
mergeStage := runManifest.Stages["merge"]
|
||||
if mergeStage == nil || len(mergeStage.Logs) == 0 {
|
||||
t.Fatalf("merge logs missing in run manifest: %#v", mergeStage)
|
||||
}
|
||||
for _, logPath := range mergeStage.Logs {
|
||||
if !strings.Contains(logPath, filepath.Join("runs", summary.RunID, "merge", "logs")) {
|
||||
t.Fatalf("run manifest merge log path = %q, want run-local merge logs path", logPath)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestExecuteStagesSkippedStagePreservesExistingOutputsProvenance(t *testing.T) {
|
||||
cfg := testConfig(t)
|
||||
store := &manifest.LocalStore{}
|
||||
manifestPath := manifestPathFor(cfg)
|
||||
|
||||
existing := manifest.New(cfg.Session.SessionID, time.Date(2026, 5, 3, 1, 0, 0, 0, time.UTC))
|
||||
existing.MarkStageSucceeded("transcribe", time.Date(2026, 5, 3, 1, 1, 0, 0, time.UTC), []manifest.ArtifactRecord{
|
||||
{
|
||||
Kind: "transcript_raw",
|
||||
LocalPath: "transcripts/raw/alice.json",
|
||||
ProducerRunID: "20260501T000000Z-deadbeef",
|
||||
},
|
||||
})
|
||||
if err := os.MkdirAll(filepath.Dir(manifestPath), 0o755); err != nil {
|
||||
t.Fatalf("MkdirAll() error = %v", err)
|
||||
}
|
||||
if err := store.Save(context.Background(), manifestPath, existing); err != nil {
|
||||
t.Fatalf("Save manifest error = %v", err)
|
||||
}
|
||||
|
||||
summary, err := executeStages(context.Background(), cfg, []stage.Stage{BuildFullPlan()[1]}, RunOptions{})
|
||||
if err != nil {
|
||||
t.Fatalf("executeStages() error = %v", err)
|
||||
}
|
||||
if len(summary.Skipped) != 1 || summary.Skipped[0] != "transcribe" {
|
||||
t.Fatalf("summary = %#v, want skipped transcribe", summary)
|
||||
}
|
||||
|
||||
loaded, err := store.Load(context.Background(), manifestPath)
|
||||
if err != nil {
|
||||
t.Fatalf("Load manifest error = %v", err)
|
||||
}
|
||||
got := loaded.Stages["transcribe"]
|
||||
if got == nil || len(got.Outputs) != 1 {
|
||||
t.Fatalf("transcribe outputs = %#v, want one preserved output", got)
|
||||
}
|
||||
if got.Outputs[0].ProducerRunID != "20260501T000000Z-deadbeef" {
|
||||
t.Fatalf("producer_run_id = %q, want preserved value", got.Outputs[0].ProducerRunID)
|
||||
}
|
||||
}
|
||||
|
||||
func TestAdapterBackedStageFailureMarksManifestFailed(t *testing.T) {
|
||||
cases := []struct {
|
||||
name string
|
||||
@@ -341,7 +600,7 @@ func TestAdapterBackedStageFailureMarksManifestFailed(t *testing.T) {
|
||||
tc.env.ArtifactStore = artifactStore
|
||||
tc.env.ManifestStore = &manifest.LocalStore{}
|
||||
if tc.name == "transcribe" {
|
||||
paths, ensureErr := artifactStore.EnsureLayout(cfg.Session.SessionID)
|
||||
paths, ensureErr := artifactStore.EnsureLayoutFor(cfg.Session.Campaign, cfg.Session.SessionID)
|
||||
if ensureErr != nil {
|
||||
t.Fatalf("EnsureLayout() error = %v", ensureErr)
|
||||
}
|
||||
@@ -356,7 +615,7 @@ func TestAdapterBackedStageFailureMarksManifestFailed(t *testing.T) {
|
||||
}
|
||||
}
|
||||
if tc.name == "merge" {
|
||||
paths, ensureErr := artifactStore.EnsureLayout(cfg.Session.SessionID)
|
||||
paths, ensureErr := artifactStore.EnsureLayoutFor(cfg.Session.Campaign, cfg.Session.SessionID)
|
||||
if ensureErr != nil {
|
||||
t.Fatalf("EnsureLayout() error = %v", ensureErr)
|
||||
}
|
||||
@@ -375,7 +634,7 @@ func TestAdapterBackedStageFailureMarksManifestFailed(t *testing.T) {
|
||||
}
|
||||
}
|
||||
if tc.name == "polish" {
|
||||
paths, ensureErr := artifactStore.EnsureLayout(cfg.Session.SessionID)
|
||||
paths, ensureErr := artifactStore.EnsureLayoutFor(cfg.Session.Campaign, cfg.Session.SessionID)
|
||||
if ensureErr != nil {
|
||||
t.Fatalf("EnsureLayout() error = %v", ensureErr)
|
||||
}
|
||||
@@ -387,7 +646,7 @@ func TestAdapterBackedStageFailureMarksManifestFailed(t *testing.T) {
|
||||
}
|
||||
}
|
||||
if tc.name == "analyze" {
|
||||
paths, ensureErr := artifactStore.EnsureLayout(cfg.Session.SessionID)
|
||||
paths, ensureErr := artifactStore.EnsureLayoutFor(cfg.Session.Campaign, cfg.Session.SessionID)
|
||||
if ensureErr != nil {
|
||||
t.Fatalf("EnsureLayout() error = %v", ensureErr)
|
||||
}
|
||||
|
||||
@@ -63,7 +63,7 @@ func TestExecuteStagesDefaultWiringUsesWhisperXHTTPClient(t *testing.T) {
|
||||
t.Fatal("audio file payload was empty")
|
||||
}
|
||||
|
||||
outPath := filepath.Join(cfg.Pipeline.Workspace.Root, "work", cfg.Session.SessionID, "transcripts", "raw", "alice.json")
|
||||
outPath := filepath.Join(cfg.Pipeline.Workspace.Root, "work", cfg.Session.Campaign, cfg.Session.SessionID, "transcripts", "raw", "alice.json")
|
||||
data, err := os.ReadFile(outPath)
|
||||
if err != nil {
|
||||
t.Fatalf("ReadFile(%q) error = %v", outPath, err)
|
||||
|
||||
294
internal/artifacts/artifact_resolver.go
Normal file
294
internal/artifacts/artifact_resolver.go
Normal file
@@ -0,0 +1,294 @@
|
||||
package artifacts
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
|
||||
"gitea.maximumdirect.net/eric/narratio/internal/manifest"
|
||||
)
|
||||
|
||||
const (
|
||||
ArtifactTranscriptMerged = "narratio.transcript.merged"
|
||||
ArtifactTranscriptPolished = "narratio.transcript.polished"
|
||||
ArtifactTranscriptFull = "narratio.transcript.full"
|
||||
ArtifactTranscriptTrimmed = "narratio.transcript.trimmed"
|
||||
ArtifactBoundsSession = "narratio.bounds.session"
|
||||
ArtifactSessionRecap = "narratio.artifact.session_recap"
|
||||
)
|
||||
|
||||
// ErrSessionArtifactNotFound is returned when no readable artifact exists for a known ID.
|
||||
var ErrSessionArtifactNotFound = errors.New("session artifact not found")
|
||||
|
||||
type artifactContentKind string
|
||||
|
||||
const (
|
||||
contentTranscriptJSON artifactContentKind = "transcript_json"
|
||||
contentJSON artifactContentKind = "json"
|
||||
contentText artifactContentKind = "text"
|
||||
)
|
||||
|
||||
type artifactSpec struct {
|
||||
ID string
|
||||
CanonicalRelPath string
|
||||
ProducerStage string
|
||||
OutputKind string
|
||||
ContentKind artifactContentKind
|
||||
}
|
||||
|
||||
var artifactRegistry = map[string]artifactSpec{
|
||||
ArtifactTranscriptMerged: {
|
||||
ID: ArtifactTranscriptMerged,
|
||||
CanonicalRelPath: "transcripts/merged.json",
|
||||
ProducerStage: "merge",
|
||||
OutputKind: "transcript_merged",
|
||||
ContentKind: contentTranscriptJSON,
|
||||
},
|
||||
ArtifactTranscriptPolished: {
|
||||
ID: ArtifactTranscriptPolished,
|
||||
CanonicalRelPath: "transcripts/processed.json",
|
||||
ProducerStage: "polish",
|
||||
OutputKind: "transcript_processed",
|
||||
ContentKind: contentTranscriptJSON,
|
||||
},
|
||||
ArtifactTranscriptFull: {
|
||||
ID: ArtifactTranscriptFull,
|
||||
CanonicalRelPath: "transcripts/normalized.json",
|
||||
ProducerStage: "normalize",
|
||||
OutputKind: "transcript_normalized",
|
||||
ContentKind: contentTranscriptJSON,
|
||||
},
|
||||
ArtifactTranscriptTrimmed: {
|
||||
ID: ArtifactTranscriptTrimmed,
|
||||
CanonicalRelPath: "transcripts/trimmed.json",
|
||||
ProducerStage: "trim",
|
||||
OutputKind: "transcript_trimmed",
|
||||
ContentKind: contentTranscriptJSON,
|
||||
},
|
||||
ArtifactBoundsSession: {
|
||||
ID: ArtifactBoundsSession,
|
||||
CanonicalRelPath: "artifacts/session_bounds.json",
|
||||
ProducerStage: "trim",
|
||||
OutputKind: "session_bounds",
|
||||
ContentKind: contentJSON,
|
||||
},
|
||||
ArtifactSessionRecap: {
|
||||
ID: ArtifactSessionRecap,
|
||||
CanonicalRelPath: "artifacts/session_recap.md",
|
||||
ProducerStage: "analyze",
|
||||
OutputKind: "session_recap",
|
||||
ContentKind: contentText,
|
||||
},
|
||||
}
|
||||
|
||||
var artifactAliases = map[string]string{
|
||||
"processed_transcript": ArtifactTranscriptPolished,
|
||||
"normalized_transcript": ArtifactTranscriptFull,
|
||||
"trimmed_transcript": ArtifactTranscriptTrimmed,
|
||||
}
|
||||
|
||||
// ResolvedSessionArtifact describes one session-level artifact lookup result.
|
||||
type ResolvedSessionArtifact struct {
|
||||
ID string
|
||||
Path string
|
||||
ProducerStage string
|
||||
OutputKind string
|
||||
ProducerRunID string
|
||||
Provenance string
|
||||
}
|
||||
|
||||
// SessionArtifactNotFoundError includes context when a known artifact cannot be read.
|
||||
type SessionArtifactNotFoundError struct {
|
||||
ArtifactID string
|
||||
}
|
||||
|
||||
func (e *SessionArtifactNotFoundError) Error() string {
|
||||
return fmt.Sprintf("%s: %q", ErrSessionArtifactNotFound, e.ArtifactID)
|
||||
}
|
||||
|
||||
func (e *SessionArtifactNotFoundError) Unwrap() error {
|
||||
return ErrSessionArtifactNotFound
|
||||
}
|
||||
|
||||
// NormalizeSessionArtifactSource maps legacy aliases to canonical IDs and validates IDs.
|
||||
func NormalizeSessionArtifactSource(source string) (string, error) {
|
||||
normalized := strings.TrimSpace(source)
|
||||
if normalized == "" {
|
||||
return "", fmt.Errorf("artifact source is required")
|
||||
}
|
||||
if alias, ok := artifactAliases[normalized]; ok {
|
||||
normalized = alias
|
||||
}
|
||||
if _, ok := artifactRegistry[normalized]; !ok {
|
||||
return "", fmt.Errorf("unsupported artifact source %q", source)
|
||||
}
|
||||
return normalized, nil
|
||||
}
|
||||
|
||||
// ResolveSessionArtifact resolves a symbolic source to a readable local session artifact path.
|
||||
// Resolution order is manifest producer outputs first, then canonical session path fallback.
|
||||
func ResolveSessionArtifact(paths SessionPaths, m *manifest.Manifest, source string) (ResolvedSessionArtifact, error) {
|
||||
id, err := NormalizeSessionArtifactSource(source)
|
||||
if err != nil {
|
||||
return ResolvedSessionArtifact{}, err
|
||||
}
|
||||
spec := artifactRegistry[id]
|
||||
|
||||
for _, candidate := range manifestArtifactCandidates(paths, m, spec) {
|
||||
exists, isDir, statErr := pathExists(candidate.Path)
|
||||
if statErr != nil {
|
||||
return ResolvedSessionArtifact{}, fmt.Errorf("stat %q: %w", candidate.Path, statErr)
|
||||
}
|
||||
if !exists || isDir {
|
||||
continue
|
||||
}
|
||||
resolved := candidate
|
||||
resolved.ID = spec.ID
|
||||
resolved.ProducerStage = spec.ProducerStage
|
||||
resolved.OutputKind = spec.OutputKind
|
||||
if err := validateResolvedContent(resolved.Path, spec.ContentKind); err != nil {
|
||||
return ResolvedSessionArtifact{}, fmt.Errorf("validate %q: %w", resolved.ID, err)
|
||||
}
|
||||
return resolved, nil
|
||||
}
|
||||
|
||||
fallbackPath := filepath.Join(paths.Root, filepath.FromSlash(spec.CanonicalRelPath))
|
||||
exists, isDir, statErr := pathExists(fallbackPath)
|
||||
if statErr != nil {
|
||||
return ResolvedSessionArtifact{}, fmt.Errorf("stat %q: %w", fallbackPath, statErr)
|
||||
}
|
||||
if exists && !isDir {
|
||||
if err := validateResolvedContent(fallbackPath, spec.ContentKind); err != nil {
|
||||
return ResolvedSessionArtifact{}, fmt.Errorf("validate %q: %w", spec.ID, err)
|
||||
}
|
||||
return ResolvedSessionArtifact{
|
||||
ID: spec.ID,
|
||||
Path: filepath.Clean(fallbackPath),
|
||||
ProducerStage: spec.ProducerStage,
|
||||
OutputKind: spec.OutputKind,
|
||||
Provenance: "fallback.canonical_path",
|
||||
}, nil
|
||||
}
|
||||
|
||||
return ResolvedSessionArtifact{}, &SessionArtifactNotFoundError{ArtifactID: spec.ID}
|
||||
}
|
||||
|
||||
func manifestArtifactCandidates(paths SessionPaths, m *manifest.Manifest, spec artifactSpec) []ResolvedSessionArtifact {
|
||||
if m == nil || len(m.Stages) == 0 || spec.ProducerStage == "" || spec.OutputKind == "" {
|
||||
return nil
|
||||
}
|
||||
sr := m.Stages[spec.ProducerStage]
|
||||
if sr == nil {
|
||||
return nil
|
||||
}
|
||||
|
||||
candidates := make([]ResolvedSessionArtifact, 0, len(sr.Outputs))
|
||||
for _, out := range sr.Outputs {
|
||||
if strings.TrimSpace(out.Kind) != spec.OutputKind {
|
||||
continue
|
||||
}
|
||||
if strings.TrimSpace(out.LocalPath) == "" {
|
||||
continue
|
||||
}
|
||||
resolved := filepath.Clean(ResolveSessionLocalPathForRead(paths, out.LocalPath))
|
||||
if resolved == "" {
|
||||
continue
|
||||
}
|
||||
candidates = append(candidates, ResolvedSessionArtifact{
|
||||
Path: resolved,
|
||||
ProducerRunID: strings.TrimSpace(out.ProducerRunID),
|
||||
Provenance: "manifest." + spec.ProducerStage + ".outputs",
|
||||
})
|
||||
}
|
||||
return dedupeResolvedArtifacts(candidates)
|
||||
}
|
||||
|
||||
func dedupeResolvedArtifacts(values []ResolvedSessionArtifact) []ResolvedSessionArtifact {
|
||||
seen := map[string]struct{}{}
|
||||
out := make([]ResolvedSessionArtifact, 0, len(values))
|
||||
for _, value := range values {
|
||||
key := filepath.Clean(strings.TrimSpace(value.Path))
|
||||
if key == "" {
|
||||
continue
|
||||
}
|
||||
if _, ok := seen[key]; ok {
|
||||
continue
|
||||
}
|
||||
seen[key] = struct{}{}
|
||||
value.Path = key
|
||||
out = append(out, value)
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
func pathExists(path string) (exists bool, isDir bool, err error) {
|
||||
info, err := os.Stat(path)
|
||||
if err == nil {
|
||||
return true, info.IsDir(), nil
|
||||
}
|
||||
if errors.Is(err, os.ErrNotExist) {
|
||||
return false, false, nil
|
||||
}
|
||||
return false, false, err
|
||||
}
|
||||
|
||||
func validateResolvedContent(path string, kind artifactContentKind) error {
|
||||
switch kind {
|
||||
case contentTranscriptJSON:
|
||||
return validateTranscriptSegmentsJSON(path)
|
||||
case contentJSON:
|
||||
return validateJSONContent(path)
|
||||
case contentText:
|
||||
return validateNonEmptyContent(path)
|
||||
default:
|
||||
return fmt.Errorf("unsupported content kind %q", kind)
|
||||
}
|
||||
}
|
||||
|
||||
func validateTranscriptSegmentsJSON(path string) error {
|
||||
data, err := os.ReadFile(path)
|
||||
if err != nil {
|
||||
return fmt.Errorf("read file: %w", err)
|
||||
}
|
||||
var payload map[string]any
|
||||
if err := json.Unmarshal(data, &payload); err != nil {
|
||||
return fmt.Errorf("decode json: %w", err)
|
||||
}
|
||||
segments, ok := payload["segments"]
|
||||
if !ok {
|
||||
return fmt.Errorf("top-level segments is required")
|
||||
}
|
||||
if _, ok := segments.([]any); !ok {
|
||||
return fmt.Errorf("top-level segments must be an array")
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func validateJSONContent(path string) error {
|
||||
data, err := os.ReadFile(path)
|
||||
if err != nil {
|
||||
return fmt.Errorf("read file: %w", err)
|
||||
}
|
||||
var payload any
|
||||
if err := json.Unmarshal(data, &payload); err != nil {
|
||||
return fmt.Errorf("decode json: %w", err)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func validateNonEmptyContent(path string) error {
|
||||
info, err := os.Stat(path)
|
||||
if err != nil {
|
||||
return fmt.Errorf("stat file: %w", err)
|
||||
}
|
||||
if info.IsDir() {
|
||||
return fmt.Errorf("path is a directory")
|
||||
}
|
||||
if info.Size() <= 0 {
|
||||
return fmt.Errorf("file is empty")
|
||||
}
|
||||
return nil
|
||||
}
|
||||
139
internal/artifacts/artifact_resolver_test.go
Normal file
139
internal/artifacts/artifact_resolver_test.go
Normal file
@@ -0,0 +1,139 @@
|
||||
package artifacts
|
||||
|
||||
import (
|
||||
"errors"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"gitea.maximumdirect.net/eric/narratio/internal/manifest"
|
||||
)
|
||||
|
||||
func TestNormalizeSessionArtifactSource(t *testing.T) {
|
||||
tests := []struct {
|
||||
name string
|
||||
source string
|
||||
wantID string
|
||||
wantErr string
|
||||
}{
|
||||
{name: "legacy alias processed", source: "processed_transcript", wantID: ArtifactTranscriptPolished},
|
||||
{name: "legacy alias normalized", source: "normalized_transcript", wantID: ArtifactTranscriptFull},
|
||||
{name: "legacy alias trimmed", source: "trimmed_transcript", wantID: ArtifactTranscriptTrimmed},
|
||||
{name: "canonical", source: ArtifactTranscriptTrimmed, wantID: ArtifactTranscriptTrimmed},
|
||||
{name: "unsupported", source: "narratio.unknown", wantErr: "unsupported artifact source"},
|
||||
}
|
||||
|
||||
for _, tt := range tests {
|
||||
t.Run(tt.name, func(t *testing.T) {
|
||||
got, err := NormalizeSessionArtifactSource(tt.source)
|
||||
if tt.wantErr != "" {
|
||||
if err == nil || !strings.Contains(err.Error(), tt.wantErr) {
|
||||
t.Fatalf("NormalizeSessionArtifactSource() error = %v, want contains %q", err, tt.wantErr)
|
||||
}
|
||||
return
|
||||
}
|
||||
if err != nil {
|
||||
t.Fatalf("NormalizeSessionArtifactSource() error = %v", err)
|
||||
}
|
||||
if got != tt.wantID {
|
||||
t.Fatalf("NormalizeSessionArtifactSource() = %q, want %q", got, tt.wantID)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestResolveSessionArtifactPrefersManifestOutput(t *testing.T) {
|
||||
workspace := t.TempDir()
|
||||
paths := buildSessionPaths(workspace, "campaign", "session")
|
||||
if err := os.MkdirAll(paths.ArtifactsDir, 0o755); err != nil {
|
||||
t.Fatalf("MkdirAll() error = %v", err)
|
||||
}
|
||||
manifestPath := filepath.Join(paths.ArtifactsDir, "normalized.from-manifest.json")
|
||||
if err := os.WriteFile(manifestPath, []byte(`{"segments":[]}`), 0o644); err != nil {
|
||||
t.Fatalf("WriteFile() error = %v", err)
|
||||
}
|
||||
canonicalPath := filepath.Join(paths.TranscriptsDir, "normalized.json")
|
||||
if err := os.MkdirAll(filepath.Dir(canonicalPath), 0o755); err != nil {
|
||||
t.Fatalf("MkdirAll() error = %v", err)
|
||||
}
|
||||
if err := os.WriteFile(canonicalPath, []byte(`{"segments":[{"id":123}]}`), 0o644); err != nil {
|
||||
t.Fatalf("WriteFile() error = %v", err)
|
||||
}
|
||||
|
||||
m := manifest.New("session", time.Now().UTC())
|
||||
m.MarkStageSucceeded("normalize", time.Now().UTC(), []manifest.ArtifactRecord{
|
||||
{Kind: "transcript_normalized", LocalPath: manifestPath, ProducerRunID: "run-123"},
|
||||
})
|
||||
|
||||
resolved, err := ResolveSessionArtifact(paths, m, "normalized_transcript")
|
||||
if err != nil {
|
||||
t.Fatalf("ResolveSessionArtifact() error = %v", err)
|
||||
}
|
||||
if resolved.Path != manifestPath {
|
||||
t.Fatalf("resolved path = %q, want %q", resolved.Path, manifestPath)
|
||||
}
|
||||
if resolved.Provenance != "manifest.normalize.outputs" {
|
||||
t.Fatalf("provenance = %q, want manifest.normalize.outputs", resolved.Provenance)
|
||||
}
|
||||
if resolved.ProducerRunID != "run-123" {
|
||||
t.Fatalf("producer run id = %q, want run-123", resolved.ProducerRunID)
|
||||
}
|
||||
}
|
||||
|
||||
func TestResolveSessionArtifactFallsBackToCanonicalPath(t *testing.T) {
|
||||
workspace := t.TempDir()
|
||||
paths := buildSessionPaths(workspace, "campaign", "session")
|
||||
canonicalPath := filepath.Join(paths.TranscriptsDir, "trimmed.json")
|
||||
if err := os.MkdirAll(filepath.Dir(canonicalPath), 0o755); err != nil {
|
||||
t.Fatalf("MkdirAll() error = %v", err)
|
||||
}
|
||||
if err := os.WriteFile(canonicalPath, []byte(`{"segments":[]}`), 0o644); err != nil {
|
||||
t.Fatalf("WriteFile() error = %v", err)
|
||||
}
|
||||
|
||||
resolved, err := ResolveSessionArtifact(paths, nil, ArtifactTranscriptTrimmed)
|
||||
if err != nil {
|
||||
t.Fatalf("ResolveSessionArtifact() error = %v", err)
|
||||
}
|
||||
if resolved.Path != canonicalPath {
|
||||
t.Fatalf("resolved path = %q, want %q", resolved.Path, canonicalPath)
|
||||
}
|
||||
if resolved.Provenance != "fallback.canonical_path" {
|
||||
t.Fatalf("provenance = %q, want fallback.canonical_path", resolved.Provenance)
|
||||
}
|
||||
}
|
||||
|
||||
func TestResolveSessionArtifactMissingReturnsTypedError(t *testing.T) {
|
||||
workspace := t.TempDir()
|
||||
paths := buildSessionPaths(workspace, "campaign", "session")
|
||||
|
||||
_, err := ResolveSessionArtifact(paths, nil, ArtifactTranscriptTrimmed)
|
||||
if err == nil {
|
||||
t.Fatal("expected error, got nil")
|
||||
}
|
||||
if !errors.Is(err, ErrSessionArtifactNotFound) {
|
||||
t.Fatalf("errors.Is(err, ErrSessionArtifactNotFound) = false; err=%v", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestResolveSessionArtifactValidatesTranscriptShape(t *testing.T) {
|
||||
workspace := t.TempDir()
|
||||
paths := buildSessionPaths(workspace, "campaign", "session")
|
||||
canonicalPath := filepath.Join(paths.TranscriptsDir, "processed.json")
|
||||
if err := os.MkdirAll(filepath.Dir(canonicalPath), 0o755); err != nil {
|
||||
t.Fatalf("MkdirAll() error = %v", err)
|
||||
}
|
||||
if err := os.WriteFile(canonicalPath, []byte(`{"not_segments":[]}`), 0o644); err != nil {
|
||||
t.Fatalf("WriteFile() error = %v", err)
|
||||
}
|
||||
|
||||
_, err := ResolveSessionArtifact(paths, nil, ArtifactTranscriptPolished)
|
||||
if err == nil {
|
||||
t.Fatal("expected error, got nil")
|
||||
}
|
||||
if !strings.Contains(err.Error(), "top-level segments is required") {
|
||||
t.Fatalf("error = %q, want segments validation error", err.Error())
|
||||
}
|
||||
}
|
||||
@@ -30,13 +30,13 @@ func NewLocalStore(workspaceRoot string) *LocalStore {
|
||||
return &LocalStore{WorkspaceRoot: workspaceRoot}
|
||||
}
|
||||
|
||||
// SessionPaths resolves canonical paths for a session workdir.
|
||||
func (s *LocalStore) SessionPaths(sessionID string) SessionPaths {
|
||||
return buildSessionPaths(s.WorkspaceRoot, sessionID)
|
||||
// SessionPathsFor resolves canonical campaign-aware paths for a session workdir.
|
||||
func (s *LocalStore) SessionPathsFor(campaign, sessionID string) SessionPaths {
|
||||
return buildSessionPaths(s.WorkspaceRoot, campaign, sessionID)
|
||||
}
|
||||
|
||||
// EnsureLayout creates and verifies the canonical session workdir directory layout.
|
||||
func (s *LocalStore) EnsureLayout(sessionID string) (SessionPaths, error) {
|
||||
// EnsureLayoutFor creates and verifies campaign-aware session layout.
|
||||
func (s *LocalStore) EnsureLayoutFor(campaign, sessionID string) (SessionPaths, error) {
|
||||
if strings.TrimSpace(s.WorkspaceRoot) == "" {
|
||||
return SessionPaths{}, fmt.Errorf("workspace root is required")
|
||||
}
|
||||
@@ -44,7 +44,22 @@ func (s *LocalStore) EnsureLayout(sessionID string) (SessionPaths, error) {
|
||||
return SessionPaths{}, fmt.Errorf("sessionID is required")
|
||||
}
|
||||
|
||||
paths := s.SessionPaths(sessionID)
|
||||
campaign = strings.TrimSpace(campaign)
|
||||
if campaign == "" {
|
||||
return SessionPaths{}, fmt.Errorf("campaign is required")
|
||||
}
|
||||
|
||||
return s.ensureLayout(s.SessionPathsFor(campaign, sessionID))
|
||||
}
|
||||
|
||||
func (s *LocalStore) ensureLayout(paths SessionPaths) (SessionPaths, error) {
|
||||
if strings.TrimSpace(s.WorkspaceRoot) == "" {
|
||||
return SessionPaths{}, fmt.Errorf("workspace root is required")
|
||||
}
|
||||
if strings.TrimSpace(paths.SessionID) == "" {
|
||||
return SessionPaths{}, fmt.Errorf("sessionID is required")
|
||||
}
|
||||
|
||||
dirs := []string{
|
||||
paths.Root,
|
||||
paths.InputsDir,
|
||||
@@ -53,8 +68,11 @@ func (s *LocalStore) EnsureLayout(sessionID string) (SessionPaths, error) {
|
||||
paths.TranscriptsRawDir,
|
||||
paths.TranscriptsTrimmedDir,
|
||||
paths.ArtifactsDir,
|
||||
paths.ReportsDir,
|
||||
paths.ConfigDir,
|
||||
paths.LogsDir,
|
||||
paths.CurrentDir,
|
||||
paths.RunsDir,
|
||||
}
|
||||
|
||||
for _, dir := range dirs {
|
||||
@@ -66,13 +84,16 @@ func (s *LocalStore) EnsureLayout(sessionID string) (SessionPaths, error) {
|
||||
return paths, nil
|
||||
}
|
||||
|
||||
// CopyInput copies an input file into the session workdir under destRelativePath.
|
||||
func (s *LocalStore) CopyInput(sessionID, srcPath, destRelativePath string) (Ref, error) {
|
||||
paths, err := s.EnsureLayout(sessionID)
|
||||
// CopyInputFor copies an input file into the campaign-aware session workdir under destRelativePath.
|
||||
func (s *LocalStore) CopyInputFor(campaign, sessionID, srcPath, destRelativePath string) (Ref, error) {
|
||||
paths, err := s.EnsureLayoutFor(campaign, sessionID)
|
||||
if err != nil {
|
||||
return Ref{}, err
|
||||
}
|
||||
return s.copyInputWithPaths(paths, sessionID, srcPath, destRelativePath)
|
||||
}
|
||||
|
||||
func (s *LocalStore) copyInputWithPaths(paths SessionPaths, sessionID, srcPath, destRelativePath string) (Ref, error) {
|
||||
destAbs, err := resolveInRoot(paths.Root, destRelativePath)
|
||||
if err != nil {
|
||||
return Ref{}, fmt.Errorf("copy input: %w", err)
|
||||
@@ -173,13 +194,16 @@ func (s *LocalStore) Checksum(path string) (string, error) {
|
||||
return digest, nil
|
||||
}
|
||||
|
||||
// AcquireSessionLock acquires an exclusive lock file for a session workdir.
|
||||
func (s *LocalStore) AcquireSessionLock(sessionID string) (*LockHandle, error) {
|
||||
paths, err := s.EnsureLayout(sessionID)
|
||||
// AcquireSessionLockFor acquires an exclusive lock file for a campaign/session workdir.
|
||||
func (s *LocalStore) AcquireSessionLockFor(campaign, sessionID string) (*LockHandle, error) {
|
||||
paths, err := s.EnsureLayoutFor(campaign, sessionID)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return s.acquireSessionLockForPaths(paths)
|
||||
}
|
||||
|
||||
func (s *LocalStore) acquireSessionLockForPaths(paths SessionPaths) (*LockHandle, error) {
|
||||
f, err := os.OpenFile(paths.LockPath, os.O_CREATE|os.O_EXCL|os.O_WRONLY, 0o644)
|
||||
if err != nil {
|
||||
if errors.Is(err, os.ErrExist) {
|
||||
|
||||
@@ -10,9 +10,9 @@ import (
|
||||
|
||||
func TestEnsureLayoutCreatesExpectedDirectories(t *testing.T) {
|
||||
store := NewLocalStore(t.TempDir())
|
||||
paths, err := store.EnsureLayout("session-1")
|
||||
paths, err := store.EnsureLayoutFor("sample-campaign", "session-1")
|
||||
if err != nil {
|
||||
t.Fatalf("EnsureLayout() error = %v", err)
|
||||
t.Fatalf("EnsureLayoutFor() error = %v", err)
|
||||
}
|
||||
|
||||
checkDirExists(t, paths.Root)
|
||||
@@ -22,8 +22,11 @@ func TestEnsureLayoutCreatesExpectedDirectories(t *testing.T) {
|
||||
checkDirExists(t, paths.TranscriptsRawDir)
|
||||
checkDirExists(t, paths.TranscriptsTrimmedDir)
|
||||
checkDirExists(t, paths.ArtifactsDir)
|
||||
checkDirExists(t, paths.ReportsDir)
|
||||
checkDirExists(t, paths.ConfigDir)
|
||||
checkDirExists(t, paths.LogsDir)
|
||||
checkDirExists(t, paths.CurrentDir)
|
||||
checkDirExists(t, paths.RunsDir)
|
||||
|
||||
if filepath.Base(paths.ManifestPath) != "manifest.json" {
|
||||
t.Fatalf("ManifestPath = %q, want basename manifest.json", paths.ManifestPath)
|
||||
@@ -33,6 +36,17 @@ func TestEnsureLayoutCreatesExpectedDirectories(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
func TestEnsureLayoutForRequiresCampaign(t *testing.T) {
|
||||
store := NewLocalStore(t.TempDir())
|
||||
_, err := store.EnsureLayoutFor("", "session-1")
|
||||
if err == nil {
|
||||
t.Fatal("expected campaign-required error, got nil")
|
||||
}
|
||||
if !strings.Contains(err.Error(), "campaign is required") {
|
||||
t.Fatalf("error = %v, want campaign-required error", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestChecksumCalculation(t *testing.T) {
|
||||
store := NewLocalStore(t.TempDir())
|
||||
path := filepath.Join(t.TempDir(), "sample.txt")
|
||||
@@ -53,9 +67,9 @@ func TestChecksumCalculation(t *testing.T) {
|
||||
func TestLockAcquireRelease(t *testing.T) {
|
||||
store := NewLocalStore(t.TempDir())
|
||||
|
||||
lock, err := store.AcquireSessionLock("session-1")
|
||||
lock, err := store.AcquireSessionLockFor("sample-campaign", "session-1")
|
||||
if err != nil {
|
||||
t.Fatalf("AcquireSessionLock() error = %v", err)
|
||||
t.Fatalf("AcquireSessionLockFor() error = %v", err)
|
||||
}
|
||||
|
||||
exists, err := store.Exists(lock.path)
|
||||
@@ -81,15 +95,15 @@ func TestLockAcquireRelease(t *testing.T) {
|
||||
|
||||
func TestLockConflict(t *testing.T) {
|
||||
store := NewLocalStore(t.TempDir())
|
||||
lock1, err := store.AcquireSessionLock("session-1")
|
||||
lock1, err := store.AcquireSessionLockFor("sample-campaign", "session-1")
|
||||
if err != nil {
|
||||
t.Fatalf("first AcquireSessionLock() error = %v", err)
|
||||
t.Fatalf("first AcquireSessionLockFor() error = %v", err)
|
||||
}
|
||||
defer func() {
|
||||
_ = store.ReleaseSessionLock(lock1)
|
||||
}()
|
||||
|
||||
_, err = store.AcquireSessionLock("session-1")
|
||||
_, err = store.AcquireSessionLockFor("sample-campaign", "session-1")
|
||||
if err == nil {
|
||||
t.Fatal("expected lock conflict error, got nil")
|
||||
}
|
||||
@@ -138,9 +152,9 @@ func TestCopyInput(t *testing.T) {
|
||||
t.Fatalf("WriteFile() error = %v", err)
|
||||
}
|
||||
|
||||
ref, err := store.CopyInput("session-1", srcPath, "inputs/speakers.yml")
|
||||
ref, err := store.CopyInputFor("sample-campaign", "session-1", srcPath, "inputs/speakers.yml")
|
||||
if err != nil {
|
||||
t.Fatalf("CopyInput() error = %v", err)
|
||||
t.Fatalf("CopyInputFor() error = %v", err)
|
||||
}
|
||||
|
||||
if ref.Kind != "input" {
|
||||
|
||||
@@ -2,11 +2,15 @@ package artifacts
|
||||
|
||||
import (
|
||||
"path/filepath"
|
||||
|
||||
"gitea.maximumdirect.net/eric/narratio/internal/config"
|
||||
)
|
||||
|
||||
// SessionPaths contains canonical local paths for one session work directory.
|
||||
type SessionPaths struct {
|
||||
WorkspaceRoot string
|
||||
CampaignID string
|
||||
SessionID string
|
||||
Root string
|
||||
InputsDir string
|
||||
AudioDir string
|
||||
@@ -14,42 +18,73 @@ type SessionPaths struct {
|
||||
TranscriptsRawDir string
|
||||
TranscriptsTrimmedDir string
|
||||
ArtifactsDir string
|
||||
ReportsDir string
|
||||
ConfigDir string
|
||||
LogsDir string
|
||||
CurrentDir string
|
||||
RunsDir string
|
||||
ManifestPath string
|
||||
LockPath string
|
||||
}
|
||||
|
||||
// SessionWorkDir returns the work directory for one session.
|
||||
func SessionWorkDir(rootDir, sessionID string) string {
|
||||
return filepath.Join(rootDir, "work", sessionID)
|
||||
// SessionWorkDirForCampaign returns the canonical campaign-aware work directory for one session.
|
||||
func SessionWorkDirForCampaign(rootDir, campaign, sessionID string) string {
|
||||
return filepath.Join(rootDir, config.PathWorkDirSegment, campaign, sessionID)
|
||||
}
|
||||
|
||||
// SessionRunWorkDir returns the campaign/session/run scoped local work directory.
|
||||
func SessionRunWorkDir(rootDir, campaign, sessionID, runID string) string {
|
||||
return filepath.Join(rootDir, "work", campaign, sessionID, runID)
|
||||
// SessionManifestPathForCampaign returns the canonical session manifest path.
|
||||
func SessionManifestPathForCampaign(rootDir, campaign, sessionID string) string {
|
||||
return filepath.Join(SessionWorkDirForCampaign(rootDir, campaign, sessionID), config.PathManifestFile)
|
||||
}
|
||||
|
||||
// SessionRunsDirForCampaign returns the canonical runs directory for one session.
|
||||
func SessionRunsDirForCampaign(rootDir, campaign, sessionID string) string {
|
||||
return filepath.Join(SessionWorkDirForCampaign(rootDir, campaign, sessionID), config.PathRunsDirSegment)
|
||||
}
|
||||
|
||||
// SessionRunRootForCampaign returns the canonical run root under runs/{run_id}.
|
||||
func SessionRunRootForCampaign(rootDir, campaign, sessionID, runID string) string {
|
||||
return filepath.Join(SessionRunsDirForCampaign(rootDir, campaign, sessionID), runID)
|
||||
}
|
||||
|
||||
// SessionRunManifestPathForCampaign returns the canonical run manifest path under runs/{run_id}/manifest.json.
|
||||
func SessionRunManifestPathForCampaign(rootDir, campaign, sessionID, runID string) string {
|
||||
return filepath.Join(SessionRunRootForCampaign(rootDir, campaign, sessionID, runID), config.PathManifestFile)
|
||||
}
|
||||
|
||||
// SessionRunStageDirForCampaign returns the canonical stage directory under runs/{run_id}/{stage}.
|
||||
func SessionRunStageDirForCampaign(rootDir, campaign, sessionID, runID, stageName string) string {
|
||||
return filepath.Join(SessionRunRootForCampaign(rootDir, campaign, sessionID, runID), stageName)
|
||||
}
|
||||
|
||||
// SessionSpoolAudioDir returns the campaign/session/run scoped local spool audio path.
|
||||
func SessionSpoolAudioDir(spoolRoot, campaign, sessionID, runID string) string {
|
||||
return filepath.Join(spoolRoot, campaign, sessionID, runID, "audio")
|
||||
return filepath.Join(spoolRoot, campaign, sessionID, runID, config.PathAudioDirSegment)
|
||||
}
|
||||
|
||||
func buildSessionPaths(workspaceRoot, sessionID string) SessionPaths {
|
||||
root := SessionWorkDir(workspaceRoot, sessionID)
|
||||
transcripts := filepath.Join(root, "transcripts")
|
||||
func buildSessionPaths(workspaceRoot, campaign, sessionID string) SessionPaths {
|
||||
root := SessionWorkDirForCampaign(workspaceRoot, campaign, sessionID)
|
||||
return buildSessionPathsFromRoot(workspaceRoot, campaign, sessionID, root)
|
||||
}
|
||||
|
||||
func buildSessionPathsFromRoot(workspaceRoot, campaign, sessionID, root string) SessionPaths {
|
||||
return SessionPaths{
|
||||
WorkspaceRoot: workspaceRoot,
|
||||
CampaignID: campaign,
|
||||
SessionID: sessionID,
|
||||
Root: root,
|
||||
InputsDir: filepath.Join(root, "inputs"),
|
||||
AudioDir: filepath.Join(root, "audio"),
|
||||
TranscriptsDir: transcripts,
|
||||
TranscriptsRawDir: filepath.Join(transcripts, "raw"),
|
||||
TranscriptsTrimmedDir: filepath.Join(transcripts, "trimmed"),
|
||||
ArtifactsDir: filepath.Join(root, "artifacts"),
|
||||
ConfigDir: filepath.Join(root, "config"),
|
||||
LogsDir: filepath.Join(root, "logs"),
|
||||
ManifestPath: filepath.Join(root, "manifest.json"),
|
||||
LockPath: filepath.Join(root, ".lock"),
|
||||
InputsDir: filepath.Join(root, config.PathInputsDirSegment),
|
||||
AudioDir: filepath.Join(root, config.PathAudioDirSegment),
|
||||
TranscriptsDir: filepath.Join(root, config.PathTranscriptsSegment),
|
||||
TranscriptsRawDir: filepath.Join(root, filepath.FromSlash(config.PathTranscriptsRaw)),
|
||||
TranscriptsTrimmedDir: filepath.Join(root, filepath.FromSlash(config.PathTranscriptsTrimmed)),
|
||||
ArtifactsDir: filepath.Join(root, config.PathArtifactsDirSegment),
|
||||
ReportsDir: filepath.Join(root, config.PathReportsDirSegment),
|
||||
ConfigDir: filepath.Join(root, config.PathConfigDirSegment),
|
||||
LogsDir: filepath.Join(root, config.PathLogsDirSegment),
|
||||
CurrentDir: filepath.Join(root, config.PathCurrentDirSegment),
|
||||
RunsDir: filepath.Join(root, config.PathRunsDirSegment),
|
||||
ManifestPath: filepath.Join(root, config.PathManifestFile),
|
||||
LockPath: filepath.Join(root, config.PathLockFile),
|
||||
}
|
||||
}
|
||||
|
||||
@@ -5,12 +5,47 @@ import (
|
||||
"testing"
|
||||
)
|
||||
|
||||
func TestSessionRunWorkDir(t *testing.T) {
|
||||
func TestSessionWorkDirForCampaign(t *testing.T) {
|
||||
root := "/tmp/workspace"
|
||||
got := SessionRunWorkDir(root, "forsaken", "2026-04-19", "20260515T031522Z-a1b2c3d4")
|
||||
want := filepath.Join(root, "work", "forsaken", "2026-04-19", "20260515T031522Z-a1b2c3d4")
|
||||
got := SessionWorkDirForCampaign(root, "forsaken", "2026-04-19")
|
||||
want := filepath.Join(root, "work", "forsaken", "2026-04-19")
|
||||
if got != want {
|
||||
t.Fatalf("SessionRunWorkDir() = %q, want %q", got, want)
|
||||
t.Fatalf("SessionWorkDirForCampaign() = %q, want %q", got, want)
|
||||
}
|
||||
}
|
||||
|
||||
func TestSessionManifestPathForCampaign(t *testing.T) {
|
||||
root := "/tmp/workspace"
|
||||
got := SessionManifestPathForCampaign(root, "forsaken", "2026-04-19")
|
||||
want := filepath.Join(root, "work", "forsaken", "2026-04-19", "manifest.json")
|
||||
if got != want {
|
||||
t.Fatalf("SessionManifestPathForCampaign() = %q, want %q", got, want)
|
||||
}
|
||||
}
|
||||
|
||||
func TestSessionRunRootAndStageDirForCampaign(t *testing.T) {
|
||||
root := "/tmp/workspace"
|
||||
runID := "20260515T031522Z-a1b2c3d4"
|
||||
runRoot := SessionRunRootForCampaign(root, "forsaken", "2026-04-19", runID)
|
||||
wantRoot := filepath.Join(root, "work", "forsaken", "2026-04-19", "runs", runID)
|
||||
if runRoot != wantRoot {
|
||||
t.Fatalf("SessionRunRootForCampaign() = %q, want %q", runRoot, wantRoot)
|
||||
}
|
||||
|
||||
stageDir := SessionRunStageDirForCampaign(root, "forsaken", "2026-04-19", runID, "transcribe")
|
||||
wantStage := filepath.Join(root, "work", "forsaken", "2026-04-19", "runs", runID, "transcribe")
|
||||
if stageDir != wantStage {
|
||||
t.Fatalf("SessionRunStageDirForCampaign() = %q, want %q", stageDir, wantStage)
|
||||
}
|
||||
}
|
||||
|
||||
func TestSessionRunManifestPathForCampaign(t *testing.T) {
|
||||
root := "/tmp/workspace"
|
||||
runID := "20260515T031522Z-a1b2c3d4"
|
||||
got := SessionRunManifestPathForCampaign(root, "forsaken", "2026-04-19", runID)
|
||||
want := filepath.Join(root, "work", "forsaken", "2026-04-19", "runs", runID, "manifest.json")
|
||||
if got != want {
|
||||
t.Fatalf("SessionRunManifestPathForCampaign() = %q, want %q", got, want)
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
@@ -8,7 +8,7 @@ import (
|
||||
|
||||
func TestResolveSessionLocalPathForRead(t *testing.T) {
|
||||
workspace := t.TempDir()
|
||||
paths := buildSessionPaths(workspace, "s-1")
|
||||
paths := buildSessionPaths(workspace, "sample-campaign", "s-1")
|
||||
if err := os.MkdirAll(paths.TranscriptsRawDir, 0o755); err != nil {
|
||||
t.Fatalf("MkdirAll() error = %v", err)
|
||||
}
|
||||
@@ -41,7 +41,7 @@ func TestResolveSessionLocalPathForReadRelativeWorkspaceRootQualifiedPath(t *tes
|
||||
t.Fatalf("Rel() error = %v", err)
|
||||
}
|
||||
|
||||
paths := buildSessionPaths(workspaceRel, "s-1")
|
||||
paths := buildSessionPaths(workspaceRel, "sample-campaign", "s-1")
|
||||
target := filepath.Join(paths.TranscriptsRawDir, "alice.json")
|
||||
if err := os.MkdirAll(filepath.Dir(target), 0o755); err != nil {
|
||||
t.Fatalf("MkdirAll() error = %v", err)
|
||||
@@ -50,7 +50,7 @@ func TestResolveSessionLocalPathForReadRelativeWorkspaceRootQualifiedPath(t *tes
|
||||
t.Fatalf("WriteFile() error = %v", err)
|
||||
}
|
||||
|
||||
manifestPath := filepath.Join(workspaceRel, "work", "s-1", "transcripts", "raw", "alice.json")
|
||||
manifestPath := filepath.Join(workspaceRel, "work", "sample-campaign", "s-1", "transcripts", "raw", "alice.json")
|
||||
got := ResolveSessionLocalPathForRead(paths, manifestPath)
|
||||
if got != filepath.Clean(manifestPath) {
|
||||
t.Fatalf("resolution = %q, want %q", got, filepath.Clean(manifestPath))
|
||||
|
||||
@@ -11,19 +11,19 @@ type S3Store struct {
|
||||
Prefix string
|
||||
}
|
||||
|
||||
// SessionPaths is not implemented for S3-backed storage.
|
||||
func (s *S3Store) SessionPaths(_ string) SessionPaths {
|
||||
// SessionPathsFor is not implemented for S3-backed storage.
|
||||
func (s *S3Store) SessionPathsFor(_, _ string) SessionPaths {
|
||||
return SessionPaths{}
|
||||
}
|
||||
|
||||
// EnsureLayout returns a not-yet-implemented error in the scaffold.
|
||||
func (s *S3Store) EnsureLayout(_ string) (SessionPaths, error) {
|
||||
return SessionPaths{}, fmt.Errorf("artifacts s3 ensure layout: not yet implemented")
|
||||
// EnsureLayoutFor returns a not-yet-implemented error in the scaffold.
|
||||
func (s *S3Store) EnsureLayoutFor(_, _ string) (SessionPaths, error) {
|
||||
return SessionPaths{}, fmt.Errorf("artifacts s3 ensure layout for campaign/session: not yet implemented")
|
||||
}
|
||||
|
||||
// CopyInput returns a not-yet-implemented error in the scaffold.
|
||||
func (s *S3Store) CopyInput(_, _, _ string) (Ref, error) {
|
||||
return Ref{}, fmt.Errorf("artifacts s3 copy input: not yet implemented")
|
||||
// CopyInputFor returns a not-yet-implemented error in the scaffold.
|
||||
func (s *S3Store) CopyInputFor(_, _, _, _ string) (Ref, error) {
|
||||
return Ref{}, fmt.Errorf("artifacts s3 copy input for campaign/session: not yet implemented")
|
||||
}
|
||||
|
||||
// Exists returns a not-yet-implemented error in the scaffold.
|
||||
@@ -46,9 +46,9 @@ func (s *S3Store) Checksum(_ string) (string, error) {
|
||||
return "", fmt.Errorf("artifacts s3 checksum: not yet implemented")
|
||||
}
|
||||
|
||||
// AcquireSessionLock returns a not-yet-implemented error in the scaffold.
|
||||
func (s *S3Store) AcquireSessionLock(_ string) (*LockHandle, error) {
|
||||
return nil, fmt.Errorf("artifacts s3 acquire lock: not yet implemented")
|
||||
// AcquireSessionLockFor returns a not-yet-implemented error in the scaffold.
|
||||
func (s *S3Store) AcquireSessionLockFor(_, _ string) (*LockHandle, error) {
|
||||
return nil, fmt.Errorf("artifacts s3 acquire lock for campaign/session: not yet implemented")
|
||||
}
|
||||
|
||||
// ReleaseSessionLock returns a not-yet-implemented error in the scaffold.
|
||||
|
||||
@@ -3,6 +3,8 @@ package artifacts
|
||||
import (
|
||||
"path"
|
||||
"strings"
|
||||
|
||||
"gitea.maximumdirect.net/eric/narratio/internal/config"
|
||||
)
|
||||
|
||||
// S3SessionPrefix builds the canonical S3 session prefix.
|
||||
@@ -10,9 +12,9 @@ import (
|
||||
func S3SessionPrefix(rootPrefix, campaign, sessionID string) string {
|
||||
prefix := path.Join(
|
||||
cleanS3PathPart(rootPrefix),
|
||||
"campaigns",
|
||||
config.S3CampaignsSegment,
|
||||
cleanS3PathPart(campaign),
|
||||
"sessions",
|
||||
config.S3SessionsSegment,
|
||||
cleanS3PathPart(sessionID),
|
||||
)
|
||||
return ensureS3TrailingSlash(prefix)
|
||||
@@ -21,7 +23,7 @@ func S3SessionPrefix(rootPrefix, campaign, sessionID string) string {
|
||||
// S3RunPrefix builds the canonical S3 run prefix.
|
||||
// Format: {session_prefix}/runs/{run_id}/
|
||||
func S3RunPrefix(sessionPrefix, runID string) string {
|
||||
prefix := path.Join(strings.TrimSuffix(cleanS3Key(sessionPrefix), "/"), "runs", cleanS3PathPart(runID))
|
||||
prefix := path.Join(strings.TrimSuffix(cleanS3Key(sessionPrefix), "/"), config.S3RunsSegment, cleanS3PathPart(runID))
|
||||
return ensureS3TrailingSlash(prefix)
|
||||
}
|
||||
|
||||
@@ -35,13 +37,13 @@ func S3AudioPrefix(sessionPrefix, audioPrefix string) string {
|
||||
// S3CurrentManifestKey returns the current manifest pointer key.
|
||||
// Format: {session_prefix}/current/manifest.json
|
||||
func S3CurrentManifestKey(sessionPrefix string) string {
|
||||
return path.Join(strings.TrimSuffix(cleanS3Key(sessionPrefix), "/"), "current", "manifest.json")
|
||||
return path.Join(strings.TrimSuffix(cleanS3Key(sessionPrefix), "/"), config.S3CurrentSegment, config.S3ManifestFile)
|
||||
}
|
||||
|
||||
// S3CurrentRunPointerKey returns the current run pointer key.
|
||||
// Format: {session_prefix}/current/run_id.txt
|
||||
func S3CurrentRunPointerKey(sessionPrefix string) string {
|
||||
return path.Join(strings.TrimSuffix(cleanS3Key(sessionPrefix), "/"), "current", "run_id.txt")
|
||||
return path.Join(strings.TrimSuffix(cleanS3Key(sessionPrefix), "/"), config.S3CurrentSegment, config.S3RunIDFile)
|
||||
}
|
||||
|
||||
// S3PromotedArtifactKey returns the destination key for one promoted artifact.
|
||||
|
||||
@@ -15,13 +15,13 @@ type Ref struct {
|
||||
|
||||
// Store is the local artifact/workdir abstraction used by orchestration code.
|
||||
type Store interface {
|
||||
SessionPaths(sessionID string) SessionPaths
|
||||
EnsureLayout(sessionID string) (SessionPaths, error)
|
||||
CopyInput(sessionID, srcPath, destRelativePath string) (Ref, error)
|
||||
SessionPathsFor(campaign, sessionID string) SessionPaths
|
||||
EnsureLayoutFor(campaign, sessionID string) (SessionPaths, error)
|
||||
CopyInputFor(campaign, sessionID, srcPath, destRelativePath string) (Ref, error)
|
||||
Exists(path string) (bool, error)
|
||||
ExistsRef(ref Ref) (bool, error)
|
||||
WriteFileAtomic(path string, data []byte, perm os.FileMode) error
|
||||
Checksum(path string) (string, error)
|
||||
AcquireSessionLock(sessionID string) (*LockHandle, error)
|
||||
AcquireSessionLockFor(campaign, sessionID string) (*LockHandle, error)
|
||||
ReleaseSessionLock(lock *LockHandle) error
|
||||
}
|
||||
|
||||
@@ -10,8 +10,72 @@ const (
|
||||
DefaultSessionConfigPathEtc = "/etc/narratio/session.yml"
|
||||
DefaultS3AccessKeyIDEnv = "OBJECT_STORAGE_KEY_ID"
|
||||
DefaultS3SecretAccessKeyEnv = "OBJECT_STORAGE_KEY"
|
||||
DefaultStorageS3RootPrefix = "dnd"
|
||||
DefaultSpoolRoot = "/var/spool/narratio"
|
||||
|
||||
DefaultWhisperXLanguage = "en"
|
||||
DefaultWhisperXTimeout = "30m"
|
||||
DefaultWhisperXRetryDelay = "2s"
|
||||
DefaultWhisperXConcurrency = 2
|
||||
DefaultWhisperXRetries = 3
|
||||
|
||||
DefaultSeriatimBinary = "seriatim"
|
||||
DefaultSeriatimTimeout = "10m"
|
||||
DefaultSeriatimOutputSchema = "seriatim-intermediate"
|
||||
DefaultSeriatimCoalesceGap = 3.0
|
||||
DefaultSeriatimReport = true
|
||||
|
||||
DefaultAuditaBinary = "audita"
|
||||
DefaultAuditaTimeout = "3h"
|
||||
DefaultAuditaReport = true
|
||||
|
||||
DefaultScriptoriumBinary = "scriptorium"
|
||||
DefaultScriptoriumTimeout = "10m"
|
||||
|
||||
DefaultTrimBoundsTimeout = "10m"
|
||||
DefaultTrimSeriatimReport = false
|
||||
|
||||
DefaultNormalizeOutputPath = "transcripts/normalized.json"
|
||||
DefaultNormalizeOutputSchema = "seriatim-intermediate"
|
||||
DefaultNormalizeReport = true
|
||||
|
||||
DefaultArchiveEnabled = true
|
||||
DefaultArchiveUploadRun = true
|
||||
|
||||
PathWorkDirSegment = "work"
|
||||
PathInputsDirSegment = "inputs"
|
||||
PathAudioDirSegment = "audio"
|
||||
PathTranscriptsSegment = "transcripts"
|
||||
PathTranscriptsRaw = "transcripts/raw"
|
||||
PathTranscriptsTrimmed = "transcripts/trimmed"
|
||||
PathArtifactsDirSegment = "artifacts"
|
||||
PathReportsDirSegment = "reports"
|
||||
PathConfigDirSegment = "config"
|
||||
PathLogsDirSegment = "logs"
|
||||
PathCurrentDirSegment = "current"
|
||||
PathRunsDirSegment = "runs"
|
||||
PathManifestFile = "manifest.json"
|
||||
PathLockFile = ".lock"
|
||||
PathTranscriptMerged = "transcripts/merged.json"
|
||||
PathTranscriptProcessed = "transcripts/processed.json"
|
||||
PathTranscriptNormalized = "transcripts/normalized.json"
|
||||
PathTranscriptTrimmed = "transcripts/trimmed.json"
|
||||
|
||||
S3CampaignsSegment = "campaigns"
|
||||
S3SessionsSegment = "sessions"
|
||||
S3RunsSegment = "runs"
|
||||
S3CurrentSegment = "current"
|
||||
S3ManifestFile = "manifest.json"
|
||||
S3RunIDFile = "run_id.txt"
|
||||
)
|
||||
|
||||
// DefaultArchivePromoteArtifacts defines the default archive promotion rules.
|
||||
// Callers should copy this slice before mutating.
|
||||
var DefaultArchivePromoteArtifacts = []ArchivePromotionRule{
|
||||
{From: PathTranscriptTrimmed, To: PathTranscriptTrimmed},
|
||||
{From: "artifacts/session_recap.md", To: "artifacts/session_recap.md"},
|
||||
}
|
||||
|
||||
// DefaultPipelineConfigSearchPaths defines the default search order for
|
||||
// pipeline.yml when callers do not provide an explicit path.
|
||||
//
|
||||
|
||||
@@ -174,7 +174,7 @@ func applyStorageDefaults(cfg *StorageConfig) {
|
||||
cfg.S3 = &StorageS3Config{}
|
||||
}
|
||||
if cfg.S3.RootPrefix == "" {
|
||||
cfg.S3.RootPrefix = "dnd"
|
||||
cfg.S3.RootPrefix = DefaultStorageS3RootPrefix
|
||||
}
|
||||
if cfg.S3.AccessKeyIDEnv == "" {
|
||||
cfg.S3.AccessKeyIDEnv = DefaultS3AccessKeyIDEnv
|
||||
@@ -189,7 +189,7 @@ func applySpoolDefaults(cfg *SpoolConfig) {
|
||||
return
|
||||
}
|
||||
if cfg.Root == "" {
|
||||
cfg.Root = "/var/spool/narratio"
|
||||
cfg.Root = DefaultSpoolRoot
|
||||
}
|
||||
}
|
||||
|
||||
@@ -202,16 +202,13 @@ func applyArchiveDefaults(cfg **ArchiveConfig) {
|
||||
}
|
||||
|
||||
if (*cfg).Enabled == nil {
|
||||
(*cfg).Enabled = boolPtr(true)
|
||||
(*cfg).Enabled = boolPtr(DefaultArchiveEnabled)
|
||||
}
|
||||
if (*cfg).UploadRun == nil {
|
||||
(*cfg).UploadRun = boolPtr(true)
|
||||
(*cfg).UploadRun = boolPtr(DefaultArchiveUploadRun)
|
||||
}
|
||||
if len((*cfg).PromoteArtifacts) == 0 {
|
||||
(*cfg).PromoteArtifacts = []ArchivePromotionRule{
|
||||
{From: "transcripts/trimmed.json", To: "transcripts/trimmed.json", Required: boolPtr(true)},
|
||||
{From: "artifacts/session_recap.md", To: "artifacts/session_recap.md", Required: boolPtr(true)},
|
||||
}
|
||||
(*cfg).PromoteArtifacts = append([]ArchivePromotionRule(nil), DefaultArchivePromoteArtifacts...)
|
||||
}
|
||||
for i := range (*cfg).PromoteArtifacts {
|
||||
if (*cfg).PromoteArtifacts[i].Required == nil {
|
||||
@@ -225,19 +222,19 @@ func applyWhisperXDefaults(cfg *WhisperXConfig) {
|
||||
return
|
||||
}
|
||||
if cfg.Language == "" {
|
||||
cfg.Language = "en"
|
||||
cfg.Language = DefaultWhisperXLanguage
|
||||
}
|
||||
if cfg.Timeout == "" {
|
||||
cfg.Timeout = "30m"
|
||||
cfg.Timeout = DefaultWhisperXTimeout
|
||||
}
|
||||
if cfg.RetryDelay == "" {
|
||||
cfg.RetryDelay = "2s"
|
||||
cfg.RetryDelay = DefaultWhisperXRetryDelay
|
||||
}
|
||||
if cfg.Concurrency == nil {
|
||||
cfg.Concurrency = intPtr(2)
|
||||
cfg.Concurrency = intPtr(DefaultWhisperXConcurrency)
|
||||
}
|
||||
if cfg.Retries == nil {
|
||||
cfg.Retries = intPtr(3)
|
||||
cfg.Retries = intPtr(DefaultWhisperXRetries)
|
||||
}
|
||||
}
|
||||
|
||||
@@ -251,19 +248,19 @@ func applySeriatimDefaults(cfg *SeriatimConfig) {
|
||||
return
|
||||
}
|
||||
if cfg.Binary == "" {
|
||||
cfg.Binary = "seriatim"
|
||||
cfg.Binary = DefaultSeriatimBinary
|
||||
}
|
||||
if cfg.Timeout == "" {
|
||||
cfg.Timeout = "10m"
|
||||
cfg.Timeout = DefaultSeriatimTimeout
|
||||
}
|
||||
if cfg.OutputSchema == "" {
|
||||
cfg.OutputSchema = "seriatim-intermediate"
|
||||
cfg.OutputSchema = DefaultSeriatimOutputSchema
|
||||
}
|
||||
if cfg.CoalesceGap == nil {
|
||||
cfg.CoalesceGap = float64Ptr(3.0)
|
||||
cfg.CoalesceGap = float64Ptr(DefaultSeriatimCoalesceGap)
|
||||
}
|
||||
if cfg.Report == nil {
|
||||
cfg.Report = boolPtr(true)
|
||||
cfg.Report = boolPtr(DefaultSeriatimReport)
|
||||
}
|
||||
}
|
||||
|
||||
@@ -272,13 +269,13 @@ func applyAuditaDefaults(cfg *AuditaConfig) {
|
||||
return
|
||||
}
|
||||
if cfg.Binary == "" {
|
||||
cfg.Binary = "audita"
|
||||
cfg.Binary = DefaultAuditaBinary
|
||||
}
|
||||
if cfg.Timeout == "" {
|
||||
cfg.Timeout = "3h"
|
||||
cfg.Timeout = DefaultAuditaTimeout
|
||||
}
|
||||
if cfg.Report == nil {
|
||||
cfg.Report = boolPtr(true)
|
||||
cfg.Report = boolPtr(DefaultAuditaReport)
|
||||
}
|
||||
}
|
||||
|
||||
@@ -287,10 +284,10 @@ func applyScriptoriumDefaults(cfg *ScriptoriumConfig) {
|
||||
return
|
||||
}
|
||||
if cfg.Binary == "" {
|
||||
cfg.Binary = "scriptorium"
|
||||
cfg.Binary = DefaultScriptoriumBinary
|
||||
}
|
||||
if cfg.Timeout == "" {
|
||||
cfg.Timeout = "10m"
|
||||
cfg.Timeout = DefaultScriptoriumTimeout
|
||||
}
|
||||
}
|
||||
|
||||
@@ -299,10 +296,10 @@ func applyTrimDefaults(cfg *TrimConfig) {
|
||||
return
|
||||
}
|
||||
if cfg.Bounds.Timeout == "" {
|
||||
cfg.Bounds.Timeout = "10m"
|
||||
cfg.Bounds.Timeout = DefaultTrimBoundsTimeout
|
||||
}
|
||||
if cfg.Seriatim.Report == nil {
|
||||
cfg.Seriatim.Report = boolPtr(false)
|
||||
cfg.Seriatim.Report = boolPtr(DefaultTrimSeriatimReport)
|
||||
}
|
||||
}
|
||||
|
||||
@@ -311,13 +308,13 @@ func applyNormalizeDefaults(cfg *NormalizeConfig) {
|
||||
return
|
||||
}
|
||||
if cfg.OutputSchema == "" {
|
||||
cfg.OutputSchema = defaultNormalizeOutputSchema
|
||||
cfg.OutputSchema = DefaultNormalizeOutputSchema
|
||||
}
|
||||
if cfg.OutputPath == "" && !cfg.outputPathWasSet() {
|
||||
cfg.OutputPath = defaultNormalizeOutputPath
|
||||
cfg.OutputPath = DefaultNormalizeOutputPath
|
||||
}
|
||||
if cfg.Report == nil {
|
||||
cfg.Report = boolPtr(true)
|
||||
cfg.Report = boolPtr(DefaultNormalizeReport)
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
@@ -6,11 +6,6 @@ import (
|
||||
"gopkg.in/yaml.v3"
|
||||
)
|
||||
|
||||
const (
|
||||
defaultNormalizeOutputPath = "transcripts/normalized.json"
|
||||
defaultNormalizeOutputSchema = "seriatim-intermediate"
|
||||
)
|
||||
|
||||
// UnmarshalYAML tracks explicit normalize.output_path presence so validation can
|
||||
// distinguish omitted vs explicitly empty values.
|
||||
func (cfg *NormalizeConfig) UnmarshalYAML(node *yaml.Node) error {
|
||||
|
||||
@@ -116,6 +116,37 @@ func TestScriptoriumLoadAndValidate(t *testing.T) {
|
||||
output_kind: session_recap
|
||||
`,
|
||||
},
|
||||
{
|
||||
name: "canonical artifact source is accepted",
|
||||
scriptoriumYAML: `scriptorium:
|
||||
binary: scriptorium
|
||||
artifacts:
|
||||
session_recap:
|
||||
enabled: true
|
||||
prompt_id: dnd.session_recap
|
||||
output_path: artifacts/session_recap.md
|
||||
inputs:
|
||||
transcript:
|
||||
source: narratio.transcript.trimmed
|
||||
required: true
|
||||
`,
|
||||
},
|
||||
{
|
||||
name: "unknown artifact source fails validation",
|
||||
scriptoriumYAML: `scriptorium:
|
||||
binary: scriptorium
|
||||
artifacts:
|
||||
session_recap:
|
||||
enabled: true
|
||||
prompt_id: dnd.session_recap
|
||||
output_path: artifacts/session_recap.md
|
||||
inputs:
|
||||
transcript:
|
||||
source: narratio.unknown
|
||||
required: true
|
||||
`,
|
||||
wantValidateErr: `pipeline.scriptorium.artifacts.session_recap.inputs.transcript.source "narratio.unknown" is unsupported`,
|
||||
},
|
||||
{
|
||||
name: "artifact render_debug override is accepted",
|
||||
scriptoriumYAML: `scriptorium:
|
||||
|
||||
@@ -343,9 +343,13 @@ func validateScriptorium(cfg *ScriptoriumConfig) error {
|
||||
if trimmedInputName == "" {
|
||||
return fmt.Errorf("pipeline.scriptorium.artifacts.%s.inputs keys must be non-empty", artifactName)
|
||||
}
|
||||
if strings.TrimSpace(inputCfg.Source) == "" {
|
||||
source := strings.TrimSpace(inputCfg.Source)
|
||||
if source == "" {
|
||||
return fmt.Errorf("pipeline.scriptorium.artifacts.%s.inputs.%s.source is required", artifactName, inputName)
|
||||
}
|
||||
if !isSupportedScriptoriumInputSource(source) {
|
||||
return fmt.Errorf("pipeline.scriptorium.artifacts.%s.inputs.%s.source %q is unsupported", artifactName, inputName, inputCfg.Source)
|
||||
}
|
||||
}
|
||||
for varName, varValue := range artifactCfg.Vars {
|
||||
if strings.TrimSpace(varName) == "" {
|
||||
@@ -435,6 +439,33 @@ func archiveUploadConfiguredForS3(pipeline *PipelineConfig) bool {
|
||||
return enabled && upload
|
||||
}
|
||||
|
||||
func isSupportedScriptoriumInputSource(source string) bool {
|
||||
switch strings.TrimSpace(source) {
|
||||
case "previous_session_artifact":
|
||||
return true
|
||||
case "processed_transcript":
|
||||
return true
|
||||
case "normalized_transcript":
|
||||
return true
|
||||
case "trimmed_transcript":
|
||||
return true
|
||||
case "narratio.transcript.merged":
|
||||
return true
|
||||
case "narratio.transcript.polished":
|
||||
return true
|
||||
case "narratio.transcript.full":
|
||||
return true
|
||||
case "narratio.transcript.trimmed":
|
||||
return true
|
||||
case "narratio.bounds.session":
|
||||
return true
|
||||
case "narratio.artifact.session_recap":
|
||||
return true
|
||||
default:
|
||||
return false
|
||||
}
|
||||
}
|
||||
|
||||
var windowsAbsPathRE = regexp.MustCompile(`^[A-Za-z]:[\\/].*`)
|
||||
var envVarNameRE = regexp.MustCompile(`^[A-Za-z_][A-Za-z0-9_]*$`)
|
||||
|
||||
|
||||
@@ -29,8 +29,10 @@ type InputRecord struct {
|
||||
type ArtifactRecord struct {
|
||||
Kind string `json:"kind"`
|
||||
LocalPath string `json:"local_path"`
|
||||
RemoteKey string `json:"remote_key,omitempty"`
|
||||
Checksum string `json:"checksum,omitempty"`
|
||||
// ProducerRunID identifies the run that produced this durable artifact.
|
||||
ProducerRunID string `json:"producer_run_id,omitempty"`
|
||||
RemoteKey string `json:"remote_key,omitempty"`
|
||||
Checksum string `json:"checksum,omitempty"`
|
||||
}
|
||||
|
||||
// StageRecord tracks lifecycle and provenance for one pipeline stage.
|
||||
@@ -120,6 +122,15 @@ func (m *Manifest) MarkStageSkipped(name string, at time.Time, reason string) {
|
||||
m.UpdatedAt = at
|
||||
}
|
||||
|
||||
// MarkStageStale marks a stage as stale so it is not skipped as idempotently complete.
|
||||
func (m *Manifest) MarkStageStale(name string, at time.Time, reason string) {
|
||||
s := m.ensureStage(name, at)
|
||||
s.Status = StatusStale
|
||||
s.Error = &ErrorRecord{Message: strings.TrimSpace(reason), Code: "stale", At: timePtr(at)}
|
||||
s.UpdatedAt = at
|
||||
m.UpdatedAt = at
|
||||
}
|
||||
|
||||
func (m *Manifest) ensureStage(name string, at time.Time) *StageRecord {
|
||||
if m.Stages == nil {
|
||||
m.Stages = map[string]*StageRecord{}
|
||||
|
||||
164
internal/manifest/run_manifest.go
Normal file
164
internal/manifest/run_manifest.go
Normal file
@@ -0,0 +1,164 @@
|
||||
package manifest
|
||||
|
||||
import (
|
||||
"strings"
|
||||
"time"
|
||||
)
|
||||
|
||||
type RunManifestStatus string
|
||||
|
||||
const (
|
||||
RunManifestStatusRunning RunManifestStatus = "running"
|
||||
RunManifestStatusSucceeded RunManifestStatus = "succeeded"
|
||||
RunManifestStatusFailed RunManifestStatus = "failed"
|
||||
)
|
||||
|
||||
type RunStageAction string
|
||||
|
||||
const (
|
||||
RunStageActionRun RunStageAction = "run"
|
||||
RunStageActionSkip RunStageAction = "skip"
|
||||
)
|
||||
|
||||
// RunStageRecord tracks lifecycle and provenance for one stage within a single invocation.
|
||||
type RunStageRecord struct {
|
||||
Name string `json:"name"`
|
||||
Action RunStageAction `json:"action"`
|
||||
Status StageStatus `json:"status"`
|
||||
CreatedAt time.Time `json:"created_at"`
|
||||
UpdatedAt time.Time `json:"updated_at"`
|
||||
StartedAt *time.Time `json:"started_at,omitempty"`
|
||||
CompletedAt *time.Time `json:"completed_at,omitempty"`
|
||||
Outputs []ArtifactRecord `json:"outputs,omitempty"`
|
||||
Logs []string `json:"logs,omitempty"`
|
||||
GeneratedConfigs []string `json:"generated_configs,omitempty"`
|
||||
Error *ErrorRecord `json:"error,omitempty"`
|
||||
Metadata map[string]any `json:"metadata,omitempty"`
|
||||
}
|
||||
|
||||
// RunManifest is the invocation-scoped execution record under runs/{run_id}/manifest.json.
|
||||
type RunManifest struct {
|
||||
SessionID string `json:"session_id"`
|
||||
Campaign string `json:"campaign,omitempty"`
|
||||
RunID string `json:"run_id"`
|
||||
Force bool `json:"force"`
|
||||
RequestedStages []string `json:"requested_stages,omitempty"`
|
||||
SessionManifestPath string `json:"session_manifest_path,omitempty"`
|
||||
LocalWorkDir string `json:"local_workdir,omitempty"`
|
||||
LocalSpoolDir string `json:"local_spool_dir,omitempty"`
|
||||
S3Bucket string `json:"s3_bucket,omitempty"`
|
||||
S3SessionPrefix string `json:"s3_session_prefix,omitempty"`
|
||||
S3RunPrefix string `json:"s3_run_prefix,omitempty"`
|
||||
CreatedAt time.Time `json:"created_at"`
|
||||
UpdatedAt time.Time `json:"updated_at"`
|
||||
StartedAt *time.Time `json:"started_at,omitempty"`
|
||||
CompletedAt *time.Time `json:"completed_at,omitempty"`
|
||||
Status RunManifestStatus `json:"status"`
|
||||
LastError *ErrorRecord `json:"last_error,omitempty"`
|
||||
Stages map[string]*RunStageRecord `json:"stages"`
|
||||
Metadata map[string]any `json:"metadata,omitempty"`
|
||||
}
|
||||
|
||||
// NewRun constructs a new run manifest with deterministic timestamps.
|
||||
func NewRun(sessionID, campaign, runID string, force bool, requestedStages []string, now time.Time) *RunManifest {
|
||||
return &RunManifest{
|
||||
SessionID: strings.TrimSpace(sessionID),
|
||||
Campaign: strings.TrimSpace(campaign),
|
||||
RunID: strings.TrimSpace(runID),
|
||||
Force: force,
|
||||
RequestedStages: append([]string(nil), requestedStages...),
|
||||
CreatedAt: now,
|
||||
UpdatedAt: now,
|
||||
StartedAt: timePtr(now),
|
||||
Status: RunManifestStatusRunning,
|
||||
Stages: map[string]*RunStageRecord{},
|
||||
}
|
||||
}
|
||||
|
||||
func (m *RunManifest) SetStageAction(name string, action RunStageAction, at time.Time) {
|
||||
s := m.ensureStage(name, at)
|
||||
s.Action = action
|
||||
s.UpdatedAt = at
|
||||
m.UpdatedAt = at
|
||||
}
|
||||
|
||||
func (m *RunManifest) MarkStageRunning(name string, at time.Time) {
|
||||
s := m.ensureStage(name, at)
|
||||
s.Status = StatusRunning
|
||||
s.StartedAt = timePtr(at)
|
||||
s.CompletedAt = nil
|
||||
s.Error = nil
|
||||
s.UpdatedAt = at
|
||||
m.UpdatedAt = at
|
||||
}
|
||||
|
||||
func (m *RunManifest) MarkStageSucceeded(name string, at time.Time, outputs []ArtifactRecord) {
|
||||
s := m.ensureStage(name, at)
|
||||
s.Status = StatusSucceeded
|
||||
s.CompletedAt = timePtr(at)
|
||||
s.Error = nil
|
||||
s.Outputs = append([]ArtifactRecord(nil), outputs...)
|
||||
s.UpdatedAt = at
|
||||
m.UpdatedAt = at
|
||||
}
|
||||
|
||||
func (m *RunManifest) MarkStageFailed(name string, at time.Time, message string) {
|
||||
s := m.ensureStage(name, at)
|
||||
s.Status = StatusFailed
|
||||
s.CompletedAt = timePtr(at)
|
||||
s.Error = &ErrorRecord{Message: strings.TrimSpace(message), At: timePtr(at)}
|
||||
s.UpdatedAt = at
|
||||
m.LastError = &ErrorRecord{Message: strings.TrimSpace(message), At: timePtr(at)}
|
||||
m.UpdatedAt = at
|
||||
m.Status = RunManifestStatusFailed
|
||||
m.CompletedAt = timePtr(at)
|
||||
}
|
||||
|
||||
func (m *RunManifest) MarkStageSkipped(name string, at time.Time, reason string) {
|
||||
s := m.ensureStage(name, at)
|
||||
s.Status = StatusSkipped
|
||||
s.CompletedAt = timePtr(at)
|
||||
s.Error = &ErrorRecord{Message: strings.TrimSpace(reason), Code: "skipped", At: timePtr(at)}
|
||||
s.UpdatedAt = at
|
||||
m.UpdatedAt = at
|
||||
}
|
||||
|
||||
func (m *RunManifest) MarkSucceeded(at time.Time) {
|
||||
m.Status = RunManifestStatusSucceeded
|
||||
m.CompletedAt = timePtr(at)
|
||||
m.UpdatedAt = at
|
||||
}
|
||||
|
||||
func (m *RunManifest) MarkFailed(at time.Time, message string) {
|
||||
m.Status = RunManifestStatusFailed
|
||||
m.CompletedAt = timePtr(at)
|
||||
m.LastError = &ErrorRecord{Message: strings.TrimSpace(message), At: timePtr(at)}
|
||||
m.UpdatedAt = at
|
||||
}
|
||||
|
||||
func (m *RunManifest) ensureStage(name string, at time.Time) *RunStageRecord {
|
||||
if m.Stages == nil {
|
||||
m.Stages = map[string]*RunStageRecord{}
|
||||
}
|
||||
|
||||
stageName := strings.TrimSpace(name)
|
||||
s, ok := m.Stages[stageName]
|
||||
if !ok || s == nil {
|
||||
s = &RunStageRecord{
|
||||
Name: stageName,
|
||||
Action: RunStageActionRun,
|
||||
Status: StatusPending,
|
||||
CreatedAt: at,
|
||||
UpdatedAt: at,
|
||||
}
|
||||
m.Stages[stageName] = s
|
||||
}
|
||||
if s.Name == "" {
|
||||
s.Name = stageName
|
||||
}
|
||||
if s.CreatedAt.IsZero() {
|
||||
s.CreatedAt = at
|
||||
}
|
||||
|
||||
return s
|
||||
}
|
||||
50
internal/manifest/run_manifest_test.go
Normal file
50
internal/manifest/run_manifest_test.go
Normal file
@@ -0,0 +1,50 @@
|
||||
package manifest
|
||||
|
||||
import (
|
||||
"testing"
|
||||
"time"
|
||||
)
|
||||
|
||||
func TestRunManifestStageMarkHelpers(t *testing.T) {
|
||||
rm := NewRun("2026-05-03", "forsaken", "20260517T000000Z-abcdef12", false, []string{"prepare"}, time.Date(2026, 5, 3, 10, 0, 0, 0, time.UTC))
|
||||
if rm.Status != RunManifestStatusRunning {
|
||||
t.Fatalf("status = %q, want %q", rm.Status, RunManifestStatusRunning)
|
||||
}
|
||||
|
||||
runningAt := time.Date(2026, 5, 3, 10, 1, 0, 0, time.UTC)
|
||||
rm.SetStageAction("prepare", RunStageActionRun, runningAt)
|
||||
rm.MarkStageRunning("prepare", runningAt)
|
||||
rm.MarkStageSucceeded("prepare", runningAt.Add(30*time.Second), []ArtifactRecord{
|
||||
{Kind: "input", LocalPath: "inputs/session.yml"},
|
||||
})
|
||||
|
||||
stage := rm.Stages["prepare"]
|
||||
if stage == nil {
|
||||
t.Fatal("prepare stage missing")
|
||||
}
|
||||
if stage.Action != RunStageActionRun {
|
||||
t.Fatalf("action = %q, want %q", stage.Action, RunStageActionRun)
|
||||
}
|
||||
if stage.Status != StatusSucceeded {
|
||||
t.Fatalf("status = %q, want %q", stage.Status, StatusSucceeded)
|
||||
}
|
||||
|
||||
skippedAt := runningAt.Add(1 * time.Minute)
|
||||
rm.SetStageAction("notify", RunStageActionSkip, skippedAt)
|
||||
rm.MarkStageSkipped("notify", skippedAt, "already_succeeded")
|
||||
skipped := rm.Stages["notify"]
|
||||
if skipped == nil {
|
||||
t.Fatal("notify stage missing")
|
||||
}
|
||||
if skipped.Action != RunStageActionSkip {
|
||||
t.Fatalf("action = %q, want %q", skipped.Action, RunStageActionSkip)
|
||||
}
|
||||
if skipped.Status != StatusSkipped {
|
||||
t.Fatalf("status = %q, want %q", skipped.Status, StatusSkipped)
|
||||
}
|
||||
|
||||
rm.MarkSucceeded(skippedAt.Add(10 * time.Second))
|
||||
if rm.Status != RunManifestStatusSucceeded {
|
||||
t.Fatalf("status = %q, want %q", rm.Status, RunManifestStatusSucceeded)
|
||||
}
|
||||
}
|
||||
@@ -129,6 +129,89 @@ func (s *LocalStore) Save(ctx context.Context, path string, m *Manifest) error {
|
||||
return nil
|
||||
}
|
||||
|
||||
// CreateRun returns a new in-memory run manifest for one invocation.
|
||||
func (s *LocalStore) CreateRun(
|
||||
ctx context.Context,
|
||||
sessionID, campaign, runID string,
|
||||
force bool,
|
||||
requestedStages []string,
|
||||
) (*RunManifest, error) {
|
||||
if err := checkContext(ctx); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if strings.TrimSpace(sessionID) == "" {
|
||||
return nil, fmt.Errorf("create run manifest: session_id is required")
|
||||
}
|
||||
if strings.TrimSpace(runID) == "" {
|
||||
return nil, fmt.Errorf("create run manifest: run_id is required")
|
||||
}
|
||||
|
||||
now := time.Now().UTC()
|
||||
return NewRun(sessionID, campaign, runID, force, requestedStages, now), nil
|
||||
}
|
||||
|
||||
// LoadRun reads and validates a local JSON run manifest from path.
|
||||
func (s *LocalStore) LoadRun(ctx context.Context, path string) (*RunManifest, error) {
|
||||
if err := checkContext(ctx); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if strings.TrimSpace(path) == "" {
|
||||
return nil, fmt.Errorf("load run manifest: path is required")
|
||||
}
|
||||
|
||||
data, err := os.ReadFile(path)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("load run manifest %q: %w", path, err)
|
||||
}
|
||||
|
||||
var m RunManifest
|
||||
if err := json.Unmarshal(data, &m); err != nil {
|
||||
return nil, fmt.Errorf("decode run manifest %q: %w", path, err)
|
||||
}
|
||||
|
||||
if err := validateLoadedRunManifest(&m); err != nil {
|
||||
return nil, fmt.Errorf("run manifest %q invalid: %w", path, err)
|
||||
}
|
||||
normalizeRunManifest(&m)
|
||||
|
||||
return &m, nil
|
||||
}
|
||||
|
||||
// SaveRun writes the run manifest to path atomically via temp file + rename.
|
||||
func (s *LocalStore) SaveRun(ctx context.Context, path string, m *RunManifest) error {
|
||||
if err := checkContext(ctx); err != nil {
|
||||
return err
|
||||
}
|
||||
if strings.TrimSpace(path) == "" {
|
||||
return fmt.Errorf("save run manifest: path is required")
|
||||
}
|
||||
if m == nil {
|
||||
return fmt.Errorf("save run manifest: manifest is nil")
|
||||
}
|
||||
if strings.TrimSpace(m.SessionID) == "" {
|
||||
return fmt.Errorf("save run manifest: session_id is required")
|
||||
}
|
||||
if strings.TrimSpace(m.RunID) == "" {
|
||||
return fmt.Errorf("save run manifest: run_id is required")
|
||||
}
|
||||
if m.CreatedAt.IsZero() {
|
||||
return fmt.Errorf("save run manifest: created_at is required")
|
||||
}
|
||||
|
||||
m.UpdatedAt = time.Now().UTC()
|
||||
if m.Stages == nil {
|
||||
m.Stages = map[string]*RunStageRecord{}
|
||||
}
|
||||
|
||||
data, err := json.MarshalIndent(m, "", " ")
|
||||
if err != nil {
|
||||
return fmt.Errorf("save run manifest: marshal: %w", err)
|
||||
}
|
||||
data = append(data, '\n')
|
||||
|
||||
return writeJSONAtomically(ctx, path, ".run-manifest.json.tmp-*", data)
|
||||
}
|
||||
|
||||
func validateLoadedManifest(m *Manifest) error {
|
||||
if m == nil {
|
||||
return fmt.Errorf("manifest is nil")
|
||||
@@ -161,6 +244,88 @@ func normalizeManifest(m *Manifest) {
|
||||
}
|
||||
}
|
||||
|
||||
func validateLoadedRunManifest(m *RunManifest) error {
|
||||
if m == nil {
|
||||
return fmt.Errorf("manifest is nil")
|
||||
}
|
||||
if strings.TrimSpace(m.SessionID) == "" {
|
||||
return fmt.Errorf("session_id is required")
|
||||
}
|
||||
if strings.TrimSpace(m.RunID) == "" {
|
||||
return fmt.Errorf("run_id is required")
|
||||
}
|
||||
if m.CreatedAt.IsZero() {
|
||||
return fmt.Errorf("created_at is required")
|
||||
}
|
||||
if m.UpdatedAt.IsZero() {
|
||||
return fmt.Errorf("updated_at is required")
|
||||
}
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
func normalizeRunManifest(m *RunManifest) {
|
||||
if m.Stages == nil {
|
||||
m.Stages = map[string]*RunStageRecord{}
|
||||
}
|
||||
for name, stage := range m.Stages {
|
||||
if stage == nil {
|
||||
stage = &RunStageRecord{
|
||||
Name: name,
|
||||
Action: RunStageActionRun,
|
||||
Status: StatusPending,
|
||||
CreatedAt: m.CreatedAt,
|
||||
UpdatedAt: m.UpdatedAt,
|
||||
}
|
||||
m.Stages[name] = stage
|
||||
}
|
||||
if stage.Name == "" {
|
||||
stage.Name = name
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func writeJSONAtomically(ctx context.Context, path, tempPattern string, data []byte) error {
|
||||
dir := filepath.Dir(path)
|
||||
if err := os.MkdirAll(dir, 0o755); err != nil {
|
||||
return fmt.Errorf("create directory %q: %w", dir, err)
|
||||
}
|
||||
|
||||
tmp, err := os.CreateTemp(dir, tempPattern)
|
||||
if err != nil {
|
||||
return fmt.Errorf("create temp file: %w", err)
|
||||
}
|
||||
tmpName := tmp.Name()
|
||||
removeTmp := true
|
||||
defer func() {
|
||||
if removeTmp {
|
||||
_ = os.Remove(tmpName)
|
||||
}
|
||||
}()
|
||||
|
||||
if _, err := tmp.Write(data); err != nil {
|
||||
_ = tmp.Close()
|
||||
return fmt.Errorf("write temp file: %w", err)
|
||||
}
|
||||
if err := tmp.Sync(); err != nil {
|
||||
_ = tmp.Close()
|
||||
return fmt.Errorf("sync temp file: %w", err)
|
||||
}
|
||||
if err := tmp.Close(); err != nil {
|
||||
return fmt.Errorf("close temp file: %w", err)
|
||||
}
|
||||
if err := checkContext(ctx); err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
if err := os.Rename(tmpName, path); err != nil {
|
||||
return fmt.Errorf("rename temp file: %w", err)
|
||||
}
|
||||
removeTmp = false
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
func checkContext(ctx context.Context) error {
|
||||
if ctx == nil {
|
||||
return nil
|
||||
|
||||
@@ -150,3 +150,73 @@ func TestLoadRejectsInvalidManifest(t *testing.T) {
|
||||
t.Fatalf("error = %q, want session_id validation", err.Error())
|
||||
}
|
||||
}
|
||||
|
||||
func TestLocalStoreCreateSaveLoadRunManifestRoundTrip(t *testing.T) {
|
||||
store := &LocalStore{}
|
||||
ctx := context.Background()
|
||||
|
||||
run, err := store.CreateRun(
|
||||
ctx,
|
||||
"2026-05-03",
|
||||
"forsaken",
|
||||
"20260517T000000Z-abcdef12",
|
||||
true,
|
||||
[]string{"prepare", "transcribe"},
|
||||
)
|
||||
if err != nil {
|
||||
t.Fatalf("CreateRun() error = %v", err)
|
||||
}
|
||||
run.SessionManifestPath = "/var/lib/narratio/work/forsaken/2026-05-03/manifest.json"
|
||||
run.MarkStageRunning("prepare", time.Date(2026, 5, 3, 12, 1, 0, 0, time.UTC))
|
||||
run.MarkStageSucceeded("prepare", time.Date(2026, 5, 3, 12, 2, 0, 0, time.UTC), []ArtifactRecord{
|
||||
{Kind: "input", LocalPath: "inputs/session.yml"},
|
||||
})
|
||||
run.MarkSucceeded(time.Date(2026, 5, 3, 12, 3, 0, 0, time.UTC))
|
||||
|
||||
path := filepath.Join(t.TempDir(), "run-manifest.json")
|
||||
if err := store.SaveRun(ctx, path, run); err != nil {
|
||||
t.Fatalf("SaveRun() error = %v", err)
|
||||
}
|
||||
|
||||
loaded, err := store.LoadRun(ctx, path)
|
||||
if err != nil {
|
||||
t.Fatalf("LoadRun() error = %v", err)
|
||||
}
|
||||
|
||||
if loaded.SessionID != "2026-05-03" {
|
||||
t.Fatalf("SessionID = %q, want %q", loaded.SessionID, "2026-05-03")
|
||||
}
|
||||
if loaded.Campaign != "forsaken" {
|
||||
t.Fatalf("Campaign = %q, want %q", loaded.Campaign, "forsaken")
|
||||
}
|
||||
if loaded.RunID != "20260517T000000Z-abcdef12" {
|
||||
t.Fatalf("RunID = %q, want %q", loaded.RunID, "20260517T000000Z-abcdef12")
|
||||
}
|
||||
if loaded.Status != RunManifestStatusSucceeded {
|
||||
t.Fatalf("Status = %q, want %q", loaded.Status, RunManifestStatusSucceeded)
|
||||
}
|
||||
if loaded.Stages["prepare"] == nil || loaded.Stages["prepare"].Status != StatusSucceeded {
|
||||
t.Fatalf("prepare stage = %#v, want succeeded", loaded.Stages["prepare"])
|
||||
}
|
||||
if loaded.Stages["prepare"].Action != RunStageActionRun {
|
||||
t.Fatalf("prepare action = %q, want %q", loaded.Stages["prepare"].Action, RunStageActionRun)
|
||||
}
|
||||
}
|
||||
|
||||
func TestLoadRunRejectsInvalidManifest(t *testing.T) {
|
||||
store := &LocalStore{}
|
||||
ctx := context.Background()
|
||||
|
||||
path := filepath.Join(t.TempDir(), "run-manifest.json")
|
||||
if err := os.WriteFile(path, []byte(`{"session_id":"2026-05-03"}`), 0o644); err != nil {
|
||||
t.Fatalf("WriteFile() error = %v", err)
|
||||
}
|
||||
|
||||
_, err := store.LoadRun(ctx, path)
|
||||
if err == nil {
|
||||
t.Fatal("expected error, got nil")
|
||||
}
|
||||
if !strings.Contains(err.Error(), "run_id is required") {
|
||||
t.Fatalf("error = %q, want run_id validation", err.Error())
|
||||
}
|
||||
}
|
||||
|
||||
@@ -3,6 +3,7 @@ package stage
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"os"
|
||||
"path/filepath"
|
||||
@@ -58,7 +59,11 @@ func (analyzeStage) Run(ctx context.Context, env *Env, m *manifest.Manifest) (*S
|
||||
return nil, fmt.Errorf("analyze: session id is required")
|
||||
}
|
||||
|
||||
paths := env.ArtifactStore.SessionPaths(sessionID)
|
||||
paths := sessionPathsForEnv(env, sessionID)
|
||||
runLayout, err := resolveRunStageLayout(env, m, paths, sessionID, "analyze")
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("analyze: resolve run-stage layout: %w", err)
|
||||
}
|
||||
if env.Config.Pipeline.Scriptorium == nil {
|
||||
return &StageResult{
|
||||
Metadata: map[string]any{
|
||||
@@ -83,27 +88,7 @@ func (analyzeStage) Run(ctx context.Context, env *Env, m *manifest.Manifest) (*S
|
||||
}, nil
|
||||
}
|
||||
|
||||
processedTranscriptPath, processedSource, err := discoverProcessedTranscript(m, paths)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("analyze: resolve processed transcript: %w", err)
|
||||
}
|
||||
normalizedTranscriptPath, normalizedSource, err := discoverNormalizedTranscript(m, paths)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("analyze: resolve normalized transcript: %w", err)
|
||||
}
|
||||
trimmedTranscriptPath, trimmedSource, err := discoverTrimmedTranscript(m, paths)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("analyze: resolve trimmed transcript: %w", err)
|
||||
}
|
||||
|
||||
transcriptInputs := analyzeTranscriptInputs{
|
||||
ProcessedPath: processedTranscriptPath,
|
||||
ProcessedSource: processedSource,
|
||||
NormalizedPath: normalizedTranscriptPath,
|
||||
NormalizedSource: normalizedSource,
|
||||
TrimmedPath: trimmedTranscriptPath,
|
||||
TrimmedSource: trimmedSource,
|
||||
}
|
||||
transcriptRefs := discoverAnalyzeTranscriptRefs(m, paths)
|
||||
|
||||
inputPaths := map[string]string{}
|
||||
omittedOptionalInputs := []string{}
|
||||
@@ -111,7 +96,7 @@ func (analyzeStage) Run(ctx context.Context, env *Env, m *manifest.Manifest) (*S
|
||||
inputNames := sortedScriptoriumInputNames(artifactCfg.Inputs)
|
||||
for _, inputName := range inputNames {
|
||||
inputCfg := artifactCfg.Inputs[inputName]
|
||||
resolvedPath, resolved, resolveErr := resolveScriptoriumInput(inputName, inputCfg, transcriptInputs, paths, sessionDir)
|
||||
resolvedPath, resolved, resolveErr := resolveScriptoriumInput(inputName, inputCfg, m, paths, sessionDir)
|
||||
if resolveErr != nil {
|
||||
return nil, fmt.Errorf("analyze: resolve input %q: %w", inputName, resolveErr)
|
||||
}
|
||||
@@ -130,13 +115,22 @@ func (analyzeStage) Run(ctx context.Context, env *Env, m *manifest.Manifest) (*S
|
||||
return nil, fmt.Errorf("analyze: resolve vars: %w", err)
|
||||
}
|
||||
|
||||
outputPath, err := resolveScriptoriumOutputPath(paths, artifactCfg.OutputPath)
|
||||
canonicalOutputPath, err := resolveScriptoriumOutputPath(paths, artifactCfg.OutputPath)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("analyze: resolve output path: %w", err)
|
||||
}
|
||||
outputPath, err := runLocalPathForCanonical(runLayout, paths, canonicalOutputPath)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("analyze: resolve run-local output path: %w", err)
|
||||
}
|
||||
stdoutLogPath := filepath.Join(paths.LogsDir, "scriptorium."+artifactName+".stdout.log")
|
||||
stderrLogPath := filepath.Join(paths.LogsDir, "scriptorium."+artifactName+".stderr.log")
|
||||
generatedConfigPath := filepath.Join(paths.ConfigDir, "scriptorium."+artifactName+".generated.yml")
|
||||
if runLayout.Enabled {
|
||||
stdoutLogPath = filepath.Join(runLayout.LogsDir, "scriptorium."+artifactName+".stdout.log")
|
||||
stderrLogPath = filepath.Join(runLayout.LogsDir, "scriptorium."+artifactName+".stderr.log")
|
||||
generatedConfigPath = filepath.Join(runLayout.ConfigDir, "scriptorium."+artifactName+".generated.yml")
|
||||
}
|
||||
|
||||
timeout, err := resolveScriptoriumTimeout(env.Config.Pipeline.Scriptorium.Timeout, artifactCfg.Timeout)
|
||||
if err != nil {
|
||||
@@ -157,20 +151,28 @@ func (analyzeStage) Run(ctx context.Context, env *Env, m *manifest.Manifest) (*S
|
||||
"omitted_optional_inputs": omittedOptionalInputs,
|
||||
"vars": vars,
|
||||
"timeout": timeout.String(),
|
||||
"processed_transcript_path": processedTranscriptPath,
|
||||
"processed_transcript_source": processedSource,
|
||||
"normalized_transcript_path": normalizedTranscriptPath,
|
||||
"normalized_transcript_source": normalizedSource,
|
||||
"trimmed_transcript_path": trimmedTranscriptPath,
|
||||
"trimmed_transcript_source": trimmedSource,
|
||||
"processed_transcript_path": transcriptRefs.ProcessedPath,
|
||||
"processed_transcript_source": transcriptRefs.ProcessedSource,
|
||||
"normalized_transcript_path": transcriptRefs.NormalizedPath,
|
||||
"normalized_transcript_source": transcriptRefs.NormalizedSource,
|
||||
"trimmed_transcript_path": transcriptRefs.TrimmedPath,
|
||||
"trimmed_transcript_source": transcriptRefs.TrimmedSource,
|
||||
"render_debug_enabled": resolveRenderDebugEnabled(env.Config.Pipeline.Scriptorium.RenderDebug, artifactCfg.RenderDebug),
|
||||
}
|
||||
|
||||
if meta["render_debug_enabled"] == true {
|
||||
renderOutputPath := filepath.Join(paths.ArtifactsDir, artifactName+".render.json")
|
||||
if runLayout.Enabled {
|
||||
renderOutputPath = filepath.Join(runLayout.ReportsDir, artifactName+".render.json")
|
||||
}
|
||||
renderStdoutPath := filepath.Join(paths.LogsDir, "scriptorium."+artifactName+".render.stdout.log")
|
||||
renderStderrPath := filepath.Join(paths.LogsDir, "scriptorium."+artifactName+".render.stderr.log")
|
||||
renderGeneratedConfigPath := filepath.Join(paths.ConfigDir, "scriptorium."+artifactName+".render.generated.yml")
|
||||
if runLayout.Enabled {
|
||||
renderStdoutPath = filepath.Join(runLayout.LogsDir, "scriptorium."+artifactName+".render.stdout.log")
|
||||
renderStderrPath = filepath.Join(runLayout.LogsDir, "scriptorium."+artifactName+".render.stderr.log")
|
||||
renderGeneratedConfigPath = filepath.Join(runLayout.ConfigDir, "scriptorium."+artifactName+".render.generated.yml")
|
||||
}
|
||||
|
||||
renderReq := scriptorium.RenderArtifactRequest{
|
||||
Binary: env.Config.Pipeline.Scriptorium.Binary,
|
||||
@@ -257,17 +259,19 @@ func (analyzeStage) Run(ctx context.Context, env *Env, m *manifest.Manifest) (*S
|
||||
if err := requireNonEmptyFile(finalOutputPath, artifactName+" output"); err != nil {
|
||||
return nil, fmt.Errorf("analyze: %w", err)
|
||||
}
|
||||
|
||||
artifactRef := artifacts.Ref{
|
||||
Kind: artifactName,
|
||||
Category: "artifacts",
|
||||
SessionID: sessionID,
|
||||
AbsolutePath: finalOutputPath,
|
||||
promotedArtifact, err := promoteRunLocalOutput(env.ArtifactStore, finalOutputPath, canonicalOutputPath, artifacts.Ref{
|
||||
Kind: artifactName,
|
||||
Category: "artifacts",
|
||||
SessionID: sessionID,
|
||||
})
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("analyze: promote artifact output: %w", err)
|
||||
}
|
||||
|
||||
logPaths = append(logPaths, stdoutLogPath, stderrLogPath)
|
||||
generatedConfigs = append(generatedConfigs, generatedConfigPath)
|
||||
meta["output_path"] = finalOutputPath
|
||||
meta["run_output_path"] = finalOutputPath
|
||||
meta["output_path"] = canonicalOutputPath
|
||||
meta["generated_config_path"] = generatedConfigPath
|
||||
meta["stdout_log_path"] = stdoutLogPath
|
||||
meta["stderr_log_path"] = stderrLogPath
|
||||
@@ -286,7 +290,7 @@ func (analyzeStage) Run(ctx context.Context, env *Env, m *manifest.Manifest) (*S
|
||||
}
|
||||
|
||||
return &StageResult{
|
||||
Outputs: []artifacts.Ref{artifactRef},
|
||||
Outputs: []artifacts.Ref{promotedArtifact},
|
||||
Logs: logPaths,
|
||||
GeneratedConfigs: generatedConfigs,
|
||||
Metadata: meta,
|
||||
@@ -350,40 +354,6 @@ func discoverProcessedTranscript(m *manifest.Manifest, paths artifacts.SessionPa
|
||||
return "", "", nil
|
||||
}
|
||||
|
||||
func discoverTrimmedTranscript(m *manifest.Manifest, paths artifacts.SessionPaths) (string, string, error) {
|
||||
candidates := []string{}
|
||||
if m != nil && m.Stages != nil {
|
||||
if sr := m.Stages["trim"]; sr != nil {
|
||||
for _, out := range sr.Outputs {
|
||||
if out.Kind != "transcript_trimmed" {
|
||||
continue
|
||||
}
|
||||
p := strings.TrimSpace(out.LocalPath)
|
||||
if p == "" {
|
||||
continue
|
||||
}
|
||||
resolved := artifacts.ResolveSessionLocalPathForRead(paths, p)
|
||||
candidates = append(candidates, filepath.Clean(resolved))
|
||||
}
|
||||
}
|
||||
}
|
||||
deduped := dedupeAndSortPaths(candidates)
|
||||
for _, p := range deduped {
|
||||
if info, err := os.Stat(p); err == nil && !info.IsDir() {
|
||||
return p, "manifest.trim.outputs", nil
|
||||
}
|
||||
}
|
||||
|
||||
fallback := filepath.Join(paths.TranscriptsDir, "trimmed.json")
|
||||
if info, err := os.Stat(fallback); err == nil && !info.IsDir() {
|
||||
return filepath.Clean(fallback), "fallback.transcripts_dir", nil
|
||||
}
|
||||
if len(deduped) > 0 {
|
||||
return deduped[0], "manifest.trim.outputs", nil
|
||||
}
|
||||
return "", "", nil
|
||||
}
|
||||
|
||||
type analyzeTranscriptInputs struct {
|
||||
ProcessedPath string
|
||||
ProcessedSource string
|
||||
@@ -393,38 +363,36 @@ type analyzeTranscriptInputs struct {
|
||||
TrimmedSource string
|
||||
}
|
||||
|
||||
func discoverAnalyzeTranscriptRefs(m *manifest.Manifest, paths artifacts.SessionPaths) analyzeTranscriptInputs {
|
||||
processedPath, processedSource := discoverAnalyzeArtifactRef(m, paths, artifacts.ArtifactTranscriptPolished)
|
||||
normalizedPath, normalizedSource := discoverAnalyzeArtifactRef(m, paths, artifacts.ArtifactTranscriptFull)
|
||||
trimmedPath, trimmedSource := discoverAnalyzeArtifactRef(m, paths, artifacts.ArtifactTranscriptTrimmed)
|
||||
return analyzeTranscriptInputs{
|
||||
ProcessedPath: processedPath,
|
||||
ProcessedSource: processedSource,
|
||||
NormalizedPath: normalizedPath,
|
||||
NormalizedSource: normalizedSource,
|
||||
TrimmedPath: trimmedPath,
|
||||
TrimmedSource: trimmedSource,
|
||||
}
|
||||
}
|
||||
|
||||
func discoverAnalyzeArtifactRef(m *manifest.Manifest, paths artifacts.SessionPaths, source string) (string, string) {
|
||||
resolved, err := artifacts.ResolveSessionArtifact(paths, m, source)
|
||||
if err != nil {
|
||||
return "", ""
|
||||
}
|
||||
return resolved.Path, resolved.Provenance
|
||||
}
|
||||
|
||||
func resolveScriptoriumInput(
|
||||
inputName string,
|
||||
inputCfg config.ScriptoriumInputConfig,
|
||||
transcriptInputs analyzeTranscriptInputs,
|
||||
m *manifest.Manifest,
|
||||
paths artifacts.SessionPaths,
|
||||
sessionDir string,
|
||||
) (string, bool, error) {
|
||||
switch strings.TrimSpace(inputCfg.Source) {
|
||||
case "processed_transcript":
|
||||
if strings.TrimSpace(transcriptInputs.ProcessedPath) == "" {
|
||||
return "", false, nil
|
||||
}
|
||||
if err := validateProcessedTranscriptOutput(transcriptInputs.ProcessedPath); err != nil {
|
||||
return "", false, fmt.Errorf("processed transcript %q invalid: %w", transcriptInputs.ProcessedPath, err)
|
||||
}
|
||||
return transcriptInputs.ProcessedPath, true, nil
|
||||
case "normalized_transcript":
|
||||
if strings.TrimSpace(transcriptInputs.NormalizedPath) == "" {
|
||||
return "", false, fmt.Errorf("normalized transcript input is unavailable; run normalize stage first")
|
||||
}
|
||||
if err := validateProcessedTranscriptOutput(transcriptInputs.NormalizedPath); err != nil {
|
||||
return "", false, fmt.Errorf("normalized transcript %q invalid: %w", transcriptInputs.NormalizedPath, err)
|
||||
}
|
||||
return transcriptInputs.NormalizedPath, true, nil
|
||||
case "trimmed_transcript":
|
||||
if strings.TrimSpace(transcriptInputs.TrimmedPath) == "" {
|
||||
return "", false, fmt.Errorf("trimmed transcript input is unavailable; run trim stage first")
|
||||
}
|
||||
if err := validateProcessedTranscriptOutput(transcriptInputs.TrimmedPath); err != nil {
|
||||
return "", false, fmt.Errorf("trimmed transcript %q invalid: %w", transcriptInputs.TrimmedPath, err)
|
||||
}
|
||||
return transcriptInputs.TrimmedPath, true, nil
|
||||
case "previous_session_artifact":
|
||||
if strings.TrimSpace(inputCfg.Path) == "" {
|
||||
return "", false, nil
|
||||
@@ -435,7 +403,27 @@ func resolveScriptoriumInput(
|
||||
}
|
||||
return resolved, true, nil
|
||||
default:
|
||||
return "", false, fmt.Errorf("unsupported source %q", inputCfg.Source)
|
||||
resolved, err := artifacts.ResolveSessionArtifact(paths, m, inputCfg.Source)
|
||||
if err == nil {
|
||||
return resolved.Path, true, nil
|
||||
}
|
||||
if errors.Is(err, artifacts.ErrSessionArtifactNotFound) {
|
||||
normalized, normalizeErr := artifacts.NormalizeSessionArtifactSource(inputCfg.Source)
|
||||
if normalizeErr != nil {
|
||||
return "", false, normalizeErr
|
||||
}
|
||||
switch normalized {
|
||||
case artifacts.ArtifactTranscriptPolished:
|
||||
return "", false, nil
|
||||
case artifacts.ArtifactTranscriptFull:
|
||||
return "", false, fmt.Errorf("normalized transcript input is unavailable; run normalize stage first")
|
||||
case artifacts.ArtifactTranscriptTrimmed:
|
||||
return "", false, fmt.Errorf("trimmed transcript input is unavailable; run trim stage first")
|
||||
default:
|
||||
return "", false, nil
|
||||
}
|
||||
}
|
||||
return "", false, err
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
@@ -17,7 +17,7 @@ import (
|
||||
|
||||
func TestAnalyzeGeneratesSessionRecapFromTrimmedTranscript(t *testing.T) {
|
||||
env, m, fake := setupAnalyzeEnv(t)
|
||||
paths := env.ArtifactStore.SessionPaths(m.SessionID)
|
||||
paths := sessionPathsForEnv(env, m.SessionID)
|
||||
writeAnalyzeFile(t, filepath.Join(paths.TranscriptsDir, "trimmed.json"), `{"segments":[]}`)
|
||||
|
||||
result, err := (analyzeStage{}).Run(context.Background(), env, m)
|
||||
@@ -70,7 +70,7 @@ func TestAnalyzeGeneratesSessionRecapFromTrimmedTranscript(t *testing.T) {
|
||||
|
||||
func TestAnalyzeRenderDebugFalseDoesNotCallRenderArtifact(t *testing.T) {
|
||||
env, m, fake := setupAnalyzeEnv(t)
|
||||
paths := env.ArtifactStore.SessionPaths(m.SessionID)
|
||||
paths := sessionPathsForEnv(env, m.SessionID)
|
||||
writeAnalyzeFile(t, filepath.Join(paths.TranscriptsDir, "trimmed.json"), `{"segments":[]}`)
|
||||
|
||||
env.Config.Pipeline.Scriptorium.RenderDebug = false
|
||||
@@ -86,7 +86,7 @@ func TestAnalyzeRenderDebugFalseDoesNotCallRenderArtifact(t *testing.T) {
|
||||
|
||||
func TestAnalyzeRenderDebugArtifactOverrideFalseWinsOverGlobalTrue(t *testing.T) {
|
||||
env, m, fake := setupAnalyzeEnv(t)
|
||||
paths := env.ArtifactStore.SessionPaths(m.SessionID)
|
||||
paths := sessionPathsForEnv(env, m.SessionID)
|
||||
writeAnalyzeFile(t, filepath.Join(paths.TranscriptsDir, "trimmed.json"), `{"segments":[]}`)
|
||||
|
||||
env.Config.Pipeline.Scriptorium.RenderDebug = true
|
||||
@@ -109,7 +109,7 @@ func TestAnalyzeRenderDebugArtifactOverrideFalseWinsOverGlobalTrue(t *testing.T)
|
||||
|
||||
func TestAnalyzeRenderDebugTrueCallsRenderBeforeRun(t *testing.T) {
|
||||
env, m, _ := setupAnalyzeEnv(t)
|
||||
paths := env.ArtifactStore.SessionPaths(m.SessionID)
|
||||
paths := sessionPathsForEnv(env, m.SessionID)
|
||||
writeAnalyzeFile(t, filepath.Join(paths.TranscriptsDir, "trimmed.json"), `{"segments":[]}`)
|
||||
|
||||
env.Config.Pipeline.Scriptorium.RenderDebug = true
|
||||
@@ -130,7 +130,7 @@ func TestAnalyzeRenderDebugTrueCallsRenderBeforeRun(t *testing.T) {
|
||||
|
||||
func TestAnalyzeRenderOutputPathIsRecorded(t *testing.T) {
|
||||
env, m, fake := setupAnalyzeEnv(t)
|
||||
paths := env.ArtifactStore.SessionPaths(m.SessionID)
|
||||
paths := sessionPathsForEnv(env, m.SessionID)
|
||||
writeAnalyzeFile(t, filepath.Join(paths.TranscriptsDir, "trimmed.json"), `{"segments":[]}`)
|
||||
env.Config.Pipeline.Scriptorium.RenderDebug = true
|
||||
|
||||
@@ -154,7 +154,7 @@ func TestAnalyzeRenderOutputPathIsRecorded(t *testing.T) {
|
||||
|
||||
func TestAnalyzeRenderFailurePreventsRun(t *testing.T) {
|
||||
env, m, fake := setupAnalyzeEnv(t)
|
||||
paths := env.ArtifactStore.SessionPaths(m.SessionID)
|
||||
paths := sessionPathsForEnv(env, m.SessionID)
|
||||
writeAnalyzeFile(t, filepath.Join(paths.TranscriptsDir, "trimmed.json"), `{"segments":[]}`)
|
||||
env.Config.Pipeline.Scriptorium.RenderDebug = true
|
||||
fake.RenderErr = errors.New("render boom")
|
||||
@@ -176,7 +176,7 @@ func TestAnalyzeRenderFailurePreventsRun(t *testing.T) {
|
||||
|
||||
func TestAnalyzeRenderInvalidJSONFailsClearly(t *testing.T) {
|
||||
env, m, _ := setupAnalyzeEnv(t)
|
||||
paths := env.ArtifactStore.SessionPaths(m.SessionID)
|
||||
paths := sessionPathsForEnv(env, m.SessionID)
|
||||
writeAnalyzeFile(t, filepath.Join(paths.TranscriptsDir, "trimmed.json"), `{"segments":[]}`)
|
||||
env.Config.Pipeline.Scriptorium.RenderDebug = true
|
||||
runner := &orderedScriptoriumRunner{
|
||||
@@ -199,7 +199,7 @@ func TestAnalyzeRenderInvalidJSONFailsClearly(t *testing.T) {
|
||||
|
||||
func TestAnalyzeRunStillSucceedsWhenRenderSucceeds(t *testing.T) {
|
||||
env, m, fake := setupAnalyzeEnv(t)
|
||||
paths := env.ArtifactStore.SessionPaths(m.SessionID)
|
||||
paths := sessionPathsForEnv(env, m.SessionID)
|
||||
writeAnalyzeFile(t, filepath.Join(paths.TranscriptsDir, "trimmed.json"), `{"segments":[]}`)
|
||||
env.Config.Pipeline.Scriptorium.RenderDebug = true
|
||||
|
||||
@@ -220,7 +220,7 @@ func TestAnalyzeRunStillSucceedsWhenRenderSucceeds(t *testing.T) {
|
||||
|
||||
func TestAnalyzeOmitsOptionalPreviousRecapWhenUnavailable(t *testing.T) {
|
||||
env, m, fake := setupAnalyzeEnv(t)
|
||||
paths := env.ArtifactStore.SessionPaths(m.SessionID)
|
||||
paths := sessionPathsForEnv(env, m.SessionID)
|
||||
writeAnalyzeFile(t, filepath.Join(paths.TranscriptsDir, "processed.json"), `{"segments":[]}`)
|
||||
|
||||
env.Config.Pipeline.Scriptorium.Artifacts["session_recap"] = config.ScriptoriumArtifactConfig{
|
||||
@@ -256,6 +256,32 @@ func TestAnalyzeOmitsOptionalPreviousRecapWhenUnavailable(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
func TestAnalyzeUsesRunLocalPathsAndPromotesCanonical(t *testing.T) {
|
||||
env, m, fake := setupAnalyzeEnv(t)
|
||||
env.Config.Session.Campaign = "sample-campaign"
|
||||
m.Campaign = "sample-campaign"
|
||||
m.RunID = "20260518T010203Z-abcdef12"
|
||||
paths := sessionPathsForEnv(env, m.SessionID)
|
||||
writeAnalyzeFile(t, filepath.Join(paths.TranscriptsDir, "trimmed.json"), `{"segments":[]}`)
|
||||
|
||||
result, err := (analyzeStage{}).Run(context.Background(), env, m)
|
||||
if err != nil {
|
||||
t.Fatalf("Run() error = %v", err)
|
||||
}
|
||||
if len(fake.RunRequests) != 1 {
|
||||
t.Fatalf("run requests = %d, want 1", len(fake.RunRequests))
|
||||
}
|
||||
if !strings.Contains(fake.RunRequests[0].OutputPath, filepath.Join("runs", m.RunID, "analyze", "outputs")) {
|
||||
t.Fatalf("run output path = %q, want run-local path", fake.RunRequests[0].OutputPath)
|
||||
}
|
||||
if len(result.Outputs) != 1 {
|
||||
t.Fatalf("outputs len = %d, want 1", len(result.Outputs))
|
||||
}
|
||||
if strings.Contains(result.Outputs[0].AbsolutePath, string(filepath.Separator)+"runs"+string(filepath.Separator)) {
|
||||
t.Fatalf("promoted output path = %q, want canonical session path", result.Outputs[0].AbsolutePath)
|
||||
}
|
||||
}
|
||||
|
||||
type orderedScriptoriumRunner struct {
|
||||
Calls []string
|
||||
RenderErr error
|
||||
@@ -318,7 +344,7 @@ func (r *orderedScriptoriumRunner) RunArtifact(_ context.Context, req scriptoriu
|
||||
|
||||
func TestAnalyzeIncludesPreviousRecapWhenConfiguredAndAvailable(t *testing.T) {
|
||||
env, m, fake := setupAnalyzeEnv(t)
|
||||
paths := env.ArtifactStore.SessionPaths(m.SessionID)
|
||||
paths := sessionPathsForEnv(env, m.SessionID)
|
||||
writeAnalyzeFile(t, filepath.Join(paths.TranscriptsDir, "processed.json"), `{"segments":[]}`)
|
||||
|
||||
previousRecapPath := filepath.Join(filepath.Dir(env.Config.SessionPath), "previous", "session_recap.md")
|
||||
@@ -359,7 +385,7 @@ func TestAnalyzeIncludesPreviousRecapWhenConfiguredAndAvailable(t *testing.T) {
|
||||
|
||||
func TestAnalyzeFailsWhenRequiredPreviousRecapMissing(t *testing.T) {
|
||||
env, m, _ := setupAnalyzeEnv(t)
|
||||
paths := env.ArtifactStore.SessionPaths(m.SessionID)
|
||||
paths := sessionPathsForEnv(env, m.SessionID)
|
||||
writeAnalyzeFile(t, filepath.Join(paths.TranscriptsDir, "processed.json"), `{"segments":[]}`)
|
||||
|
||||
env.Config.Pipeline.Scriptorium.Artifacts["session_recap"] = config.ScriptoriumArtifactConfig{
|
||||
@@ -390,7 +416,7 @@ func TestAnalyzeFailsWhenRequiredPreviousRecapMissing(t *testing.T) {
|
||||
|
||||
func TestAnalyzeFailsWhenOutputPathMissing(t *testing.T) {
|
||||
env, m, fake := setupAnalyzeEnv(t)
|
||||
paths := env.ArtifactStore.SessionPaths(m.SessionID)
|
||||
paths := sessionPathsForEnv(env, m.SessionID)
|
||||
writeAnalyzeFile(t, filepath.Join(paths.TranscriptsDir, "trimmed.json"), `{"segments":[]}`)
|
||||
|
||||
artifact := env.Config.Pipeline.Scriptorium.Artifacts["session_recap"]
|
||||
@@ -425,7 +451,7 @@ func TestAnalyzeFailsWhenTrimmedTranscriptMissing(t *testing.T) {
|
||||
|
||||
func TestAnalyzeSupportsProcessedTranscriptSourceWhenConfigured(t *testing.T) {
|
||||
env, m, fake := setupAnalyzeEnv(t)
|
||||
paths := env.ArtifactStore.SessionPaths(m.SessionID)
|
||||
paths := sessionPathsForEnv(env, m.SessionID)
|
||||
writeAnalyzeFile(t, filepath.Join(paths.TranscriptsDir, "processed.json"), `{"segments":[]}`)
|
||||
|
||||
artifact := env.Config.Pipeline.Scriptorium.Artifacts["session_recap"]
|
||||
@@ -447,9 +473,33 @@ func TestAnalyzeSupportsProcessedTranscriptSourceWhenConfigured(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
func TestAnalyzeSupportsCanonicalTrimmedTranscriptSourceWhenConfigured(t *testing.T) {
|
||||
env, m, fake := setupAnalyzeEnv(t)
|
||||
paths := sessionPathsForEnv(env, m.SessionID)
|
||||
writeAnalyzeFile(t, filepath.Join(paths.TranscriptsDir, "trimmed.json"), `{"segments":[]}`)
|
||||
|
||||
artifact := env.Config.Pipeline.Scriptorium.Artifacts["session_recap"]
|
||||
artifact.Inputs["transcript"] = config.ScriptoriumInputConfig{
|
||||
Source: "narratio.transcript.trimmed",
|
||||
Required: true,
|
||||
}
|
||||
env.Config.Pipeline.Scriptorium.Artifacts["session_recap"] = artifact
|
||||
|
||||
_, err := (analyzeStage{}).Run(context.Background(), env, m)
|
||||
if err != nil {
|
||||
t.Fatalf("Run() error = %v", err)
|
||||
}
|
||||
if len(fake.RunRequests) != 1 {
|
||||
t.Fatalf("run requests = %d, want 1", len(fake.RunRequests))
|
||||
}
|
||||
if fake.RunRequests[0].InputPaths["transcript"] != filepath.Join(paths.TranscriptsDir, "trimmed.json") {
|
||||
t.Fatalf("transcript input = %q, want trimmed transcript path", fake.RunRequests[0].InputPaths["transcript"])
|
||||
}
|
||||
}
|
||||
|
||||
func TestAnalyzeSupportsNormalizedTranscriptSourceWhenConfigured(t *testing.T) {
|
||||
env, m, fake := setupAnalyzeEnv(t)
|
||||
paths := env.ArtifactStore.SessionPaths(m.SessionID)
|
||||
paths := sessionPathsForEnv(env, m.SessionID)
|
||||
normalizedPath := filepath.Join(paths.TranscriptsDir, "normalized.json")
|
||||
writeAnalyzeFile(t, normalizedPath, `{"segments":[{"id":1}]}`)
|
||||
|
||||
@@ -472,9 +522,39 @@ func TestAnalyzeSupportsNormalizedTranscriptSourceWhenConfigured(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
func TestAnalyzeSupportsCanonicalNormalizedTranscriptSourceFromManifestOutput(t *testing.T) {
|
||||
env, m, fake := setupAnalyzeEnv(t)
|
||||
paths := sessionPathsForEnv(env, m.SessionID)
|
||||
fallbackPath := filepath.Join(paths.TranscriptsDir, "normalized.json")
|
||||
manifestPath := filepath.Join(paths.ArtifactsDir, "normalized.from-manifest.json")
|
||||
writeAnalyzeFile(t, fallbackPath, `{"segments":[{"id":999}]}`)
|
||||
writeAnalyzeFile(t, manifestPath, `{"segments":[{"id":10}]}`)
|
||||
m.MarkStageSucceeded("normalize", time.Now().UTC(), []manifest.ArtifactRecord{
|
||||
{Kind: "transcript_normalized", LocalPath: manifestPath},
|
||||
})
|
||||
|
||||
artifact := env.Config.Pipeline.Scriptorium.Artifacts["session_recap"]
|
||||
artifact.Inputs["transcript"] = config.ScriptoriumInputConfig{
|
||||
Source: "narratio.transcript.full",
|
||||
Required: true,
|
||||
}
|
||||
env.Config.Pipeline.Scriptorium.Artifacts["session_recap"] = artifact
|
||||
|
||||
_, err := (analyzeStage{}).Run(context.Background(), env, m)
|
||||
if err != nil {
|
||||
t.Fatalf("Run() error = %v", err)
|
||||
}
|
||||
if len(fake.RunRequests) != 1 {
|
||||
t.Fatalf("run requests = %d, want 1", len(fake.RunRequests))
|
||||
}
|
||||
if fake.RunRequests[0].InputPaths["transcript"] != manifestPath {
|
||||
t.Fatalf("transcript input = %q, want manifest normalized transcript path", fake.RunRequests[0].InputPaths["transcript"])
|
||||
}
|
||||
}
|
||||
|
||||
func TestAnalyzeSupportsNormalizedTranscriptSourceFromManifestOutput(t *testing.T) {
|
||||
env, m, fake := setupAnalyzeEnv(t)
|
||||
paths := env.ArtifactStore.SessionPaths(m.SessionID)
|
||||
paths := sessionPathsForEnv(env, m.SessionID)
|
||||
fallbackPath := filepath.Join(paths.TranscriptsDir, "normalized.json")
|
||||
manifestPath := filepath.Join(paths.ArtifactsDir, "normalized.from-manifest.json")
|
||||
writeAnalyzeFile(t, fallbackPath, `{"segments":[{"id":999}]}`)
|
||||
@@ -528,7 +608,7 @@ func TestAnalyzeFailsWhenNormalizedTranscriptMissing(t *testing.T) {
|
||||
|
||||
func TestAnalyzeFailsWhenProcessedTranscriptInvalidJSON(t *testing.T) {
|
||||
env, m, _ := setupAnalyzeEnv(t)
|
||||
paths := env.ArtifactStore.SessionPaths(m.SessionID)
|
||||
paths := sessionPathsForEnv(env, m.SessionID)
|
||||
writeAnalyzeFile(t, filepath.Join(paths.TranscriptsDir, "trimmed.json"), `{not-json`)
|
||||
|
||||
_, err := (analyzeStage{}).Run(context.Background(), env, m)
|
||||
@@ -542,7 +622,7 @@ func TestAnalyzeFailsWhenProcessedTranscriptInvalidJSON(t *testing.T) {
|
||||
|
||||
func TestAnalyzeFailsWhenProcessedTranscriptMissingSegmentsArray(t *testing.T) {
|
||||
env, m, _ := setupAnalyzeEnv(t)
|
||||
paths := env.ArtifactStore.SessionPaths(m.SessionID)
|
||||
paths := sessionPathsForEnv(env, m.SessionID)
|
||||
writeAnalyzeFile(t, filepath.Join(paths.TranscriptsDir, "trimmed.json"), `{"not_segments":[]}`)
|
||||
|
||||
_, err := (analyzeStage{}).Run(context.Background(), env, m)
|
||||
@@ -556,7 +636,7 @@ func TestAnalyzeFailsWhenProcessedTranscriptMissingSegmentsArray(t *testing.T) {
|
||||
|
||||
func TestAnalyzeRecordsRefsAndMetadata(t *testing.T) {
|
||||
env, m, _ := setupAnalyzeEnv(t)
|
||||
paths := env.ArtifactStore.SessionPaths(m.SessionID)
|
||||
paths := sessionPathsForEnv(env, m.SessionID)
|
||||
writeAnalyzeFile(t, filepath.Join(paths.TranscriptsDir, "trimmed.json"), `{"segments":[]}`)
|
||||
|
||||
result, err := (analyzeStage{}).Run(context.Background(), env, m)
|
||||
@@ -585,7 +665,7 @@ func TestAnalyzeRecordsRefsAndMetadata(t *testing.T) {
|
||||
|
||||
func TestAnalyzeHandlesAdapterError(t *testing.T) {
|
||||
env, m, fake := setupAnalyzeEnv(t)
|
||||
paths := env.ArtifactStore.SessionPaths(m.SessionID)
|
||||
paths := sessionPathsForEnv(env, m.SessionID)
|
||||
writeAnalyzeFile(t, filepath.Join(paths.TranscriptsDir, "trimmed.json"), `{"segments":[]}`)
|
||||
fake.RunErr = errors.New("adapter boom")
|
||||
|
||||
@@ -600,7 +680,7 @@ func TestAnalyzeHandlesAdapterError(t *testing.T) {
|
||||
|
||||
func TestAnalyzeHandlesValidationFailedResultAsError(t *testing.T) {
|
||||
env, m, fake := setupAnalyzeEnv(t)
|
||||
paths := env.ArtifactStore.SessionPaths(m.SessionID)
|
||||
paths := sessionPathsForEnv(env, m.SessionID)
|
||||
writeAnalyzeFile(t, filepath.Join(paths.TranscriptsDir, "trimmed.json"), `{"segments":[]}`)
|
||||
fake.RunResult = scriptorium.ArtifactResult{
|
||||
ValidationFailed: true,
|
||||
@@ -619,7 +699,7 @@ func TestAnalyzeHandlesValidationFailedResultAsError(t *testing.T) {
|
||||
|
||||
func TestAnalyzeSkipsWhenNoEnabledScriptoriumArtifactsConfigured(t *testing.T) {
|
||||
env, m, _ := setupAnalyzeEnv(t)
|
||||
paths := env.ArtifactStore.SessionPaths(m.SessionID)
|
||||
paths := sessionPathsForEnv(env, m.SessionID)
|
||||
writeAnalyzeFile(t, filepath.Join(paths.TranscriptsDir, "trimmed.json"), `{"segments":[]}`)
|
||||
|
||||
env.Config.Pipeline.Scriptorium.Artifacts["session_recap"] = config.ScriptoriumArtifactConfig{
|
||||
@@ -692,7 +772,7 @@ func setupAnalyzeEnv(t *testing.T) (*Env, *manifest.Manifest, *scriptorium.FakeR
|
||||
}
|
||||
|
||||
store := artifacts.NewLocalStore(workspace)
|
||||
if _, err := store.EnsureLayout("2026-05-03"); err != nil {
|
||||
if _, err := store.EnsureLayoutFor("sample-campaign", "2026-05-03"); err != nil {
|
||||
t.Fatalf("EnsureLayout() error = %v", err)
|
||||
}
|
||||
|
||||
|
||||
@@ -18,6 +18,10 @@ import (
|
||||
)
|
||||
|
||||
type archiveStage struct{}
|
||||
type archiveUploadFile struct {
|
||||
RelativePath string
|
||||
LocalPath string
|
||||
}
|
||||
|
||||
var archivePrerequisiteStages = []string{
|
||||
"prepare",
|
||||
@@ -29,15 +33,6 @@ var archivePrerequisiteStages = []string{
|
||||
"analyze",
|
||||
}
|
||||
|
||||
var archiveRunUploadDirs = []string{
|
||||
"inputs",
|
||||
"transcripts",
|
||||
"artifacts",
|
||||
"reports",
|
||||
"config",
|
||||
"logs",
|
||||
}
|
||||
|
||||
func (archiveStage) Name() string { return "archive" }
|
||||
|
||||
func (archiveStage) Declares() IODecl {
|
||||
@@ -83,16 +78,16 @@ func (archiveStage) Run(ctx context.Context, env *Env, m *manifest.Manifest) (*S
|
||||
return nil, fmt.Errorf("archive: remote object store backend is required when archive run upload is enabled")
|
||||
}
|
||||
|
||||
workDir, err := archiveWorkDir(env, m)
|
||||
runRoot, err := resolveArchiveRunRoot(env, m)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("archive: resolve local workdir: %w", err)
|
||||
return nil, fmt.Errorf("archive: resolve run root: %w", err)
|
||||
}
|
||||
workDirInfo, err := os.Stat(workDir)
|
||||
runRootInfo, err := os.Stat(runRoot)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("archive: local workdir %q: %w", workDir, err)
|
||||
return nil, fmt.Errorf("archive: run root %q: %w", runRoot, err)
|
||||
}
|
||||
if !workDirInfo.IsDir() {
|
||||
return nil, fmt.Errorf("archive: local workdir %q is not a directory", workDir)
|
||||
if !runRootInfo.IsDir() {
|
||||
return nil, fmt.Errorf("archive: run root %q is not a directory", runRoot)
|
||||
}
|
||||
|
||||
runPrefix, err := archiveRunPrefix(env, m)
|
||||
@@ -112,29 +107,30 @@ func (archiveStage) Run(ctx context.Context, env *Env, m *manifest.Manifest) (*S
|
||||
return nil, fmt.Errorf("archive: run id is required")
|
||||
}
|
||||
|
||||
runFiles, err := collectArchiveRunFiles(workDir)
|
||||
manifestSource, err := resolveArchiveRunManifestSource(runRoot)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("archive: resolve run manifest source: %w", err)
|
||||
}
|
||||
|
||||
runFiles, err := collectArchiveRunFiles(runRoot, manifestSource)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("archive: collect run files: %w", err)
|
||||
}
|
||||
promotions, err := resolveArchivePromotions(workDir, env.Config.Pipeline.Archive.PromoteArtifacts)
|
||||
sessionRoot, err := resolveArchiveSessionRoot(env, m)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("archive: resolve session root for promotions: %w", err)
|
||||
}
|
||||
promotions, err := resolveArchivePromotions(sessionRoot, env.Config.Pipeline.Archive.PromoteArtifacts)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("archive: resolve promotion rules: %w", err)
|
||||
}
|
||||
currentManifestSource := filepath.Join(workDir, "manifest.json")
|
||||
if info, err := os.Stat(currentManifestSource); err != nil {
|
||||
return nil, fmt.Errorf("archive: current manifest source %q: %w", currentManifestSource, err)
|
||||
} else if info.IsDir() {
|
||||
return nil, fmt.Errorf("archive: current manifest source %q is a directory", currentManifestSource)
|
||||
}
|
||||
|
||||
runUploaded := make([]string, 0, len(runFiles))
|
||||
for _, rel := range runFiles {
|
||||
localPath := filepath.Join(workDir, filepath.FromSlash(rel))
|
||||
key := artifacts.S3RunRelativeDestinationKey(runPrefix, rel)
|
||||
if _, err := env.ObjectStore.Upload(ctx, localPath, key, storage.UploadOptions{}); err != nil {
|
||||
return nil, fmt.Errorf("archive: upload run file %q to %q: %w", rel, key, err)
|
||||
for _, file := range runFiles {
|
||||
key := artifacts.S3RunRelativeDestinationKey(runPrefix, file.RelativePath)
|
||||
if _, err := env.ObjectStore.Upload(ctx, file.LocalPath, key, storage.UploadOptions{}); err != nil {
|
||||
return nil, fmt.Errorf("archive: upload run file %q to %q: %w", file.RelativePath, key, err)
|
||||
}
|
||||
runUploaded = append(runUploaded, rel)
|
||||
runUploaded = append(runUploaded, file.RelativePath)
|
||||
}
|
||||
|
||||
promotedUploaded := make([]string, 0, len(promotions))
|
||||
@@ -247,36 +243,53 @@ func validateArchivePrerequisites(m *manifest.Manifest) error {
|
||||
return nil
|
||||
}
|
||||
|
||||
func archiveWorkDir(env *Env, m *manifest.Manifest) (string, error) {
|
||||
workDir := strings.TrimSpace(m.LocalWorkDir)
|
||||
if workDir != "" {
|
||||
cleaned := filepath.Clean(workDir)
|
||||
if info, err := os.Stat(cleaned); err == nil && info.IsDir() {
|
||||
return cleaned, nil
|
||||
}
|
||||
}
|
||||
|
||||
func resolveArchiveRunRoot(env *Env, m *manifest.Manifest) (string, error) {
|
||||
sessionID := strings.TrimSpace(env.Config.Session.SessionID)
|
||||
if sessionID == "" {
|
||||
if sessionID == "" && m != nil {
|
||||
sessionID = strings.TrimSpace(m.SessionID)
|
||||
}
|
||||
campaign := strings.TrimSpace(env.Config.Session.Campaign)
|
||||
if campaign == "" {
|
||||
if campaign == "" && m != nil {
|
||||
campaign = strings.TrimSpace(m.Campaign)
|
||||
}
|
||||
runID := strings.TrimSpace(m.RunID)
|
||||
runID := ""
|
||||
if m != nil {
|
||||
runID = strings.TrimSpace(m.RunID)
|
||||
}
|
||||
if sessionID == "" || campaign == "" {
|
||||
return "", fmt.Errorf("campaign and session id are required")
|
||||
}
|
||||
if runID == "" {
|
||||
return "", fmt.Errorf("run id is required")
|
||||
}
|
||||
if campaign == "" || sessionID == "" {
|
||||
return "", fmt.Errorf("campaign and session id are required")
|
||||
|
||||
canonical := filepath.Clean(artifacts.SessionRunRootForCampaign(env.Config.Pipeline.Workspace.Root, campaign, sessionID, runID))
|
||||
canonicalExists, err := directoryExists(canonical)
|
||||
if err != nil {
|
||||
return "", fmt.Errorf("check canonical run root %q: %w", canonical, err)
|
||||
}
|
||||
runScoped := artifacts.SessionRunWorkDir(env.Config.Pipeline.Workspace.Root, campaign, sessionID, runID)
|
||||
if info, err := os.Stat(runScoped); err == nil && info.IsDir() {
|
||||
return runScoped, nil
|
||||
if !canonicalExists {
|
||||
return "", fmt.Errorf("run root not found for campaign %q session %q run %q at canonical path %q", campaign, sessionID, runID, canonical)
|
||||
}
|
||||
legacy := artifacts.SessionWorkDir(env.Config.Pipeline.Workspace.Root, sessionID)
|
||||
return legacy, nil
|
||||
return canonical, nil
|
||||
}
|
||||
|
||||
func resolveArchiveSessionRoot(env *Env, m *manifest.Manifest) (string, error) {
|
||||
sessionID := strings.TrimSpace(env.Config.Session.SessionID)
|
||||
if sessionID == "" && m != nil {
|
||||
sessionID = strings.TrimSpace(m.SessionID)
|
||||
}
|
||||
campaign := strings.TrimSpace(env.Config.Session.Campaign)
|
||||
if campaign == "" && m != nil {
|
||||
campaign = strings.TrimSpace(m.Campaign)
|
||||
}
|
||||
if sessionID == "" {
|
||||
return "", fmt.Errorf("session id is required")
|
||||
}
|
||||
if campaign == "" {
|
||||
return "", fmt.Errorf("campaign is required")
|
||||
}
|
||||
return filepath.Clean(artifacts.SessionWorkDirForCampaign(env.Config.Pipeline.Workspace.Root, campaign, sessionID)), nil
|
||||
}
|
||||
|
||||
func archiveRunPrefix(env *Env, m *manifest.Manifest) (string, error) {
|
||||
@@ -329,22 +342,27 @@ func archiveBucket(env *Env, m *manifest.Manifest) string {
|
||||
return strings.TrimSpace(env.Config.Pipeline.Storage.S3.Bucket)
|
||||
}
|
||||
|
||||
func resolveArchivePromotions(workDir string, rules []config.ArchivePromotionRule) ([]archivePromotion, error) {
|
||||
func resolveArchivePromotions(sessionRoot string, rules []config.ArchivePromotionRule) ([]archivePromotion, error) {
|
||||
sessionRoot = filepath.Clean(strings.TrimSpace(sessionRoot))
|
||||
if sessionRoot == "" {
|
||||
return nil, fmt.Errorf("session root is required")
|
||||
}
|
||||
out := make([]archivePromotion, 0, len(rules))
|
||||
for _, rule := range rules {
|
||||
from := strings.TrimSpace(rule.From)
|
||||
to := strings.TrimSpace(rule.To)
|
||||
required := rule.Required == nil || *rule.Required
|
||||
|
||||
localPath, err := resolveWorkDirRelativePath(workDir, from)
|
||||
resolvedPath, err := resolveWorkDirRelativePath(sessionRoot, from)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("promotion from %q: %w", from, err)
|
||||
}
|
||||
info, err := os.Stat(localPath)
|
||||
info, err := os.Stat(resolvedPath)
|
||||
exists := err == nil && !info.IsDir()
|
||||
if err != nil && !os.IsNotExist(err) {
|
||||
return nil, fmt.Errorf("promotion source %q: %w", from, err)
|
||||
}
|
||||
localPath := resolvedPath
|
||||
|
||||
out = append(out, archivePromotion{
|
||||
From: from,
|
||||
@@ -375,58 +393,97 @@ func resolveWorkDirRelativePath(workDir, rel string) (string, error) {
|
||||
return cleanedFull, nil
|
||||
}
|
||||
|
||||
func collectArchiveRunFiles(workDir string) ([]string, error) {
|
||||
files := make([]string, 0, 64)
|
||||
|
||||
for _, dirName := range archiveRunUploadDirs {
|
||||
fullDir := filepath.Join(workDir, dirName)
|
||||
info, err := os.Stat(fullDir)
|
||||
if err != nil {
|
||||
if os.IsNotExist(err) {
|
||||
continue
|
||||
}
|
||||
return nil, fmt.Errorf("stat %q: %w", fullDir, err)
|
||||
func collectArchiveRunFiles(runRoot, manifestPath string) ([]archiveUploadFile, error) {
|
||||
files := make([]archiveUploadFile, 0, 64)
|
||||
err := filepath.WalkDir(runRoot, func(path string, d fs.DirEntry, walkErr error) error {
|
||||
if walkErr != nil {
|
||||
return walkErr
|
||||
}
|
||||
if !info.IsDir() {
|
||||
continue
|
||||
}
|
||||
|
||||
if err := filepath.WalkDir(fullDir, func(path string, d fs.DirEntry, walkErr error) error {
|
||||
if walkErr != nil {
|
||||
return walkErr
|
||||
}
|
||||
if d.IsDir() {
|
||||
if d.IsDir() {
|
||||
if path == runRoot {
|
||||
return nil
|
||||
}
|
||||
rel, err := filepath.Rel(workDir, path)
|
||||
relDir, err := filepath.Rel(runRoot, path)
|
||||
if err != nil {
|
||||
return fmt.Errorf("relative path from %q to %q: %w", workDir, path, err)
|
||||
return fmt.Errorf("relative dir from %q to %q: %w", runRoot, path, err)
|
||||
}
|
||||
relDir = filepath.ToSlash(relDir)
|
||||
// Preserve existing behavior: audio is not uploaded in archive run record.
|
||||
if relDir == "audio" || strings.HasPrefix(relDir, "audio/") {
|
||||
return filepath.SkipDir
|
||||
}
|
||||
rel = filepath.ToSlash(rel)
|
||||
files = append(files, rel)
|
||||
return nil
|
||||
}); err != nil {
|
||||
return nil, fmt.Errorf("walk %q: %w", fullDir, err)
|
||||
}
|
||||
rel, err := filepath.Rel(runRoot, path)
|
||||
if err != nil {
|
||||
return fmt.Errorf("relative path from %q to %q: %w", runRoot, path, err)
|
||||
}
|
||||
rel = filepath.ToSlash(rel)
|
||||
files = append(files, archiveUploadFile{
|
||||
RelativePath: rel,
|
||||
LocalPath: path,
|
||||
})
|
||||
return nil
|
||||
})
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("walk %q: %w", runRoot, err)
|
||||
}
|
||||
|
||||
manifestPath := filepath.Join(workDir, "manifest.json")
|
||||
manifestInfo, err := os.Stat(manifestPath)
|
||||
if err != nil {
|
||||
if os.IsNotExist(err) {
|
||||
return nil, fmt.Errorf("manifest.json not found in workdir %q", workDir)
|
||||
return nil, fmt.Errorf("manifest.json not found (checked path %q)", manifestPath)
|
||||
}
|
||||
return nil, fmt.Errorf("stat %q: %w", manifestPath, err)
|
||||
}
|
||||
if manifestInfo.IsDir() {
|
||||
return nil, fmt.Errorf("manifest path %q is a directory", manifestPath)
|
||||
}
|
||||
files = append(files, "manifest.json")
|
||||
files = append(files, archiveUploadFile{
|
||||
RelativePath: "manifest.json",
|
||||
LocalPath: manifestPath,
|
||||
})
|
||||
seen := map[string]archiveUploadFile{}
|
||||
for _, file := range files {
|
||||
seen[file.RelativePath] = file
|
||||
}
|
||||
files = files[:0]
|
||||
for _, file := range seen {
|
||||
files = append(files, file)
|
||||
}
|
||||
|
||||
sort.Strings(files)
|
||||
sort.Slice(files, func(i, j int) bool {
|
||||
return files[i].RelativePath < files[j].RelativePath
|
||||
})
|
||||
return files, nil
|
||||
}
|
||||
|
||||
func resolveArchiveRunManifestSource(runRoot string) (string, error) {
|
||||
path := filepath.Join(filepath.Clean(runRoot), "manifest.json")
|
||||
info, err := os.Stat(path)
|
||||
if err != nil {
|
||||
if os.IsNotExist(err) {
|
||||
return "", fmt.Errorf("manifest.json not found in run root %q", runRoot)
|
||||
}
|
||||
return "", fmt.Errorf("stat %q: %w", path, err)
|
||||
}
|
||||
if info.IsDir() {
|
||||
return "", fmt.Errorf("manifest path %q is a directory", path)
|
||||
}
|
||||
return path, nil
|
||||
}
|
||||
|
||||
func directoryExists(path string) (bool, error) {
|
||||
info, err := os.Stat(path)
|
||||
if err == nil {
|
||||
return info.IsDir(), nil
|
||||
}
|
||||
if os.IsNotExist(err) {
|
||||
return false, nil
|
||||
}
|
||||
return false, err
|
||||
}
|
||||
|
||||
func writeCurrentManifestSnapshot(m *manifest.Manifest, archiveMetadata map[string]any) (string, error) {
|
||||
if m == nil {
|
||||
return "", fmt.Errorf("manifest is required")
|
||||
|
||||
@@ -3,6 +3,7 @@ package stage
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"reflect"
|
||||
"strings"
|
||||
@@ -72,14 +73,15 @@ func TestArchiveUploadsRunRecordPromotionsAndCurrentPointer(t *testing.T) {
|
||||
runPrefix := m.S3RunPrefix
|
||||
sessionPrefix := m.S3SessionPrefix
|
||||
wantRunUploads := []string{
|
||||
"artifacts/session_recap.md",
|
||||
"config/audita.generated.yml",
|
||||
"inputs/session.yml",
|
||||
"analyze/outputs/artifacts/session_recap.md",
|
||||
"merge/config/seriatim.generated.yml",
|
||||
"prepare/inputs/session.yml",
|
||||
"prepare/outputs/audio/speaker.flac",
|
||||
"logs/audita.stderr.log",
|
||||
"manifest.json",
|
||||
"reports/audita.report.json",
|
||||
"transcripts/raw/speaker.json",
|
||||
"transcripts/trimmed.json",
|
||||
"polish/reports/audita.report.json",
|
||||
"transcribe/outputs/transcripts/raw/speaker.json",
|
||||
"trim/outputs/transcripts/trimmed.json",
|
||||
}
|
||||
for _, rel := range wantRunUploads {
|
||||
key := runPrefix + rel
|
||||
@@ -131,12 +133,11 @@ func TestArchiveUploadsRunRecordPromotionsAndCurrentPointer(t *testing.T) {
|
||||
}
|
||||
|
||||
func TestArchiveUsesCustomPromotionRules(t *testing.T) {
|
||||
env, m, workDir := archiveFixture(t)
|
||||
env, m, _ := archiveFixture(t)
|
||||
env.Config.Pipeline.Archive.PromoteArtifacts = []config.ArchivePromotionRule{
|
||||
{From: "transcripts/trimmed.json", To: "published/trimmed.json", Required: boolPtr(true)},
|
||||
{From: "artifacts/session_recap.md", To: "published/recap.md", Required: boolPtr(true)},
|
||||
}
|
||||
writeStageTestFile(t, filepath.Join(workDir, "published", "ignored.txt"), "ignore\n")
|
||||
|
||||
_, err := archiveStage{}.Run(context.Background(), env, m)
|
||||
if err != nil {
|
||||
@@ -182,6 +183,21 @@ func TestArchiveFailsWhenRequiredPromotionMissing(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
func TestArchiveFailsWhenCanonicalRunRootMissing(t *testing.T) {
|
||||
env, m, runRoot := archiveFixture(t)
|
||||
if err := os.RemoveAll(runRoot); err != nil {
|
||||
t.Fatalf("remove run root: %v", err)
|
||||
}
|
||||
|
||||
_, err := archiveStage{}.Run(context.Background(), env, m)
|
||||
if err == nil {
|
||||
t.Fatal("expected missing run-root error, got nil")
|
||||
}
|
||||
if !strings.Contains(err.Error(), "run root not found") {
|
||||
t.Fatalf("error = %v, want missing run-root error", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestArchiveDoesNotWriteCurrentPointerWhenPromotionUploadFails(t *testing.T) {
|
||||
env, m, _ := archiveFixture(t)
|
||||
fake := env.ObjectStore.(*storage.FakeBackend)
|
||||
@@ -240,22 +256,27 @@ func archiveFixture(t *testing.T) (*Env, *manifest.Manifest, string) {
|
||||
runID := "20260516T010203Z-1a2b3c4d"
|
||||
campaign := "forsaken"
|
||||
sessionID := "2026-04-19"
|
||||
workDir := filepath.Join(root, "work", campaign, sessionID, runID)
|
||||
sessionRoot := artifacts.SessionWorkDirForCampaign(root, campaign, sessionID)
|
||||
runRoot := artifacts.SessionRunRootForCampaign(root, campaign, sessionID, runID)
|
||||
|
||||
writeStageTestFile(t, filepath.Join(workDir, "inputs", "session.yml"), "session_id: 2026-04-19\n")
|
||||
writeStageTestFile(t, filepath.Join(workDir, "transcripts", "raw", "speaker.json"), "{}\n")
|
||||
writeStageTestFile(t, filepath.Join(workDir, "transcripts", "trimmed.json"), "{}\n")
|
||||
writeStageTestFile(t, filepath.Join(workDir, "artifacts", "session_recap.md"), "# recap\n")
|
||||
writeStageTestFile(t, filepath.Join(workDir, "reports", "audita.report.json"), "{}\n")
|
||||
writeStageTestFile(t, filepath.Join(workDir, "config", "audita.generated.yml"), "key: value\n")
|
||||
writeStageTestFile(t, filepath.Join(workDir, "logs", "audita.stderr.log"), "stderr\n")
|
||||
writeStageTestFile(t, filepath.Join(workDir, "audio", "speaker.flac"), "flac")
|
||||
writeStageTestFile(t, filepath.Join(workDir, "manifest.json"), "{}\n")
|
||||
writeStageTestFile(t, filepath.Join(sessionRoot, "transcripts", "trimmed.json"), "{}\n")
|
||||
writeStageTestFile(t, filepath.Join(sessionRoot, "artifacts", "session_recap.md"), "# recap\n")
|
||||
|
||||
writeStageTestFile(t, filepath.Join(runRoot, "prepare", "inputs", "session.yml"), "session_id: 2026-04-19\n")
|
||||
writeStageTestFile(t, filepath.Join(runRoot, "prepare", "outputs", "audio", "speaker.flac"), "flac\n")
|
||||
writeStageTestFile(t, filepath.Join(runRoot, "transcribe", "outputs", "transcripts", "raw", "speaker.json"), "{}\n")
|
||||
writeStageTestFile(t, filepath.Join(runRoot, "trim", "outputs", "transcripts", "trimmed.json"), "{}\n")
|
||||
writeStageTestFile(t, filepath.Join(runRoot, "analyze", "outputs", "artifacts", "session_recap.md"), "# recap\n")
|
||||
writeStageTestFile(t, filepath.Join(runRoot, "polish", "reports", "audita.report.json"), "{}\n")
|
||||
writeStageTestFile(t, filepath.Join(runRoot, "merge", "config", "seriatim.generated.yml"), "key: value\n")
|
||||
writeStageTestFile(t, filepath.Join(runRoot, "logs", "audita.stderr.log"), "stderr\n")
|
||||
writeStageTestFile(t, filepath.Join(runRoot, "audio", "speaker.flac"), "flac")
|
||||
writeStageTestFile(t, filepath.Join(runRoot, "manifest.json"), "{}\n")
|
||||
|
||||
m := manifest.New(sessionID, time.Date(2026, 5, 16, 1, 2, 3, 0, time.UTC))
|
||||
m.Campaign = campaign
|
||||
m.RunID = runID
|
||||
m.LocalWorkDir = workDir
|
||||
m.LocalWorkDir = runRoot
|
||||
m.S3Bucket = "my-dnd-archive"
|
||||
m.S3SessionPrefix = artifacts.S3SessionPrefix("dnd", campaign, sessionID)
|
||||
m.S3RunPrefix = artifacts.S3RunPrefix(m.S3SessionPrefix, runID)
|
||||
@@ -289,7 +310,7 @@ func archiveFixture(t *testing.T) (*Env, *manifest.Manifest, string) {
|
||||
},
|
||||
ObjectStore: &storage.FakeBackend{},
|
||||
}
|
||||
return env, m, workDir
|
||||
return env, m, runRoot
|
||||
}
|
||||
|
||||
type promotionFailingStore struct {
|
||||
|
||||
@@ -57,7 +57,11 @@ func (mergeStage) Run(ctx context.Context, env *Env, m *manifest.Manifest) (*Sta
|
||||
return nil, fmt.Errorf("merge: session id is required")
|
||||
}
|
||||
|
||||
paths := env.ArtifactStore.SessionPaths(sessionID)
|
||||
paths := sessionPathsForEnv(env, sessionID)
|
||||
runLayout, err := resolveRunStageLayout(env, m, paths, sessionID, "merge")
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("merge: resolve run-stage layout: %w", err)
|
||||
}
|
||||
|
||||
inputs, err := discoverRawTranscripts(m, paths)
|
||||
if err != nil {
|
||||
@@ -81,13 +85,29 @@ func (mergeStage) Run(ctx context.Context, env *Env, m *manifest.Manifest) (*Sta
|
||||
return nil, fmt.Errorf("merge: %w", err)
|
||||
}
|
||||
|
||||
mergedPath := filepath.Join(paths.TranscriptsDir, "merged.json")
|
||||
reportPath := filepath.Join(paths.ArtifactsDir, "seriatim.report.json")
|
||||
canonicalMergedPath := filepath.Join(paths.TranscriptsDir, "merged.json")
|
||||
mergedPath, err := runLocalPathForCanonical(runLayout, paths, canonicalMergedPath)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("merge: resolve run-local merged transcript path: %w", err)
|
||||
}
|
||||
canonicalReportPath := filepath.Join(paths.ArtifactsDir, "seriatim.report.json")
|
||||
reportPath := canonicalReportPath
|
||||
if runLayout.Enabled {
|
||||
reportPath, err = runLocalPathForCanonical(runLayout, paths, canonicalReportPath)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("merge: resolve run-local report path: %w", err)
|
||||
}
|
||||
}
|
||||
stdoutPath := filepath.Join(paths.LogsDir, "seriatim.stdout.log")
|
||||
stderrPath := filepath.Join(paths.LogsDir, "seriatim.stderr.log")
|
||||
genCfgPath := filepath.Join(paths.ConfigDir, "seriatim.generated.yml")
|
||||
if runLayout.Enabled {
|
||||
stdoutPath = filepath.Join(runLayout.LogsDir, "seriatim.stdout.log")
|
||||
stderrPath = filepath.Join(runLayout.LogsDir, "seriatim.stderr.log")
|
||||
genCfgPath = filepath.Join(runLayout.ConfigDir, "seriatim.generated.yml")
|
||||
}
|
||||
|
||||
normalizedInputs, normalizeLogs, normalizeConfigs, normalizeMeta, err := normalizeMergeInputs(ctx, env, inputs, paths)
|
||||
normalizedInputs, normalizeLogs, normalizeConfigs, normalizeMeta, err := normalizeMergeInputs(ctx, env, inputs, paths, runLayout)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
@@ -130,25 +150,35 @@ func (mergeStage) Run(ctx context.Context, env *Env, m *manifest.Manifest) (*Sta
|
||||
}
|
||||
}
|
||||
|
||||
outputs := []artifacts.Ref{{
|
||||
Kind: "transcript_merged",
|
||||
Category: "transcripts",
|
||||
SessionID: sessionID,
|
||||
AbsolutePath: finalMergedPath,
|
||||
}}
|
||||
promotedMerged, err := promoteRunLocalOutput(env.ArtifactStore, finalMergedPath, canonicalMergedPath, artifacts.Ref{
|
||||
Kind: "transcript_merged",
|
||||
Category: "transcripts",
|
||||
SessionID: sessionID,
|
||||
})
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("merge: promote merged transcript: %w", err)
|
||||
}
|
||||
outputs := []artifacts.Ref{promotedMerged}
|
||||
if reportEnabled {
|
||||
outputs = append(outputs, artifacts.Ref{
|
||||
Kind: "seriatim_report",
|
||||
Category: "artifacts",
|
||||
SessionID: sessionID,
|
||||
AbsolutePath: finalReportPath,
|
||||
promotedReport, err := promoteRunLocalOutput(env.ArtifactStore, finalReportPath, canonicalReportPath, artifacts.Ref{
|
||||
Kind: "seriatim_report",
|
||||
Category: "artifacts",
|
||||
SessionID: sessionID,
|
||||
})
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("merge: promote report: %w", err)
|
||||
}
|
||||
outputs = append(outputs, promotedReport)
|
||||
}
|
||||
|
||||
coalesceGap := any(nil)
|
||||
if env.Config.Pipeline.Seriatim.CoalesceGap != nil {
|
||||
coalesceGap = *env.Config.Pipeline.Seriatim.CoalesceGap
|
||||
}
|
||||
reportCanonicalPath := ""
|
||||
if reportEnabled {
|
||||
reportCanonicalPath = canonicalReportPath
|
||||
}
|
||||
|
||||
meta := map[string]any{
|
||||
"stage": "merge",
|
||||
@@ -160,8 +190,10 @@ func (mergeStage) Run(ctx context.Context, env *Env, m *manifest.Manifest) (*Sta
|
||||
"output_schema": env.Config.Pipeline.Seriatim.OutputSchema,
|
||||
"coalesce_gap": coalesceGap,
|
||||
"report_enabled": reportEnabled,
|
||||
"output_path": finalMergedPath,
|
||||
"report_path": finalReportPath,
|
||||
"run_output_path": finalMergedPath,
|
||||
"output_path": canonicalMergedPath,
|
||||
"run_report_path": finalReportPath,
|
||||
"report_path": reportCanonicalPath,
|
||||
"timeout": env.Config.Pipeline.Seriatim.Timeout,
|
||||
"binary": env.Config.Pipeline.Seriatim.Binary,
|
||||
"adapter_duration_ms": res.Duration.Milliseconds(),
|
||||
@@ -201,11 +233,24 @@ type normalizeMergeInputMeta struct {
|
||||
AdapterOutputPath string `json:"adapter_output_path,omitempty"`
|
||||
}
|
||||
|
||||
func normalizeMergeInputs(ctx context.Context, env *Env, rawInputs []string, paths artifacts.SessionPaths) ([]string, []string, []string, []normalizeMergeInputMeta, error) {
|
||||
func normalizeMergeInputs(
|
||||
ctx context.Context,
|
||||
env *Env,
|
||||
rawInputs []string,
|
||||
paths artifacts.SessionPaths,
|
||||
runLayout runStageLayout,
|
||||
) ([]string, []string, []string, []normalizeMergeInputMeta, error) {
|
||||
normalizedInputs := make([]string, 0, len(rawInputs))
|
||||
logs := make([]string, 0, len(rawInputs)*2)
|
||||
configs := make([]string, 0, len(rawInputs))
|
||||
meta := make([]normalizeMergeInputMeta, 0, len(rawInputs))
|
||||
normalizedDir := filepath.Join(paths.TranscriptsRawDir, "normalized")
|
||||
if runLayout.Enabled {
|
||||
normalizedDir = filepath.Join(runLayout.ScratchDir, "normalized")
|
||||
}
|
||||
if err := os.MkdirAll(normalizedDir, 0o755); err != nil {
|
||||
return nil, nil, nil, nil, fmt.Errorf("merge: ensure normalized transcripts directory %q: %w", normalizedDir, err)
|
||||
}
|
||||
|
||||
var timeout time.Duration
|
||||
timeoutRaw := strings.TrimSpace(env.Config.Pipeline.Seriatim.Timeout)
|
||||
@@ -218,10 +263,15 @@ func normalizeMergeInputs(ctx context.Context, env *Env, rawInputs []string, pat
|
||||
}
|
||||
for _, input := range rawInputs {
|
||||
base := strings.TrimSuffix(filepath.Base(input), filepath.Ext(input))
|
||||
outPath := filepath.Join(paths.TranscriptsRawDir, "normalized", base+".normalized.json")
|
||||
outPath := filepath.Join(normalizedDir, base+".normalized.json")
|
||||
stdoutPath := filepath.Join(paths.LogsDir, "seriatim.normalize."+base+".stdout.log")
|
||||
stderrPath := filepath.Join(paths.LogsDir, "seriatim.normalize."+base+".stderr.log")
|
||||
cfgPath := filepath.Join(paths.ConfigDir, "seriatim.normalize."+base+".generated.yml")
|
||||
if runLayout.Enabled {
|
||||
stdoutPath = filepath.Join(runLayout.LogsDir, "seriatim.normalize."+base+".stdout.log")
|
||||
stderrPath = filepath.Join(runLayout.LogsDir, "seriatim.normalize."+base+".stderr.log")
|
||||
cfgPath = filepath.Join(runLayout.ConfigDir, "seriatim.normalize."+base+".generated.yml")
|
||||
}
|
||||
|
||||
req := seriatim.NormalizeRequest{
|
||||
Binary: env.Config.Pipeline.Seriatim.Binary,
|
||||
|
||||
@@ -2,6 +2,7 @@ package stage
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
@@ -14,9 +15,25 @@ import (
|
||||
"gitea.maximumdirect.net/eric/narratio/internal/manifest"
|
||||
)
|
||||
|
||||
type normalizeDirAssertingRunner struct {
|
||||
*seriatim.FakeRunner
|
||||
ExpectedDir string
|
||||
}
|
||||
|
||||
func (r *normalizeDirAssertingRunner) Normalize(ctx context.Context, req seriatim.NormalizeRequest) (seriatim.NormalizeResult, error) {
|
||||
info, err := os.Stat(r.ExpectedDir)
|
||||
if err != nil {
|
||||
return seriatim.NormalizeResult{}, fmt.Errorf("normalized directory check failed for %q: %w", r.ExpectedDir, err)
|
||||
}
|
||||
if !info.IsDir() {
|
||||
return seriatim.NormalizeResult{}, fmt.Errorf("normalized path %q exists but is not a directory", r.ExpectedDir)
|
||||
}
|
||||
return r.FakeRunner.Normalize(ctx, req)
|
||||
}
|
||||
|
||||
func TestMergeStageMergesRawTranscriptsAndRecordsMetadata(t *testing.T) {
|
||||
env, m := setupMergeEnv(t)
|
||||
paths := env.ArtifactStore.SessionPaths(m.SessionID)
|
||||
paths := sessionPathsForEnv(env, m.SessionID)
|
||||
|
||||
inA := filepath.Join(paths.TranscriptsRawDir, "alice.json")
|
||||
inB := filepath.Join(paths.TranscriptsRawDir, "bob.json")
|
||||
@@ -105,7 +122,7 @@ func TestMergeStageMergesRawTranscriptsAndRecordsMetadata(t *testing.T) {
|
||||
|
||||
func TestMergeStageFailsWhenNoRawTranscripts(t *testing.T) {
|
||||
env, m := setupMergeEnv(t)
|
||||
paths := env.ArtifactStore.SessionPaths(m.SessionID)
|
||||
paths := sessionPathsForEnv(env, m.SessionID)
|
||||
writeFile(t, filepath.Join(paths.InputsDir, "speakers.yml"), "match: []\n")
|
||||
writeFile(t, filepath.Join(paths.InputsDir, "autocorrect.yml"), "rules: []\n")
|
||||
env.Seriatim = &seriatim.FakeRunner{}
|
||||
@@ -121,7 +138,7 @@ func TestMergeStageFailsWhenNoRawTranscripts(t *testing.T) {
|
||||
|
||||
func TestMergeStageFailsOnInvalidInputJSON(t *testing.T) {
|
||||
env, m := setupMergeEnv(t)
|
||||
paths := env.ArtifactStore.SessionPaths(m.SessionID)
|
||||
paths := sessionPathsForEnv(env, m.SessionID)
|
||||
writeFile(t, filepath.Join(paths.TranscriptsRawDir, "alice.json"), "not-json")
|
||||
writeFile(t, filepath.Join(paths.InputsDir, "speakers.yml"), "match: []\n")
|
||||
writeFile(t, filepath.Join(paths.InputsDir, "autocorrect.yml"), "rules: []\n")
|
||||
@@ -138,7 +155,7 @@ func TestMergeStageFailsOnInvalidInputJSON(t *testing.T) {
|
||||
|
||||
func TestMergeStageFailsWhenAdapterFails(t *testing.T) {
|
||||
env, m := setupMergeEnv(t)
|
||||
paths := env.ArtifactStore.SessionPaths(m.SessionID)
|
||||
paths := sessionPathsForEnv(env, m.SessionID)
|
||||
writeFile(t, filepath.Join(paths.TranscriptsRawDir, "alice.json"), `{"segments":[]}`)
|
||||
writeFile(t, filepath.Join(paths.InputsDir, "speakers.yml"), "match: []\n")
|
||||
writeFile(t, filepath.Join(paths.InputsDir, "autocorrect.yml"), "rules: []\n")
|
||||
@@ -155,7 +172,7 @@ func TestMergeStageFailsWhenAdapterFails(t *testing.T) {
|
||||
|
||||
func TestMergeStageFallsBackToRawDirectoryWhenTranscribeOutputsMissing(t *testing.T) {
|
||||
env, m := setupMergeEnv(t)
|
||||
paths := env.ArtifactStore.SessionPaths(m.SessionID)
|
||||
paths := sessionPathsForEnv(env, m.SessionID)
|
||||
writeFile(t, filepath.Join(paths.TranscriptsRawDir, "alice.json"), `{"segments":[]}`)
|
||||
writeFile(t, filepath.Join(paths.InputsDir, "speakers.yml"), "match: []\n")
|
||||
writeFile(t, filepath.Join(paths.InputsDir, "autocorrect.yml"), "rules: []\n")
|
||||
@@ -176,7 +193,7 @@ func TestMergeStageFallsBackToRawDirectoryWhenTranscribeOutputsMissing(t *testin
|
||||
|
||||
func TestMergeStageResolvesWorkspaceQualifiedManifestOutputsWithoutDuplication(t *testing.T) {
|
||||
env, m := setupMergeEnvWithRelativeWorkspaceRoot(t)
|
||||
paths := env.ArtifactStore.SessionPaths(m.SessionID)
|
||||
paths := sessionPathsForEnv(env, m.SessionID)
|
||||
|
||||
rawPath := filepath.Join(paths.TranscriptsRawDir, "alice.json")
|
||||
writeFile(t, rawPath, `{"segments":[]}`)
|
||||
@@ -205,9 +222,33 @@ func TestMergeStageResolvesWorkspaceQualifiedManifestOutputsWithoutDuplication(t
|
||||
}
|
||||
}
|
||||
|
||||
func TestMergeStageCreatesNormalizedRawDirectoryBeforeNormalize(t *testing.T) {
|
||||
env, m := setupMergeEnv(t)
|
||||
paths := sessionPathsForEnv(env, m.SessionID)
|
||||
|
||||
rawPath := filepath.Join(paths.TranscriptsRawDir, "alice.json")
|
||||
writeFile(t, rawPath, `{"segments":[]}`)
|
||||
writeFile(t, filepath.Join(paths.InputsDir, "speakers.yml"), "match: []\n")
|
||||
writeFile(t, filepath.Join(paths.InputsDir, "autocorrect.yml"), "rules: []\n")
|
||||
|
||||
normalizedDir := filepath.Join(paths.TranscriptsRawDir, "normalized")
|
||||
if err := os.RemoveAll(normalizedDir); err != nil {
|
||||
t.Fatalf("remove normalized dir: %v", err)
|
||||
}
|
||||
|
||||
env.Seriatim = &normalizeDirAssertingRunner{
|
||||
FakeRunner: &seriatim.FakeRunner{},
|
||||
ExpectedDir: normalizedDir,
|
||||
}
|
||||
|
||||
if _, err := (mergeStage{}).Run(context.Background(), env, m); err != nil {
|
||||
t.Fatalf("merge.Run() error = %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestMergeStageFailsWhenNormalizeAdapterFails(t *testing.T) {
|
||||
env, m := setupMergeEnv(t)
|
||||
paths := env.ArtifactStore.SessionPaths(m.SessionID)
|
||||
paths := sessionPathsForEnv(env, m.SessionID)
|
||||
in := filepath.Join(paths.TranscriptsRawDir, "alice.json")
|
||||
writeFile(t, in, `{"segments":[]}`)
|
||||
writeFile(t, filepath.Join(paths.InputsDir, "speakers.yml"), "match: []\n")
|
||||
@@ -225,7 +266,7 @@ func TestMergeStageFailsWhenNormalizeAdapterFails(t *testing.T) {
|
||||
|
||||
func TestMergeStageFailsWhenNormalizedOutputInvalid(t *testing.T) {
|
||||
env, m := setupMergeEnv(t)
|
||||
paths := env.ArtifactStore.SessionPaths(m.SessionID)
|
||||
paths := sessionPathsForEnv(env, m.SessionID)
|
||||
in := filepath.Join(paths.TranscriptsRawDir, "alice.json")
|
||||
writeFile(t, in, `{"segments":[]}`)
|
||||
writeFile(t, filepath.Join(paths.InputsDir, "speakers.yml"), "match: []\n")
|
||||
@@ -248,9 +289,42 @@ func TestMergeStageFailsWhenNormalizedOutputInvalid(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
func TestMergeStageUsesRunLocalPathsAndPromotesCanonical(t *testing.T) {
|
||||
env, m := setupMergeEnv(t)
|
||||
env.Config.Session.Campaign = "sample-campaign"
|
||||
m.Campaign = "sample-campaign"
|
||||
m.RunID = "20260518T010203Z-abcdef12"
|
||||
paths := sessionPathsForEnv(env, m.SessionID)
|
||||
|
||||
in := filepath.Join(paths.TranscriptsRawDir, "alice.json")
|
||||
writeFile(t, in, `{"segments":[]}`)
|
||||
writeFile(t, filepath.Join(paths.InputsDir, "speakers.yml"), "match: []\n")
|
||||
writeFile(t, filepath.Join(paths.InputsDir, "autocorrect.yml"), "rules: []\n")
|
||||
|
||||
fake := &seriatim.FakeRunner{}
|
||||
env.Seriatim = fake
|
||||
result, err := (mergeStage{}).Run(context.Background(), env, m)
|
||||
if err != nil {
|
||||
t.Fatalf("merge.Run() error = %v", err)
|
||||
}
|
||||
if len(fake.Requests) != 1 {
|
||||
t.Fatalf("requests = %d, want 1", len(fake.Requests))
|
||||
}
|
||||
req := fake.Requests[0]
|
||||
if !strings.Contains(req.OutputMergedTranscriptPath, filepath.Join("runs", m.RunID, "merge", "outputs")) {
|
||||
t.Fatalf("run output path = %q, want run-local path", req.OutputMergedTranscriptPath)
|
||||
}
|
||||
if len(result.Outputs) == 0 {
|
||||
t.Fatalf("outputs = %#v, want promoted outputs", result.Outputs)
|
||||
}
|
||||
if strings.Contains(result.Outputs[0].AbsolutePath, string(filepath.Separator)+"runs"+string(filepath.Separator)) {
|
||||
t.Fatalf("promoted output path = %q, want canonical session path", result.Outputs[0].AbsolutePath)
|
||||
}
|
||||
}
|
||||
|
||||
func TestMergeStageResolvesSessionRelativeManifestOutputs(t *testing.T) {
|
||||
env, m := setupMergeEnv(t)
|
||||
paths := env.ArtifactStore.SessionPaths(m.SessionID)
|
||||
paths := sessionPathsForEnv(env, m.SessionID)
|
||||
|
||||
rawPath := filepath.Join(paths.TranscriptsRawDir, "alice.json")
|
||||
writeFile(t, rawPath, `{"segments":[]}`)
|
||||
@@ -332,7 +406,7 @@ func setupMergeEnvWithWorkspace(t *testing.T, workspace string) (*Env, *manifest
|
||||
}
|
||||
|
||||
store := artifacts.NewLocalStore(workspace)
|
||||
if _, err := store.EnsureLayout("2026-05-03"); err != nil {
|
||||
if _, err := store.EnsureLayoutFor("sample-campaign", "2026-05-03"); err != nil {
|
||||
t.Fatalf("EnsureLayout() error = %v", err)
|
||||
}
|
||||
return &Env{
|
||||
|
||||
@@ -54,7 +54,11 @@ func (normalizeStage) Run(ctx context.Context, env *Env, m *manifest.Manifest) (
|
||||
return nil, fmt.Errorf("normalize: session id is required")
|
||||
}
|
||||
|
||||
paths := env.ArtifactStore.SessionPaths(sessionID)
|
||||
paths := sessionPathsForEnv(env, sessionID)
|
||||
runLayout, err := resolveRunStageLayout(env, m, paths, sessionID, "normalize")
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("normalize: resolve run-stage layout: %w", err)
|
||||
}
|
||||
processedPath, processedSource, err := discoverProcessedTranscript(m, paths)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("normalize: resolve processed transcript: %w", err)
|
||||
@@ -67,18 +71,34 @@ func (normalizeStage) Run(ctx context.Context, env *Env, m *manifest.Manifest) (
|
||||
}
|
||||
|
||||
normalizeCfg := normalizeConfigOrDefault(env.Config.Pipeline.Normalize)
|
||||
normalizedPath, err := resolveScriptoriumOutputPath(paths, normalizeCfg.OutputPath)
|
||||
canonicalNormalizedPath, err := resolveScriptoriumOutputPath(paths, normalizeCfg.OutputPath)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("normalize: resolve normalized output path: %w", err)
|
||||
}
|
||||
normalizedPath, err := runLocalPathForCanonical(runLayout, paths, canonicalNormalizedPath)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("normalize: resolve run-local normalized output path: %w", err)
|
||||
}
|
||||
reportEnabled := normalizeCfg.Report != nil && *normalizeCfg.Report
|
||||
reportPath := ""
|
||||
canonicalReportPath := filepath.Join(paths.ArtifactsDir, "seriatim.normalize.report.json")
|
||||
if reportEnabled {
|
||||
reportPath = filepath.Join(paths.ArtifactsDir, "seriatim.normalize.report.json")
|
||||
reportPath = canonicalReportPath
|
||||
if runLayout.Enabled {
|
||||
reportPath, err = runLocalPathForCanonical(runLayout, paths, canonicalReportPath)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("normalize: resolve run-local report path: %w", err)
|
||||
}
|
||||
}
|
||||
}
|
||||
stdoutPath := filepath.Join(paths.LogsDir, "seriatim.normalize.stdout.log")
|
||||
stderrPath := filepath.Join(paths.LogsDir, "seriatim.normalize.stderr.log")
|
||||
generatedConfigPath := filepath.Join(paths.ConfigDir, "seriatim.normalize.generated.yml")
|
||||
if runLayout.Enabled {
|
||||
stdoutPath = filepath.Join(runLayout.LogsDir, "seriatim.normalize.stdout.log")
|
||||
stderrPath = filepath.Join(runLayout.LogsDir, "seriatim.normalize.stderr.log")
|
||||
generatedConfigPath = filepath.Join(runLayout.ConfigDir, "seriatim.normalize.generated.yml")
|
||||
}
|
||||
timeout, err := resolveTrimSeriatimTimeout(env.Config.Pipeline.Seriatim.Timeout)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("normalize: resolve seriatim timeout: %w", err)
|
||||
@@ -113,44 +133,56 @@ func (normalizeStage) Run(ctx context.Context, env *Env, m *manifest.Manifest) (
|
||||
}
|
||||
}
|
||||
|
||||
outputs := []artifacts.Ref{{
|
||||
Kind: "transcript_normalized",
|
||||
Category: "transcripts",
|
||||
SessionID: sessionID,
|
||||
AbsolutePath: finalNormalizedPath,
|
||||
}}
|
||||
promotedNormalized, err := promoteRunLocalOutput(env.ArtifactStore, finalNormalizedPath, canonicalNormalizedPath, artifacts.Ref{
|
||||
Kind: "transcript_normalized",
|
||||
Category: "transcripts",
|
||||
SessionID: sessionID,
|
||||
})
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("normalize: promote normalized transcript: %w", err)
|
||||
}
|
||||
outputs := []artifacts.Ref{promotedNormalized}
|
||||
if reportEnabled {
|
||||
outputs = append(outputs, artifacts.Ref{
|
||||
Kind: "seriatim_normalize_report",
|
||||
Category: "artifacts",
|
||||
SessionID: sessionID,
|
||||
AbsolutePath: finalReportPath,
|
||||
promotedReport, err := promoteRunLocalOutput(env.ArtifactStore, finalReportPath, canonicalReportPath, artifacts.Ref{
|
||||
Kind: "seriatim_normalize_report",
|
||||
Category: "artifacts",
|
||||
SessionID: sessionID,
|
||||
})
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("normalize: promote report: %w", err)
|
||||
}
|
||||
outputs = append(outputs, promotedReport)
|
||||
}
|
||||
reportCanonicalPath := ""
|
||||
if reportEnabled {
|
||||
reportCanonicalPath = canonicalReportPath
|
||||
}
|
||||
|
||||
meta := map[string]any{
|
||||
"stage": "normalize",
|
||||
"processed_transcript_path": processedPath,
|
||||
"processed_transcript_source": processedSource,
|
||||
"normalized_transcript_path": finalNormalizedPath,
|
||||
"normalized_transcript_source": "stage.normalize.output",
|
||||
"output_schema": normalizeCfg.OutputSchema,
|
||||
"report_enabled": reportEnabled,
|
||||
"report_path": finalReportPath,
|
||||
"timeout": env.Config.Pipeline.Seriatim.Timeout,
|
||||
"binary": env.Config.Pipeline.Seriatim.Binary,
|
||||
"stdout_log_path": stdoutPath,
|
||||
"stderr_log_path": stderrPath,
|
||||
"generated_config_path": generatedConfigPath,
|
||||
"adapter_duration_ms": res.Duration.Milliseconds(),
|
||||
"adapter_exit_code": res.ExitCode,
|
||||
"adapter_invoked_binary": res.InvokedBinary,
|
||||
"adapter_output_schema": res.OutputSchema,
|
||||
"adapter_output_path": res.OutputNormalizedPath,
|
||||
"adapter_report_path": res.ReportPath,
|
||||
"adapter_generated_config": res.GeneratedConfigPath,
|
||||
"adapter_stdout_log_path": res.StdoutLogPath,
|
||||
"adapter_stderr_log_path": res.StderrLogPath,
|
||||
"stage": "normalize",
|
||||
"processed_transcript_path": processedPath,
|
||||
"processed_transcript_source": processedSource,
|
||||
"run_normalized_transcript_path": finalNormalizedPath,
|
||||
"normalized_transcript_path": canonicalNormalizedPath,
|
||||
"normalized_transcript_source": "stage.normalize.output",
|
||||
"output_schema": normalizeCfg.OutputSchema,
|
||||
"report_enabled": reportEnabled,
|
||||
"run_report_path": finalReportPath,
|
||||
"report_path": reportCanonicalPath,
|
||||
"timeout": env.Config.Pipeline.Seriatim.Timeout,
|
||||
"binary": env.Config.Pipeline.Seriatim.Binary,
|
||||
"stdout_log_path": stdoutPath,
|
||||
"stderr_log_path": stderrPath,
|
||||
"generated_config_path": generatedConfigPath,
|
||||
"adapter_duration_ms": res.Duration.Milliseconds(),
|
||||
"adapter_exit_code": res.ExitCode,
|
||||
"adapter_invoked_binary": res.InvokedBinary,
|
||||
"adapter_output_schema": res.OutputSchema,
|
||||
"adapter_output_path": res.OutputNormalizedPath,
|
||||
"adapter_report_path": res.ReportPath,
|
||||
"adapter_generated_config": res.GeneratedConfigPath,
|
||||
"adapter_stdout_log_path": res.StdoutLogPath,
|
||||
"adapter_stderr_log_path": res.StderrLogPath,
|
||||
}
|
||||
if res.Metadata != nil {
|
||||
meta["adapter_metadata"] = res.Metadata
|
||||
|
||||
@@ -16,7 +16,7 @@ import (
|
||||
|
||||
func TestNormalizeStageConsumesProcessedTranscriptFromManifest(t *testing.T) {
|
||||
env, m, ser := setupNormalizeEnv(t)
|
||||
paths := env.ArtifactStore.SessionPaths(m.SessionID)
|
||||
paths := sessionPathsForEnv(env, m.SessionID)
|
||||
manifestProcessed := filepath.Join(paths.ArtifactsDir, "processed.from-manifest.json")
|
||||
writeFile(t, manifestProcessed, `{"segments":[{"id":10}]}`)
|
||||
writeFile(t, filepath.Join(paths.TranscriptsDir, "processed.json"), `{"segments":[{"id":99}]}`)
|
||||
@@ -43,7 +43,7 @@ func TestNormalizeStageConsumesProcessedTranscriptFromManifest(t *testing.T) {
|
||||
|
||||
func TestNormalizeStageFallsBackToProcessedTranscriptPath(t *testing.T) {
|
||||
env, m, ser := setupNormalizeEnv(t)
|
||||
paths := env.ArtifactStore.SessionPaths(m.SessionID)
|
||||
paths := sessionPathsForEnv(env, m.SessionID)
|
||||
fallback := filepath.Join(paths.TranscriptsDir, "processed.json")
|
||||
writeFile(t, fallback, `{"segments":[{"id":1}]}`)
|
||||
|
||||
@@ -72,7 +72,7 @@ func TestNormalizeStageFailsWhenProcessedTranscriptMissing(t *testing.T) {
|
||||
|
||||
func TestNormalizeStageFailsWhenProcessedTranscriptInvalidJSON(t *testing.T) {
|
||||
env, m, _ := setupNormalizeEnv(t)
|
||||
paths := env.ArtifactStore.SessionPaths(m.SessionID)
|
||||
paths := sessionPathsForEnv(env, m.SessionID)
|
||||
writeFile(t, filepath.Join(paths.TranscriptsDir, "processed.json"), "not-json")
|
||||
|
||||
_, err := (normalizeStage{}).Run(context.Background(), env, m)
|
||||
@@ -86,7 +86,7 @@ func TestNormalizeStageFailsWhenProcessedTranscriptInvalidJSON(t *testing.T) {
|
||||
|
||||
func TestNormalizeStageFailsWhenProcessedTranscriptMissingSegments(t *testing.T) {
|
||||
env, m, _ := setupNormalizeEnv(t)
|
||||
paths := env.ArtifactStore.SessionPaths(m.SessionID)
|
||||
paths := sessionPathsForEnv(env, m.SessionID)
|
||||
writeFile(t, filepath.Join(paths.TranscriptsDir, "processed.json"), `{"schema":"audita.processed.v1"}`)
|
||||
|
||||
_, err := (normalizeStage{}).Run(context.Background(), env, m)
|
||||
@@ -100,7 +100,7 @@ func TestNormalizeStageFailsWhenProcessedTranscriptMissingSegments(t *testing.T)
|
||||
|
||||
func TestNormalizeStagePassesConfiguredOutputSchemaToAdapter(t *testing.T) {
|
||||
env, m, ser := setupNormalizeEnv(t)
|
||||
paths := env.ArtifactStore.SessionPaths(m.SessionID)
|
||||
paths := sessionPathsForEnv(env, m.SessionID)
|
||||
writeFile(t, filepath.Join(paths.TranscriptsDir, "processed.json"), `{"segments":[{"id":1}]}`)
|
||||
env.Config.Pipeline.Normalize.OutputSchema = "seriatim-full"
|
||||
|
||||
@@ -121,7 +121,7 @@ func TestNormalizeStagePassesConfiguredOutputSchemaToAdapter(t *testing.T) {
|
||||
|
||||
func TestNormalizeStageRecordsNormalizedTranscriptOutputKind(t *testing.T) {
|
||||
env, m, _ := setupNormalizeEnv(t)
|
||||
paths := env.ArtifactStore.SessionPaths(m.SessionID)
|
||||
paths := sessionPathsForEnv(env, m.SessionID)
|
||||
writeFile(t, filepath.Join(paths.TranscriptsDir, "processed.json"), `{"segments":[{"id":1}]}`)
|
||||
|
||||
result, err := (normalizeStage{}).Run(context.Background(), env, m)
|
||||
@@ -138,7 +138,7 @@ func TestNormalizeStageRecordsNormalizedTranscriptOutputKind(t *testing.T) {
|
||||
|
||||
func TestNormalizeStageRecordsReportLogAndGeneratedConfigRefs(t *testing.T) {
|
||||
env, m, _ := setupNormalizeEnv(t)
|
||||
paths := env.ArtifactStore.SessionPaths(m.SessionID)
|
||||
paths := sessionPathsForEnv(env, m.SessionID)
|
||||
writeFile(t, filepath.Join(paths.TranscriptsDir, "processed.json"), `{"segments":[{"id":1}]}`)
|
||||
report := true
|
||||
env.Config.Pipeline.Normalize.Report = &report
|
||||
@@ -164,7 +164,7 @@ func TestNormalizeStageRecordsReportLogAndGeneratedConfigRefs(t *testing.T) {
|
||||
|
||||
func TestNormalizeStageFailsWhenAdapterReturnsError(t *testing.T) {
|
||||
env, m, ser := setupNormalizeEnv(t)
|
||||
paths := env.ArtifactStore.SessionPaths(m.SessionID)
|
||||
paths := sessionPathsForEnv(env, m.SessionID)
|
||||
writeFile(t, filepath.Join(paths.TranscriptsDir, "processed.json"), `{"segments":[{"id":1}]}`)
|
||||
ser.NormalizeErr = errors.New("normalize failed")
|
||||
|
||||
@@ -179,7 +179,7 @@ func TestNormalizeStageFailsWhenAdapterReturnsError(t *testing.T) {
|
||||
|
||||
func TestNormalizeStageFailsWhenNormalizedOutputInvalid(t *testing.T) {
|
||||
env, m, ser := setupNormalizeEnv(t)
|
||||
paths := env.ArtifactStore.SessionPaths(m.SessionID)
|
||||
paths := sessionPathsForEnv(env, m.SessionID)
|
||||
writeFile(t, filepath.Join(paths.TranscriptsDir, "processed.json"), `{"segments":[{"id":1}]}`)
|
||||
badOutput := filepath.Join(paths.TranscriptsDir, "normalized.bad.json")
|
||||
writeFile(t, badOutput, "not-json")
|
||||
@@ -196,7 +196,7 @@ func TestNormalizeStageFailsWhenNormalizedOutputInvalid(t *testing.T) {
|
||||
|
||||
func TestNormalizeStageReportEnabledFailsWhenReportMissing(t *testing.T) {
|
||||
env, m, ser := setupNormalizeEnv(t)
|
||||
paths := env.ArtifactStore.SessionPaths(m.SessionID)
|
||||
paths := sessionPathsForEnv(env, m.SessionID)
|
||||
writeFile(t, filepath.Join(paths.TranscriptsDir, "processed.json"), `{"segments":[{"id":1}]}`)
|
||||
report := true
|
||||
env.Config.Pipeline.Normalize.Report = &report
|
||||
@@ -211,6 +211,33 @@ func TestNormalizeStageReportEnabledFailsWhenReportMissing(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
func TestNormalizeStageUsesRunLocalPathsAndPromotesCanonical(t *testing.T) {
|
||||
env, m, ser := setupNormalizeEnv(t)
|
||||
env.Config.Session.Campaign = "sample-campaign"
|
||||
m.Campaign = "sample-campaign"
|
||||
m.RunID = "20260518T010203Z-abcdef12"
|
||||
paths := sessionPathsForEnv(env, m.SessionID)
|
||||
writeFile(t, filepath.Join(paths.TranscriptsDir, "processed.json"), `{"segments":[{"id":1}]}`)
|
||||
|
||||
result, err := (normalizeStage{}).Run(context.Background(), env, m)
|
||||
if err != nil {
|
||||
t.Fatalf("normalize.Run() error = %v", err)
|
||||
}
|
||||
if len(ser.NormalizeRequests) != 1 {
|
||||
t.Fatalf("normalize requests = %d, want 1", len(ser.NormalizeRequests))
|
||||
}
|
||||
req := ser.NormalizeRequests[0]
|
||||
if !strings.Contains(req.OutputNormalizedPath, filepath.Join("runs", m.RunID, "normalize", "outputs")) {
|
||||
t.Fatalf("run output path = %q, want run-local path", req.OutputNormalizedPath)
|
||||
}
|
||||
if len(result.Outputs) == 0 {
|
||||
t.Fatalf("outputs = %#v, want promoted outputs", result.Outputs)
|
||||
}
|
||||
if strings.Contains(result.Outputs[0].AbsolutePath, string(filepath.Separator)+"runs"+string(filepath.Separator)) {
|
||||
t.Fatalf("promoted output path = %q, want canonical session path", result.Outputs[0].AbsolutePath)
|
||||
}
|
||||
}
|
||||
|
||||
func setupNormalizeEnv(t *testing.T) (*Env, *manifest.Manifest, *seriatim.FakeRunner) {
|
||||
t.Helper()
|
||||
workspace := t.TempDir()
|
||||
@@ -243,7 +270,7 @@ func setupNormalizeEnv(t *testing.T) (*Env, *manifest.Manifest, *seriatim.FakeRu
|
||||
}
|
||||
|
||||
store := artifacts.NewLocalStore(workspace)
|
||||
if _, err := store.EnsureLayout("2026-05-03"); err != nil {
|
||||
if _, err := store.EnsureLayoutFor("sample-campaign", "2026-05-03"); err != nil {
|
||||
t.Fatalf("EnsureLayout() error = %v", err)
|
||||
}
|
||||
|
||||
|
||||
@@ -64,6 +64,7 @@ func TestStagesReturnExpectedMetadata(t *testing.T) {
|
||||
},
|
||||
Session: &config.SessionConfig{
|
||||
SessionID: "2026-05-03",
|
||||
Campaign: "sample-campaign",
|
||||
Inputs: config.SessionInputsConfig{
|
||||
AudioDir: "./audio",
|
||||
SpeakersFile: "./speakers.yml",
|
||||
@@ -88,7 +89,7 @@ func TestStagesReturnExpectedMetadata(t *testing.T) {
|
||||
}
|
||||
m.RunID = "20260516T000000Z-abcdef12"
|
||||
m.Campaign = "sample-campaign"
|
||||
m.LocalWorkDir = filepath.Join(root, "work", "sample-campaign", "2026-05-03", m.RunID)
|
||||
m.LocalWorkDir = artifacts.SessionRunRootForCampaign(root, "sample-campaign", "2026-05-03", m.RunID)
|
||||
m.S3RunPrefix = "dnd/campaigns/sample-campaign/sessions/2026-05-03/runs/" + m.RunID + "/"
|
||||
m.S3Bucket = "my-dnd-archive"
|
||||
if err := os.MkdirAll(filepath.Join(m.LocalWorkDir, "inputs"), 0o755); err != nil {
|
||||
@@ -224,7 +225,7 @@ func TestPlaceholderAdapterErrorPropagation(t *testing.T) {
|
||||
|
||||
root := t.TempDir()
|
||||
store := artifacts.NewLocalStore(root)
|
||||
_, err := store.EnsureLayout("2026-05-03")
|
||||
_, err := store.EnsureLayoutFor("sample-campaign", "2026-05-03")
|
||||
if err != nil {
|
||||
t.Fatalf("EnsureLayout() error = %v", err)
|
||||
}
|
||||
|
||||
@@ -55,7 +55,11 @@ func (polishStage) Run(ctx context.Context, env *Env, m *manifest.Manifest) (*St
|
||||
return nil, fmt.Errorf("polish: session id is required")
|
||||
}
|
||||
|
||||
paths := env.ArtifactStore.SessionPaths(sessionID)
|
||||
paths := sessionPathsForEnv(env, sessionID)
|
||||
runLayout, err := resolveRunStageLayout(env, m, paths, sessionID, "polish")
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("polish: resolve run-stage layout: %w", err)
|
||||
}
|
||||
mergedPath, source, err := discoverMergedTranscript(m, paths)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("polish: resolve merged transcript: %w", err)
|
||||
@@ -72,12 +76,29 @@ func (polishStage) Run(ctx context.Context, env *Env, m *manifest.Manifest) (*St
|
||||
return nil, fmt.Errorf("polish: %w", err)
|
||||
}
|
||||
|
||||
processedPath := filepath.Join(paths.TranscriptsDir, "processed.json")
|
||||
reportPath := filepath.Join(paths.ArtifactsDir, "audita.report.json")
|
||||
canonicalProcessedPath := filepath.Join(paths.TranscriptsDir, "processed.json")
|
||||
processedPath, err := runLocalPathForCanonical(runLayout, paths, canonicalProcessedPath)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("polish: resolve run-local processed transcript path: %w", err)
|
||||
}
|
||||
canonicalReportPath := filepath.Join(paths.ArtifactsDir, "audita.report.json")
|
||||
reportPath := canonicalReportPath
|
||||
if runLayout.Enabled {
|
||||
reportPath, err = runLocalPathForCanonical(runLayout, paths, canonicalReportPath)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("polish: resolve run-local report path: %w", err)
|
||||
}
|
||||
}
|
||||
workDir := filepath.Join(paths.ArtifactsDir, "audita-work")
|
||||
stdoutPath := filepath.Join(paths.LogsDir, "audita.stdout.log")
|
||||
stderrPath := filepath.Join(paths.LogsDir, "audita.stderr.log")
|
||||
generatedConfigPath := filepath.Join(paths.ConfigDir, "audita.generated.yml")
|
||||
if runLayout.Enabled {
|
||||
workDir = filepath.Join(runLayout.ScratchDir, "audita-work")
|
||||
stdoutPath = filepath.Join(runLayout.LogsDir, "audita.stdout.log")
|
||||
stderrPath = filepath.Join(runLayout.LogsDir, "audita.stderr.log")
|
||||
generatedConfigPath = filepath.Join(runLayout.ConfigDir, "audita.generated.yml")
|
||||
}
|
||||
|
||||
reportEnabled := env.Config.Pipeline.Audita.Report != nil && *env.Config.Pipeline.Audita.Report
|
||||
req := audita.PolishRequest{
|
||||
@@ -128,19 +149,25 @@ func (polishStage) Run(ctx context.Context, env *Env, m *manifest.Manifest) (*St
|
||||
}
|
||||
}
|
||||
|
||||
outputs := []artifacts.Ref{{
|
||||
Kind: "transcript_processed",
|
||||
Category: "transcripts",
|
||||
SessionID: sessionID,
|
||||
AbsolutePath: finalProcessedPath,
|
||||
}}
|
||||
promotedProcessed, err := promoteRunLocalOutput(env.ArtifactStore, finalProcessedPath, canonicalProcessedPath, artifacts.Ref{
|
||||
Kind: "transcript_processed",
|
||||
Category: "transcripts",
|
||||
SessionID: sessionID,
|
||||
})
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("polish: promote processed transcript: %w", err)
|
||||
}
|
||||
outputs := []artifacts.Ref{promotedProcessed}
|
||||
if reportEnabled {
|
||||
outputs = append(outputs, artifacts.Ref{
|
||||
Kind: "audita_report",
|
||||
Category: "artifacts",
|
||||
SessionID: sessionID,
|
||||
AbsolutePath: finalReportPath,
|
||||
promotedReport, err := promoteRunLocalOutput(env.ArtifactStore, finalReportPath, canonicalReportPath, artifacts.Ref{
|
||||
Kind: "audita_report",
|
||||
Category: "artifacts",
|
||||
SessionID: sessionID,
|
||||
})
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("polish: promote report: %w", err)
|
||||
}
|
||||
outputs = append(outputs, promotedReport)
|
||||
}
|
||||
|
||||
var validationConcurrency any
|
||||
@@ -155,14 +182,20 @@ func (polishStage) Run(ctx context.Context, env *Env, m *manifest.Manifest) (*St
|
||||
if env.Config.Pipeline.Audita.ProposalLLMConcurrency != nil {
|
||||
proposalLLMConcurrency = *env.Config.Pipeline.Audita.ProposalLLMConcurrency
|
||||
}
|
||||
reportCanonicalPath := ""
|
||||
if reportEnabled {
|
||||
reportCanonicalPath = canonicalReportPath
|
||||
}
|
||||
|
||||
meta := map[string]any{
|
||||
"stage": "polish",
|
||||
"merged_transcript_path": mergedPath,
|
||||
"merged_transcript_source": source,
|
||||
"glossary_path": glossaryPath,
|
||||
"output_path": finalProcessedPath,
|
||||
"report_path": finalReportPath,
|
||||
"run_output_path": finalProcessedPath,
|
||||
"output_path": canonicalProcessedPath,
|
||||
"run_report_path": finalReportPath,
|
||||
"report_path": reportCanonicalPath,
|
||||
"audita_work_dir": workDir,
|
||||
"report_enabled": reportEnabled,
|
||||
"modules": append([]string(nil), req.Modules...),
|
||||
|
||||
@@ -16,7 +16,7 @@ import (
|
||||
|
||||
func TestPolishStagePolishesMergedTranscriptAndRecordsMetadata(t *testing.T) {
|
||||
env, m := setupPolishEnv(t)
|
||||
paths := env.ArtifactStore.SessionPaths(m.SessionID)
|
||||
paths := sessionPathsForEnv(env, m.SessionID)
|
||||
|
||||
mergedPath := filepath.Join(paths.TranscriptsDir, "merged.json")
|
||||
writeFile(t, mergedPath, `{"segments":[]}`)
|
||||
@@ -120,7 +120,7 @@ func TestPolishStagePolishesMergedTranscriptAndRecordsMetadata(t *testing.T) {
|
||||
|
||||
func TestPolishStageFallsBackToMergedTranscriptPath(t *testing.T) {
|
||||
env, m := setupPolishEnv(t)
|
||||
paths := env.ArtifactStore.SessionPaths(m.SessionID)
|
||||
paths := sessionPathsForEnv(env, m.SessionID)
|
||||
mergedPath := filepath.Join(paths.TranscriptsDir, "merged.json")
|
||||
writeFile(t, mergedPath, `{"segments":[]}`)
|
||||
writeFile(t, filepath.Join(paths.InputsDir, "glossary.yml"), "terms: []\n")
|
||||
@@ -142,7 +142,7 @@ func TestPolishStageFallsBackToMergedTranscriptPath(t *testing.T) {
|
||||
|
||||
func TestPolishStageFailsWhenMergedTranscriptMissing(t *testing.T) {
|
||||
env, m := setupPolishEnv(t)
|
||||
paths := env.ArtifactStore.SessionPaths(m.SessionID)
|
||||
paths := sessionPathsForEnv(env, m.SessionID)
|
||||
writeFile(t, filepath.Join(paths.InputsDir, "glossary.yml"), "terms: []\n")
|
||||
env.Audita = &audita.FakeRunner{}
|
||||
|
||||
@@ -157,7 +157,7 @@ func TestPolishStageFailsWhenMergedTranscriptMissing(t *testing.T) {
|
||||
|
||||
func TestPolishStageFailsWhenMergedTranscriptInvalidJSON(t *testing.T) {
|
||||
env, m := setupPolishEnv(t)
|
||||
paths := env.ArtifactStore.SessionPaths(m.SessionID)
|
||||
paths := sessionPathsForEnv(env, m.SessionID)
|
||||
writeFile(t, filepath.Join(paths.TranscriptsDir, "merged.json"), "not-json")
|
||||
writeFile(t, filepath.Join(paths.InputsDir, "glossary.yml"), "terms: []\n")
|
||||
env.Audita = &audita.FakeRunner{}
|
||||
@@ -173,7 +173,7 @@ func TestPolishStageFailsWhenMergedTranscriptInvalidJSON(t *testing.T) {
|
||||
|
||||
func TestPolishStageFailsWhenGlossaryMissing(t *testing.T) {
|
||||
env, m := setupPolishEnv(t)
|
||||
paths := env.ArtifactStore.SessionPaths(m.SessionID)
|
||||
paths := sessionPathsForEnv(env, m.SessionID)
|
||||
writeFile(t, filepath.Join(paths.TranscriptsDir, "merged.json"), `{"segments":[]}`)
|
||||
env.Audita = &audita.FakeRunner{}
|
||||
|
||||
@@ -188,7 +188,7 @@ func TestPolishStageFailsWhenGlossaryMissing(t *testing.T) {
|
||||
|
||||
func TestPolishStageFailsWhenAdapterFails(t *testing.T) {
|
||||
env, m := setupPolishEnv(t)
|
||||
paths := env.ArtifactStore.SessionPaths(m.SessionID)
|
||||
paths := sessionPathsForEnv(env, m.SessionID)
|
||||
writeFile(t, filepath.Join(paths.TranscriptsDir, "merged.json"), `{"segments":[]}`)
|
||||
writeFile(t, filepath.Join(paths.InputsDir, "glossary.yml"), "terms: []\n")
|
||||
env.Audita = &audita.FakeRunner{Err: errors.New("audita failed")}
|
||||
@@ -204,7 +204,7 @@ func TestPolishStageFailsWhenAdapterFails(t *testing.T) {
|
||||
|
||||
func TestPolishStageFailsWhenProcessedOutputInvalid(t *testing.T) {
|
||||
env, m := setupPolishEnv(t)
|
||||
paths := env.ArtifactStore.SessionPaths(m.SessionID)
|
||||
paths := sessionPathsForEnv(env, m.SessionID)
|
||||
writeFile(t, filepath.Join(paths.TranscriptsDir, "merged.json"), `{"segments":[]}`)
|
||||
writeFile(t, filepath.Join(paths.InputsDir, "glossary.yml"), "terms: []\n")
|
||||
badPath := filepath.Join(paths.TranscriptsDir, "processed.invalid.json")
|
||||
@@ -222,7 +222,7 @@ func TestPolishStageFailsWhenProcessedOutputInvalid(t *testing.T) {
|
||||
|
||||
func TestPolishStageFailsWhenReportInvalid(t *testing.T) {
|
||||
env, m := setupPolishEnv(t)
|
||||
paths := env.ArtifactStore.SessionPaths(m.SessionID)
|
||||
paths := sessionPathsForEnv(env, m.SessionID)
|
||||
writeFile(t, filepath.Join(paths.TranscriptsDir, "merged.json"), `{"segments":[]}`)
|
||||
writeFile(t, filepath.Join(paths.InputsDir, "glossary.yml"), "terms: []\n")
|
||||
badReport := filepath.Join(paths.ArtifactsDir, "bad.report.json")
|
||||
@@ -238,6 +238,36 @@ func TestPolishStageFailsWhenReportInvalid(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
func TestPolishStageUsesRunLocalPathsAndPromotesCanonical(t *testing.T) {
|
||||
env, m := setupPolishEnv(t)
|
||||
env.Config.Session.Campaign = "sample-campaign"
|
||||
m.Campaign = "sample-campaign"
|
||||
m.RunID = "20260518T010203Z-abcdef12"
|
||||
paths := sessionPathsForEnv(env, m.SessionID)
|
||||
writeFile(t, filepath.Join(paths.TranscriptsDir, "merged.json"), `{"segments":[]}`)
|
||||
writeFile(t, filepath.Join(paths.InputsDir, "glossary.yml"), "terms: []\n")
|
||||
|
||||
fake := &audita.FakeRunner{}
|
||||
env.Audita = fake
|
||||
result, err := (polishStage{}).Run(context.Background(), env, m)
|
||||
if err != nil {
|
||||
t.Fatalf("polish.Run() error = %v", err)
|
||||
}
|
||||
if len(fake.Requests) != 1 {
|
||||
t.Fatalf("requests = %d, want 1", len(fake.Requests))
|
||||
}
|
||||
req := fake.Requests[0]
|
||||
if !strings.Contains(req.OutputProcessedPath, filepath.Join("runs", m.RunID, "polish", "outputs")) {
|
||||
t.Fatalf("run output path = %q, want run-local path", req.OutputProcessedPath)
|
||||
}
|
||||
if len(result.Outputs) == 0 {
|
||||
t.Fatalf("outputs = %#v, want promoted outputs", result.Outputs)
|
||||
}
|
||||
if strings.Contains(result.Outputs[0].AbsolutePath, string(filepath.Separator)+"runs"+string(filepath.Separator)) {
|
||||
t.Fatalf("promoted output path = %q, want canonical session path", result.Outputs[0].AbsolutePath)
|
||||
}
|
||||
}
|
||||
|
||||
func setupPolishEnv(t *testing.T) (*Env, *manifest.Manifest) {
|
||||
t.Helper()
|
||||
workspace := t.TempDir()
|
||||
@@ -284,7 +314,7 @@ func setupPolishEnv(t *testing.T) (*Env, *manifest.Manifest) {
|
||||
}
|
||||
|
||||
store := artifacts.NewLocalStore(workspace)
|
||||
if _, err := store.EnsureLayout("2026-05-03"); err != nil {
|
||||
if _, err := store.EnsureLayoutFor("sample-campaign", "2026-05-03"); err != nil {
|
||||
t.Fatalf("EnsureLayout() error = %v", err)
|
||||
}
|
||||
return &Env{
|
||||
|
||||
@@ -54,7 +54,7 @@ func (prepareStage) Run(ctx context.Context, env *Env, m *manifest.Manifest) (*S
|
||||
return nil, fmt.Errorf("prepare: session id is required")
|
||||
}
|
||||
|
||||
paths, err := env.ArtifactStore.EnsureLayout(sessionID)
|
||||
paths, err := ensureLayoutForEnv(env, sessionID)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("prepare: ensure workdir layout: %w", err)
|
||||
}
|
||||
@@ -358,6 +358,10 @@ func pathsWorkDirForManifest(env *Env, m *manifest.Manifest, sessionID string) s
|
||||
if env == nil || env.Config == nil || env.Config.Pipeline == nil || env.Config.Session == nil {
|
||||
return ""
|
||||
}
|
||||
campaign := strings.TrimSpace(env.Config.Session.Campaign)
|
||||
if campaign == "" {
|
||||
return ""
|
||||
}
|
||||
if m != nil && strings.TrimSpace(m.LocalWorkDir) != "" {
|
||||
return strings.TrimSpace(m.LocalWorkDir)
|
||||
}
|
||||
@@ -366,9 +370,9 @@ func pathsWorkDirForManifest(env *Env, m *manifest.Manifest, sessionID string) s
|
||||
runID = strings.TrimSpace(m.RunID)
|
||||
}
|
||||
if runID != "" {
|
||||
return artifacts.SessionRunWorkDir(env.Config.Pipeline.Workspace.Root, env.Config.Session.Campaign, sessionID, runID)
|
||||
return artifacts.SessionRunRootForCampaign(env.Config.Pipeline.Workspace.Root, campaign, sessionID, runID)
|
||||
}
|
||||
return artifacts.SessionWorkDir(env.Config.Pipeline.Workspace.Root, sessionID)
|
||||
return artifacts.SessionWorkDirForCampaign(env.Config.Pipeline.Workspace.Root, campaign, sessionID)
|
||||
}
|
||||
|
||||
func resolvePath(baseDir, p string) (string, error) {
|
||||
|
||||
@@ -33,7 +33,7 @@ func TestPrepareStageExplicitAudioFiles(t *testing.T) {
|
||||
t.Fatalf("result metadata = %#v, want prepared=true", result)
|
||||
}
|
||||
|
||||
paths := env.ArtifactStore.SessionPaths(m.SessionID)
|
||||
paths := sessionPathsForEnv(env, m.SessionID)
|
||||
for _, p := range []string{
|
||||
filepath.Join(paths.InputsDir, "session.yml"),
|
||||
filepath.Join(paths.InputsDir, "pipeline.resolved.yml"),
|
||||
@@ -76,7 +76,7 @@ func TestPrepareStageAudioDirEnumeration(t *testing.T) {
|
||||
t.Fatalf("audio_files_resolved = %#v, want 1", got)
|
||||
}
|
||||
|
||||
paths := env.ArtifactStore.SessionPaths(m.SessionID)
|
||||
paths := sessionPathsForEnv(env, m.SessionID)
|
||||
if _, err := os.Stat(filepath.Join(paths.AudioDir, "a.flac")); err != nil {
|
||||
t.Fatalf("expected copied flac: %v", err)
|
||||
}
|
||||
@@ -160,7 +160,7 @@ func TestPrepareStageS3AudioDownloadAndMaterialization(t *testing.T) {
|
||||
env.Config.Pipeline.Spool = config.SpoolConfig{Root: filepath.Join(t.TempDir(), "spool")}
|
||||
env.Config.Pipeline.Storage.S3 = &config.StorageS3Config{Bucket: "my-dnd-archive", RootPrefix: "dnd"}
|
||||
m.RunID = "20260515T031522Z-a1b2c3d4"
|
||||
m.LocalWorkDir = artifacts.SessionRunWorkDir(env.Config.Pipeline.Workspace.Root, "forsaken", m.SessionID, m.RunID)
|
||||
m.LocalWorkDir = artifacts.SessionRunRootForCampaign(env.Config.Pipeline.Workspace.Root, "forsaken", m.SessionID, m.RunID)
|
||||
m.LocalSpoolDir = artifacts.SessionSpoolAudioDir(env.Config.Pipeline.Spool.Root, "forsaken", m.SessionID, m.RunID)
|
||||
|
||||
fake := &storage.FakeBackend{}
|
||||
@@ -256,7 +256,7 @@ func TestPrepareStageS3AudioFailures(t *testing.T) {
|
||||
env.Config.Pipeline.Spool = config.SpoolConfig{Root: filepath.Join(t.TempDir(), "spool")}
|
||||
env.Config.Pipeline.Storage.S3 = &config.StorageS3Config{Bucket: "my-dnd-archive", RootPrefix: "dnd"}
|
||||
m.RunID = "20260515T031522Z-a1b2c3d4"
|
||||
m.LocalWorkDir = artifacts.SessionRunWorkDir(env.Config.Pipeline.Workspace.Root, "forsaken", m.SessionID, m.RunID)
|
||||
m.LocalWorkDir = artifacts.SessionRunRootForCampaign(env.Config.Pipeline.Workspace.Root, "forsaken", m.SessionID, m.RunID)
|
||||
m.LocalSpoolDir = artifacts.SessionSpoolAudioDir(env.Config.Pipeline.Spool.Root, "forsaken", m.SessionID, m.RunID)
|
||||
|
||||
fake := &storage.FakeBackend{}
|
||||
|
||||
136
internal/stage/run_local.go
Normal file
136
internal/stage/run_local.go
Normal file
@@ -0,0 +1,136 @@
|
||||
package stage
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
|
||||
"gitea.maximumdirect.net/eric/narratio/internal/artifacts"
|
||||
"gitea.maximumdirect.net/eric/narratio/internal/manifest"
|
||||
)
|
||||
|
||||
type runStageLayout struct {
|
||||
Enabled bool
|
||||
Root string
|
||||
OutputsDir string
|
||||
LogsDir string
|
||||
ReportsDir string
|
||||
ConfigDir string
|
||||
ScratchDir string
|
||||
}
|
||||
|
||||
func resolveRunStageLayout(
|
||||
env *Env,
|
||||
m *manifest.Manifest,
|
||||
sessionPaths artifacts.SessionPaths,
|
||||
sessionID, stageName string,
|
||||
) (runStageLayout, error) {
|
||||
if env == nil || env.Config == nil || env.Config.Pipeline == nil {
|
||||
return runStageLayout{}, fmt.Errorf("stage environment pipeline config is required")
|
||||
}
|
||||
if strings.TrimSpace(sessionID) == "" {
|
||||
return runStageLayout{}, fmt.Errorf("session id is required")
|
||||
}
|
||||
stageName = strings.TrimSpace(stageName)
|
||||
if stageName == "" {
|
||||
return runStageLayout{}, fmt.Errorf("stage name is required")
|
||||
}
|
||||
|
||||
runID := ""
|
||||
if m != nil {
|
||||
runID = strings.TrimSpace(m.RunID)
|
||||
}
|
||||
campaign := strings.TrimSpace(env.Config.Session.Campaign)
|
||||
if campaign == "" && m != nil {
|
||||
campaign = strings.TrimSpace(m.Campaign)
|
||||
}
|
||||
|
||||
// Compatibility fallback for direct stage tests and older call paths
|
||||
// that execute a stage without a run id.
|
||||
if runID == "" || campaign == "" {
|
||||
return runStageLayout{}, nil
|
||||
}
|
||||
|
||||
root := artifacts.SessionRunStageDirForCampaign(
|
||||
env.Config.Pipeline.Workspace.Root,
|
||||
campaign,
|
||||
sessionID,
|
||||
runID,
|
||||
stageName,
|
||||
)
|
||||
layout := runStageLayout{
|
||||
Enabled: true,
|
||||
Root: root,
|
||||
OutputsDir: filepath.Join(root, "outputs"),
|
||||
LogsDir: filepath.Join(root, "logs"),
|
||||
ReportsDir: filepath.Join(root, "reports"),
|
||||
ConfigDir: filepath.Join(root, "config"),
|
||||
ScratchDir: filepath.Join(root, "scratch"),
|
||||
}
|
||||
for _, dir := range []string{
|
||||
layout.Root,
|
||||
layout.OutputsDir,
|
||||
layout.LogsDir,
|
||||
layout.ReportsDir,
|
||||
layout.ConfigDir,
|
||||
layout.ScratchDir,
|
||||
} {
|
||||
if err := os.MkdirAll(dir, 0o755); err != nil {
|
||||
return runStageLayout{}, fmt.Errorf("create run-stage directory %q: %w", dir, err)
|
||||
}
|
||||
}
|
||||
return layout, nil
|
||||
}
|
||||
|
||||
func runLocalPathForCanonical(layout runStageLayout, sessionPaths artifacts.SessionPaths, canonicalPath string) (string, error) {
|
||||
if !layout.Enabled {
|
||||
return filepath.Clean(canonicalPath), nil
|
||||
}
|
||||
cleanCanonical := filepath.Clean(strings.TrimSpace(canonicalPath))
|
||||
if cleanCanonical == "" {
|
||||
return "", fmt.Errorf("canonical path is required")
|
||||
}
|
||||
rel, err := filepath.Rel(filepath.Clean(sessionPaths.Root), cleanCanonical)
|
||||
if err != nil {
|
||||
return "", fmt.Errorf("derive session-relative path for %q: %w", cleanCanonical, err)
|
||||
}
|
||||
rel = filepath.Clean(rel)
|
||||
if rel == "." || rel == ".." || strings.HasPrefix(rel, ".."+string(filepath.Separator)) {
|
||||
return "", fmt.Errorf("canonical path %q is outside session root %q", cleanCanonical, sessionPaths.Root)
|
||||
}
|
||||
localPath := filepath.Join(layout.OutputsDir, rel)
|
||||
if err := os.MkdirAll(filepath.Dir(localPath), 0o755); err != nil {
|
||||
return "", fmt.Errorf("create run-local output parent for %q: %w", localPath, err)
|
||||
}
|
||||
return localPath, nil
|
||||
}
|
||||
|
||||
func promoteRunLocalOutput(
|
||||
store artifacts.Store,
|
||||
srcPath, canonicalPath string,
|
||||
ref artifacts.Ref,
|
||||
) (artifacts.Ref, error) {
|
||||
srcPath = filepath.Clean(strings.TrimSpace(srcPath))
|
||||
canonicalPath = filepath.Clean(strings.TrimSpace(canonicalPath))
|
||||
if srcPath == "" {
|
||||
return artifacts.Ref{}, fmt.Errorf("source path is required")
|
||||
}
|
||||
if canonicalPath == "" {
|
||||
return artifacts.Ref{}, fmt.Errorf("canonical destination path is required")
|
||||
}
|
||||
data, err := os.ReadFile(srcPath)
|
||||
if err != nil {
|
||||
return artifacts.Ref{}, fmt.Errorf("read run-local output %q: %w", srcPath, err)
|
||||
}
|
||||
if err := store.WriteFileAtomic(canonicalPath, data, 0o644); err != nil {
|
||||
return artifacts.Ref{}, fmt.Errorf("promote output to %q: %w", canonicalPath, err)
|
||||
}
|
||||
checksum, err := store.Checksum(canonicalPath)
|
||||
if err != nil {
|
||||
return artifacts.Ref{}, fmt.Errorf("checksum promoted output %q: %w", canonicalPath, err)
|
||||
}
|
||||
ref.AbsolutePath = canonicalPath
|
||||
ref.Checksum = checksum
|
||||
return ref, nil
|
||||
}
|
||||
35
internal/stage/run_local_test.go
Normal file
35
internal/stage/run_local_test.go
Normal file
@@ -0,0 +1,35 @@
|
||||
package stage
|
||||
|
||||
import (
|
||||
"os"
|
||||
"path/filepath"
|
||||
"testing"
|
||||
|
||||
"gitea.maximumdirect.net/eric/narratio/internal/artifacts"
|
||||
)
|
||||
|
||||
func TestRunLocalPathForCanonicalCreatesParentDirectories(t *testing.T) {
|
||||
root := t.TempDir()
|
||||
sessionRoot := filepath.Join(root, "work", "dilfs", "2026-05-17")
|
||||
layout := runStageLayout{
|
||||
Enabled: true,
|
||||
OutputsDir: filepath.Join(sessionRoot, "runs", "run-1", "merge", "outputs"),
|
||||
}
|
||||
if err := os.MkdirAll(layout.OutputsDir, 0o755); err != nil {
|
||||
t.Fatalf("mkdir outputs dir: %v", err)
|
||||
}
|
||||
|
||||
canonical := filepath.Join(sessionRoot, "transcripts", "merged.json")
|
||||
got, err := runLocalPathForCanonical(layout, artifacts.SessionPaths{Root: sessionRoot}, canonical)
|
||||
if err != nil {
|
||||
t.Fatalf("runLocalPathForCanonical() error = %v", err)
|
||||
}
|
||||
|
||||
want := filepath.Join(layout.OutputsDir, "transcripts", "merged.json")
|
||||
if got != want {
|
||||
t.Fatalf("runLocalPathForCanonical() = %q, want %q", got, want)
|
||||
}
|
||||
if _, err := os.Stat(filepath.Dir(got)); err != nil {
|
||||
t.Fatalf("expected run-local parent directory to exist: %v", err)
|
||||
}
|
||||
}
|
||||
23
internal/stage/session_paths.go
Normal file
23
internal/stage/session_paths.go
Normal file
@@ -0,0 +1,23 @@
|
||||
package stage
|
||||
|
||||
import (
|
||||
"strings"
|
||||
|
||||
"gitea.maximumdirect.net/eric/narratio/internal/artifacts"
|
||||
)
|
||||
|
||||
func sessionPathsForEnv(env *Env, sessionID string) artifacts.SessionPaths {
|
||||
campaign := ""
|
||||
if env != nil && env.Config != nil && env.Config.Session != nil {
|
||||
campaign = strings.TrimSpace(env.Config.Session.Campaign)
|
||||
}
|
||||
return env.ArtifactStore.SessionPathsFor(campaign, sessionID)
|
||||
}
|
||||
|
||||
func ensureLayoutForEnv(env *Env, sessionID string) (artifacts.SessionPaths, error) {
|
||||
campaign := ""
|
||||
if env != nil && env.Config != nil && env.Config.Session != nil {
|
||||
campaign = strings.TrimSpace(env.Config.Session.Campaign)
|
||||
}
|
||||
return env.ArtifactStore.EnsureLayoutFor(campaign, sessionID)
|
||||
}
|
||||
@@ -55,7 +55,11 @@ func (transcribeStage) Run(ctx context.Context, env *Env, m *manifest.Manifest)
|
||||
return nil, fmt.Errorf("transcribe: session id is required")
|
||||
}
|
||||
|
||||
paths := env.ArtifactStore.SessionPaths(sessionID)
|
||||
paths := sessionPathsForEnv(env, sessionID)
|
||||
runLayout, err := resolveRunStageLayout(env, m, paths, sessionID, "transcribe")
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("transcribe: resolve run-stage layout: %w", err)
|
||||
}
|
||||
audioFiles, err := discoverPreparedAudio(m, paths.AudioDir)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("transcribe: resolve audio inputs: %w", err)
|
||||
@@ -88,10 +92,15 @@ func (transcribeStage) Run(ctx context.Context, env *Env, m *manifest.Manifest)
|
||||
return nil, fmt.Errorf("transcribe: duplicate speaker/audio basename %q from %q and %q", base, prev, audioPath)
|
||||
}
|
||||
seenSpeaker[base] = audioPath
|
||||
canonicalOut := filepath.Join(paths.TranscriptsRawDir, base+".json")
|
||||
runOut, err := runLocalPathForCanonical(runLayout, paths, canonicalOut)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("transcribe: resolve run-local output path for %q: %w", base, err)
|
||||
}
|
||||
jobs = append(jobs, job{
|
||||
speakerID: base,
|
||||
audioPath: audioPath,
|
||||
outPath: filepath.Join(paths.TranscriptsRawDir, base+".json"),
|
||||
outPath: runOut,
|
||||
})
|
||||
}
|
||||
|
||||
@@ -197,12 +206,19 @@ dispatch:
|
||||
sort.Strings(speakers)
|
||||
|
||||
outputs := make([]artifacts.Ref, 0, len(speakers))
|
||||
runOutputPaths := make([]string, 0, len(speakers))
|
||||
outputPaths := make([]string, 0, len(speakers))
|
||||
orderedPerFile := make(map[string]any, len(speakers))
|
||||
for _, speaker := range speakers {
|
||||
ref := outputRef[speaker]
|
||||
outputs = append(outputs, ref)
|
||||
outputPaths = append(outputPaths, ref.AbsolutePath)
|
||||
runOutputPaths = append(runOutputPaths, ref.AbsolutePath)
|
||||
canonicalOut := filepath.Join(paths.TranscriptsRawDir, speaker+".json")
|
||||
promoted, err := promoteRunLocalOutput(env.ArtifactStore, ref.AbsolutePath, canonicalOut, ref)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("transcribe: promote %q output: %w", speaker, err)
|
||||
}
|
||||
outputs = append(outputs, promoted)
|
||||
outputPaths = append(outputPaths, canonicalOut)
|
||||
orderedPerFile[speaker] = perFile[speaker]
|
||||
}
|
||||
|
||||
@@ -221,6 +237,7 @@ dispatch:
|
||||
"retries": retries,
|
||||
"retry_delay": env.Config.Pipeline.WhisperX.RetryDelay,
|
||||
"timeout": env.Config.Pipeline.WhisperX.Timeout,
|
||||
"run_output_paths": runOutputPaths,
|
||||
"output_paths": outputPaths,
|
||||
"per_file": orderedPerFile,
|
||||
},
|
||||
|
||||
@@ -53,8 +53,8 @@ func TestTranscribeStageTranscribesPreparedAudio(t *testing.T) {
|
||||
}
|
||||
sort.Strings(gotPaths)
|
||||
wantPaths := []string{
|
||||
filepath.Join(env.ArtifactStore.SessionPaths(m.SessionID).TranscriptsRawDir, "alice.json"),
|
||||
filepath.Join(env.ArtifactStore.SessionPaths(m.SessionID).TranscriptsRawDir, "bob.json"),
|
||||
filepath.Join(sessionPathsForEnv(env, m.SessionID).TranscriptsRawDir, "alice.json"),
|
||||
filepath.Join(sessionPathsForEnv(env, m.SessionID).TranscriptsRawDir, "bob.json"),
|
||||
}
|
||||
sort.Strings(wantPaths)
|
||||
if strings.Join(gotPaths, "|") != strings.Join(wantPaths, "|") {
|
||||
@@ -191,6 +191,36 @@ func TestTranscribeStageInvalidJSONFails(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
func TestTranscribeStageUsesRunLocalOutputAndPromotesCanonical(t *testing.T) {
|
||||
env, m := setupTranscribeEnv(t, []string{"alice.flac"})
|
||||
env.Config.Session.Campaign = "sample-campaign"
|
||||
m.Campaign = "sample-campaign"
|
||||
m.RunID = "20260518T010203Z-abcdef12"
|
||||
fake := &whisperx.FakeClient{}
|
||||
env.WhisperX = fake
|
||||
|
||||
if _, err := (prepareStage{}).Run(context.Background(), env, m); err != nil {
|
||||
t.Fatalf("prepare.Run() error = %v", err)
|
||||
}
|
||||
result, err := (transcribeStage{}).Run(context.Background(), env, m)
|
||||
if err != nil {
|
||||
t.Fatalf("transcribe.Run() error = %v", err)
|
||||
}
|
||||
if len(fake.Requests) != 1 {
|
||||
t.Fatalf("requests = %d, want 1", len(fake.Requests))
|
||||
}
|
||||
runOut := fake.Requests[0].OutputRawTranscriptPath
|
||||
if !strings.Contains(runOut, filepath.Join("runs", m.RunID, "transcribe", "outputs")) {
|
||||
t.Fatalf("run-local output path = %q, want runs/{run_id}/transcribe/outputs path", runOut)
|
||||
}
|
||||
if len(result.Outputs) != 1 {
|
||||
t.Fatalf("outputs = %#v, want one output", result.Outputs)
|
||||
}
|
||||
if strings.Contains(result.Outputs[0].AbsolutePath, string(filepath.Separator)+"runs"+string(filepath.Separator)) {
|
||||
t.Fatalf("promoted output path = %q, want canonical session path", result.Outputs[0].AbsolutePath)
|
||||
}
|
||||
}
|
||||
|
||||
func setupTranscribeEnv(t *testing.T, audioFiles []string) (*Env, *manifest.Manifest) {
|
||||
t.Helper()
|
||||
|
||||
@@ -203,7 +233,7 @@ func setupTranscribeEnv(t *testing.T, audioFiles []string) (*Env, *manifest.Mani
|
||||
|
||||
sessionPath := filepath.Join(cfgDir, "session.yml")
|
||||
pipelinePath := filepath.Join(cfgDir, "pipeline.yml")
|
||||
writeFile(t, sessionPath, "session_id: 2026-05-03\n")
|
||||
writeFile(t, sessionPath, "session_id: 2026-05-03\ncampaign: sample-campaign\n")
|
||||
writeFile(t, pipelinePath, "workspace:\n root: "+workspace+"\n")
|
||||
writeFile(t, filepath.Join(cfgDir, "speakers.yml"), "alice: alice.flac\n")
|
||||
writeFile(t, filepath.Join(cfgDir, "autocorrect.yml"), "[]\n")
|
||||
@@ -227,6 +257,7 @@ func setupTranscribeEnv(t *testing.T, audioFiles []string) (*Env, *manifest.Mani
|
||||
},
|
||||
Session: &config.SessionConfig{
|
||||
SessionID: "2026-05-03",
|
||||
Campaign: "sample-campaign",
|
||||
Inputs: config.SessionInputsConfig{
|
||||
AudioDir: "./audio",
|
||||
SpeakersFile: "./speakers.yml",
|
||||
|
||||
@@ -54,7 +54,11 @@ func (trimStage) Run(ctx context.Context, env *Env, m *manifest.Manifest) (*Stag
|
||||
return nil, fmt.Errorf("trim: session id is required")
|
||||
}
|
||||
|
||||
paths := env.ArtifactStore.SessionPaths(sessionID)
|
||||
paths := sessionPathsForEnv(env, sessionID)
|
||||
runLayout, err := resolveRunStageLayout(env, m, paths, sessionID, "trim")
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("trim: resolve run-stage layout: %w", err)
|
||||
}
|
||||
normalizedPath, normalizedSource, err := discoverNormalizedTranscript(m, paths)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("trim: resolve normalized transcript: %w", err)
|
||||
@@ -69,10 +73,14 @@ func (trimStage) Run(ctx context.Context, env *Env, m *manifest.Manifest) (*Stag
|
||||
trimCfg := env.Config.Pipeline.Trim
|
||||
enabled := trimCfg != nil && trimCfg.Enabled
|
||||
|
||||
trimmedPath, err := resolveTrimmedOutputPath(paths, trimCfg)
|
||||
canonicalTrimmedPath, err := resolveTrimmedOutputPath(paths, trimCfg)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("trim: resolve trimmed output path: %w", err)
|
||||
}
|
||||
trimmedPath, err := runLocalPathForCanonical(runLayout, paths, canonicalTrimmedPath)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("trim: resolve run-local trimmed output path: %w", err)
|
||||
}
|
||||
|
||||
logPaths := []string{}
|
||||
generatedConfigs := []string{}
|
||||
@@ -81,7 +89,8 @@ func (trimStage) Run(ctx context.Context, env *Env, m *manifest.Manifest) (*Stag
|
||||
"trim_enabled": enabled,
|
||||
"normalized_transcript_path": normalizedPath,
|
||||
"normalized_transcript_source": normalizedSource,
|
||||
"trimmed_output_path": trimmedPath,
|
||||
"run_trimmed_output_path": trimmedPath,
|
||||
"trimmed_output_path": canonicalTrimmedPath,
|
||||
}
|
||||
|
||||
if !enabled {
|
||||
@@ -91,14 +100,17 @@ func (trimStage) Run(ctx context.Context, env *Env, m *manifest.Manifest) (*Stag
|
||||
if err := validateProcessedTranscriptOutput(trimmedPath); err != nil {
|
||||
return nil, fmt.Errorf("trim: copied trimmed transcript %q invalid: %w", trimmedPath, err)
|
||||
}
|
||||
promotedTrimmed, err := promoteRunLocalOutput(env.ArtifactStore, trimmedPath, canonicalTrimmedPath, artifacts.Ref{
|
||||
Kind: "transcript_trimmed",
|
||||
Category: "transcripts",
|
||||
SessionID: sessionID,
|
||||
})
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("trim: promote trimmed transcript: %w", err)
|
||||
}
|
||||
metadata["trim_action"] = "copy_disabled"
|
||||
return &StageResult{
|
||||
Outputs: []artifacts.Ref{{
|
||||
Kind: "transcript_trimmed",
|
||||
Category: "transcripts",
|
||||
SessionID: sessionID,
|
||||
AbsolutePath: trimmedPath,
|
||||
}},
|
||||
Outputs: []artifacts.Ref{promotedTrimmed},
|
||||
Metadata: metadata,
|
||||
}, nil
|
||||
}
|
||||
@@ -114,13 +126,22 @@ func (trimStage) Run(ctx context.Context, env *Env, m *manifest.Manifest) (*Stag
|
||||
}
|
||||
|
||||
boundsCfg := trimCfg.Bounds
|
||||
boundsOutputPath, err := resolveScriptoriumOutputPath(paths, boundsCfg.OutputPath)
|
||||
canonicalBoundsOutputPath, err := resolveScriptoriumOutputPath(paths, boundsCfg.OutputPath)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("trim: resolve bounds output path: %w", err)
|
||||
}
|
||||
boundsOutputPath, err := runLocalPathForCanonical(runLayout, paths, canonicalBoundsOutputPath)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("trim: resolve run-local bounds output path: %w", err)
|
||||
}
|
||||
boundsStdoutLogPath := filepath.Join(paths.LogsDir, "scriptorium.bounds.stdout.log")
|
||||
boundsStderrLogPath := filepath.Join(paths.LogsDir, "scriptorium.bounds.stderr.log")
|
||||
boundsGeneratedConfigPath := filepath.Join(paths.ConfigDir, "scriptorium.bounds.generated.yml")
|
||||
if runLayout.Enabled {
|
||||
boundsStdoutLogPath = filepath.Join(runLayout.LogsDir, "scriptorium.bounds.stdout.log")
|
||||
boundsStderrLogPath = filepath.Join(runLayout.LogsDir, "scriptorium.bounds.stderr.log")
|
||||
boundsGeneratedConfigPath = filepath.Join(runLayout.ConfigDir, "scriptorium.bounds.generated.yml")
|
||||
}
|
||||
boundsTimeout, err := resolveScriptoriumTimeout(env.Config.Pipeline.Scriptorium.Timeout, boundsCfg.Timeout)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("trim: resolve bounds timeout: %w", err)
|
||||
@@ -133,7 +154,8 @@ func (trimStage) Run(ctx context.Context, env *Env, m *manifest.Manifest) (*Stag
|
||||
|
||||
metadata["bounds_prompt_id"] = boundsCfg.PromptID
|
||||
metadata["bounds_profile_id"] = boundsCfg.ProfileID
|
||||
metadata["bounds_output_path"] = boundsOutputPath
|
||||
metadata["run_bounds_output_path"] = boundsOutputPath
|
||||
metadata["bounds_output_path"] = canonicalBoundsOutputPath
|
||||
metadata["bounds_timeout"] = boundsTimeout.String()
|
||||
metadata["bounds_input_name"] = boundsCfg.TranscriptInputName
|
||||
metadata["bounds_input_path"] = normalizedPath
|
||||
@@ -141,13 +163,22 @@ func (trimStage) Run(ctx context.Context, env *Env, m *manifest.Manifest) (*Stag
|
||||
|
||||
renderOutputPath := ""
|
||||
if boundsCfg.RenderDebug {
|
||||
renderOutputPath, err = resolveScriptoriumOutputPath(paths, boundsCfg.RenderOutputPath)
|
||||
canonicalRenderOutputPath, err := resolveScriptoriumOutputPath(paths, boundsCfg.RenderOutputPath)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("trim: resolve bounds render output path: %w", err)
|
||||
}
|
||||
renderOutputPath, err = runLocalPathForCanonical(runLayout, paths, canonicalRenderOutputPath)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("trim: resolve run-local bounds render output path: %w", err)
|
||||
}
|
||||
renderStdoutLogPath := filepath.Join(paths.LogsDir, "scriptorium.bounds.render.stdout.log")
|
||||
renderStderrLogPath := filepath.Join(paths.LogsDir, "scriptorium.bounds.render.stderr.log")
|
||||
renderGeneratedConfigPath := filepath.Join(paths.ConfigDir, "scriptorium.bounds.render.generated.yml")
|
||||
if runLayout.Enabled {
|
||||
renderStdoutLogPath = filepath.Join(runLayout.LogsDir, "scriptorium.bounds.render.stdout.log")
|
||||
renderStderrLogPath = filepath.Join(runLayout.LogsDir, "scriptorium.bounds.render.stderr.log")
|
||||
renderGeneratedConfigPath = filepath.Join(runLayout.ConfigDir, "scriptorium.bounds.render.generated.yml")
|
||||
}
|
||||
|
||||
renderReq := scriptorium.RenderArtifactRequest{
|
||||
Binary: env.Config.Pipeline.Scriptorium.Binary,
|
||||
@@ -254,7 +285,7 @@ func (trimStage) Run(ctx context.Context, env *Env, m *manifest.Manifest) (*Stag
|
||||
metadata["end_segment_id"] = boundsPayload.EndSegmentID
|
||||
metadata["warnings"] = boundsPayload.Warnings
|
||||
metadata["keep_selector"] = keepSelector
|
||||
metadata["bounds_output_path"] = finalBoundsOutputPath
|
||||
metadata["run_bounds_output_path"] = finalBoundsOutputPath
|
||||
metadata["bounds_stdout_log_path"] = boundsStdoutLogPath
|
||||
metadata["bounds_stderr_log_path"] = boundsStderrLogPath
|
||||
metadata["bounds_generated_config_path"] = boundsGeneratedConfigPath
|
||||
@@ -279,6 +310,11 @@ func (trimStage) Run(ctx context.Context, env *Env, m *manifest.Manifest) (*Stag
|
||||
trimStdoutLogPath := filepath.Join(paths.LogsDir, "seriatim.trim.stdout.log")
|
||||
trimStderrLogPath := filepath.Join(paths.LogsDir, "seriatim.trim.stderr.log")
|
||||
trimGeneratedConfigPath := filepath.Join(paths.ConfigDir, "seriatim.trim.generated.yml")
|
||||
if runLayout.Enabled {
|
||||
trimStdoutLogPath = filepath.Join(runLayout.LogsDir, "seriatim.trim.stdout.log")
|
||||
trimStderrLogPath = filepath.Join(runLayout.LogsDir, "seriatim.trim.stderr.log")
|
||||
trimGeneratedConfigPath = filepath.Join(runLayout.ConfigDir, "seriatim.trim.generated.yml")
|
||||
}
|
||||
trimTimeout, err := resolveTrimSeriatimTimeout(env.Config.Pipeline.Seriatim.Timeout)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("trim: resolve seriatim timeout: %w", err)
|
||||
@@ -315,23 +351,25 @@ func (trimStage) Run(ctx context.Context, env *Env, m *manifest.Manifest) (*Stag
|
||||
return nil, fmt.Errorf("trim: trimmed transcript %q invalid: %w", trimmedPath, err)
|
||||
}
|
||||
|
||||
outputs := []artifacts.Ref{
|
||||
{
|
||||
Kind: "transcript_trimmed",
|
||||
Category: "transcripts",
|
||||
SessionID: sessionID,
|
||||
AbsolutePath: trimmedPath,
|
||||
},
|
||||
{
|
||||
Kind: "session_bounds",
|
||||
Category: "artifacts",
|
||||
SessionID: sessionID,
|
||||
AbsolutePath: finalBoundsOutputPath,
|
||||
},
|
||||
promotedTrimmed, err := promoteRunLocalOutput(env.ArtifactStore, trimmedPath, canonicalTrimmedPath, artifacts.Ref{
|
||||
Kind: "transcript_trimmed",
|
||||
Category: "transcripts",
|
||||
SessionID: sessionID,
|
||||
})
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("trim: promote trimmed transcript: %w", err)
|
||||
}
|
||||
promotedBounds, err := promoteRunLocalOutput(env.ArtifactStore, finalBoundsOutputPath, canonicalBoundsOutputPath, artifacts.Ref{
|
||||
Kind: "session_bounds",
|
||||
Category: "artifacts",
|
||||
SessionID: sessionID,
|
||||
})
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("trim: promote session bounds: %w", err)
|
||||
}
|
||||
|
||||
return &StageResult{
|
||||
Outputs: outputs,
|
||||
Outputs: []artifacts.Ref{promotedTrimmed, promotedBounds},
|
||||
Logs: logPaths,
|
||||
GeneratedConfigs: generatedConfigs,
|
||||
Metadata: metadata,
|
||||
|
||||
@@ -18,7 +18,7 @@ import (
|
||||
|
||||
func TestTrimStageConsumesNormalizedAndProducesTrimmedTranscript(t *testing.T) {
|
||||
env, m, scr, ser := setupTrimEnv(t)
|
||||
paths := env.ArtifactStore.SessionPaths(m.SessionID)
|
||||
paths := sessionPathsForEnv(env, m.SessionID)
|
||||
normalized := filepath.Join(paths.TranscriptsDir, "normalized.json")
|
||||
writeFile(t, normalized, `{"segments":[{"id":10},{"id":868}]}`)
|
||||
m.MarkStageSucceeded("normalize", time.Now().UTC(), []manifest.ArtifactRecord{
|
||||
@@ -70,7 +70,7 @@ func TestTrimStageConsumesNormalizedAndProducesTrimmedTranscript(t *testing.T) {
|
||||
|
||||
func TestTrimStageUsesConfiguredScriptoriumInputName(t *testing.T) {
|
||||
env, m, scr, _ := setupTrimEnv(t)
|
||||
paths := env.ArtifactStore.SessionPaths(m.SessionID)
|
||||
paths := sessionPathsForEnv(env, m.SessionID)
|
||||
normalized := filepath.Join(paths.TranscriptsDir, "normalized.json")
|
||||
writeFile(t, normalized, `{"segments":[{"id":10},{"id":11}]}`)
|
||||
scr.BoundsBody = `{"trim_action":"trim","start_segment_id":10,"end_segment_id":11}`
|
||||
@@ -94,7 +94,7 @@ func TestTrimStageUsesConfiguredScriptoriumInputName(t *testing.T) {
|
||||
|
||||
func TestTrimStageRecordsLogAndGeneratedConfigRefs(t *testing.T) {
|
||||
env, m, scr, _ := setupTrimEnv(t)
|
||||
paths := env.ArtifactStore.SessionPaths(m.SessionID)
|
||||
paths := sessionPathsForEnv(env, m.SessionID)
|
||||
writeFile(t, filepath.Join(paths.TranscriptsDir, "normalized.json"), `{"segments":[{"id":1},{"id":2}]}`)
|
||||
scr.BoundsBody = `{"trim_action":"trim","start_segment_id":1,"end_segment_id":2}`
|
||||
|
||||
@@ -114,7 +114,7 @@ func TestTrimStageRecordsLogAndGeneratedConfigRefs(t *testing.T) {
|
||||
|
||||
func TestTrimStageRenderDebugDiagnosticsAreNotStageOutputs(t *testing.T) {
|
||||
env, m, scr, _ := setupTrimEnv(t)
|
||||
paths := env.ArtifactStore.SessionPaths(m.SessionID)
|
||||
paths := sessionPathsForEnv(env, m.SessionID)
|
||||
writeFile(t, filepath.Join(paths.TranscriptsDir, "normalized.json"), `{"segments":[{"id":1},{"id":2}]}`)
|
||||
scr.BoundsBody = `{"trim_action":"trim","start_segment_id":1,"end_segment_id":2}`
|
||||
|
||||
@@ -152,7 +152,7 @@ func TestTrimStageFailsWhenNormalizedTranscriptMissing(t *testing.T) {
|
||||
|
||||
func TestTrimStageDoesNotFallBackToProcessedTranscript(t *testing.T) {
|
||||
env, m, scr, ser := setupTrimEnv(t)
|
||||
paths := env.ArtifactStore.SessionPaths(m.SessionID)
|
||||
paths := sessionPathsForEnv(env, m.SessionID)
|
||||
writeFile(t, filepath.Join(paths.TranscriptsDir, "processed.json"), `{"segments":[{"id":1},{"id":2}]}`)
|
||||
|
||||
_, err := (trimStage{}).Run(context.Background(), env, m)
|
||||
@@ -172,7 +172,7 @@ func TestTrimStageDoesNotFallBackToProcessedTranscript(t *testing.T) {
|
||||
|
||||
func TestTrimStageFailsWhenNormalizedTranscriptInvalidJSON(t *testing.T) {
|
||||
env, m, _, _ := setupTrimEnv(t)
|
||||
paths := env.ArtifactStore.SessionPaths(m.SessionID)
|
||||
paths := sessionPathsForEnv(env, m.SessionID)
|
||||
writeFile(t, filepath.Join(paths.TranscriptsDir, "normalized.json"), "not-json")
|
||||
_, err := (trimStage{}).Run(context.Background(), env, m)
|
||||
if err == nil {
|
||||
@@ -185,7 +185,7 @@ func TestTrimStageFailsWhenNormalizedTranscriptInvalidJSON(t *testing.T) {
|
||||
|
||||
func TestTrimStageFailsWhenNormalizedTranscriptMissingSegmentsArray(t *testing.T) {
|
||||
env, m, _, _ := setupTrimEnv(t)
|
||||
paths := env.ArtifactStore.SessionPaths(m.SessionID)
|
||||
paths := sessionPathsForEnv(env, m.SessionID)
|
||||
writeFile(t, filepath.Join(paths.TranscriptsDir, "normalized.json"), `{"schema":"audita.processed.v1"}`)
|
||||
_, err := (trimStage{}).Run(context.Background(), env, m)
|
||||
if err == nil {
|
||||
@@ -198,7 +198,7 @@ func TestTrimStageFailsWhenNormalizedTranscriptMissingSegmentsArray(t *testing.T
|
||||
|
||||
func TestTrimStageFailsWhenBoundsOutputInvalidJSON(t *testing.T) {
|
||||
env, m, scr, _ := setupTrimEnv(t)
|
||||
paths := env.ArtifactStore.SessionPaths(m.SessionID)
|
||||
paths := sessionPathsForEnv(env, m.SessionID)
|
||||
writeFile(t, filepath.Join(paths.TranscriptsDir, "normalized.json"), `{"segments":[{"id":1},{"id":2}]}`)
|
||||
scr.BoundsBody = "not-json"
|
||||
_, err := (trimStage{}).Run(context.Background(), env, m)
|
||||
@@ -212,7 +212,7 @@ func TestTrimStageFailsWhenBoundsOutputInvalidJSON(t *testing.T) {
|
||||
|
||||
func TestTrimStageFailsWhenBoundsRangeIsDescending(t *testing.T) {
|
||||
env, m, scr, _ := setupTrimEnv(t)
|
||||
paths := env.ArtifactStore.SessionPaths(m.SessionID)
|
||||
paths := sessionPathsForEnv(env, m.SessionID)
|
||||
writeFile(t, filepath.Join(paths.TranscriptsDir, "normalized.json"), `{"segments":[{"id":1},{"id":2}]}`)
|
||||
scr.BoundsBody = `{"trim_action":"trim","start_segment_id":2,"end_segment_id":1}`
|
||||
_, err := (trimStage{}).Run(context.Background(), env, m)
|
||||
@@ -226,7 +226,7 @@ func TestTrimStageFailsWhenBoundsRangeIsDescending(t *testing.T) {
|
||||
|
||||
func TestTrimStageFailsWhenBoundsIDsMissingFromTranscript(t *testing.T) {
|
||||
env, m, scr, _ := setupTrimEnv(t)
|
||||
paths := env.ArtifactStore.SessionPaths(m.SessionID)
|
||||
paths := sessionPathsForEnv(env, m.SessionID)
|
||||
writeFile(t, filepath.Join(paths.TranscriptsDir, "normalized.json"), `{"segments":[{"id":20},{"id":21}]}`)
|
||||
scr.BoundsBody = `{"trim_action":"trim","start_segment_id":10,"end_segment_id":21}`
|
||||
_, err := (trimStage{}).Run(context.Background(), env, m)
|
||||
@@ -240,7 +240,7 @@ func TestTrimStageFailsWhenBoundsIDsMissingFromTranscript(t *testing.T) {
|
||||
|
||||
func TestTrimStageFailsWhenScriptoriumAdapterFails(t *testing.T) {
|
||||
env, m, scr, _ := setupTrimEnv(t)
|
||||
paths := env.ArtifactStore.SessionPaths(m.SessionID)
|
||||
paths := sessionPathsForEnv(env, m.SessionID)
|
||||
writeFile(t, filepath.Join(paths.TranscriptsDir, "normalized.json"), `{"segments":[{"id":1},{"id":2}]}`)
|
||||
scr.RunErr = errors.New("bounds failed")
|
||||
_, err := (trimStage{}).Run(context.Background(), env, m)
|
||||
@@ -254,7 +254,7 @@ func TestTrimStageFailsWhenScriptoriumAdapterFails(t *testing.T) {
|
||||
|
||||
func TestTrimStageFailsWhenSeriatimTrimAdapterFails(t *testing.T) {
|
||||
env, m, scr, ser := setupTrimEnv(t)
|
||||
paths := env.ArtifactStore.SessionPaths(m.SessionID)
|
||||
paths := sessionPathsForEnv(env, m.SessionID)
|
||||
writeFile(t, filepath.Join(paths.TranscriptsDir, "normalized.json"), `{"segments":[{"id":1},{"id":2}]}`)
|
||||
scr.BoundsBody = `{"trim_action":"trim","start_segment_id":1,"end_segment_id":2}`
|
||||
ser.TrimErr = errors.New("trim failed")
|
||||
@@ -269,7 +269,7 @@ func TestTrimStageFailsWhenSeriatimTrimAdapterFails(t *testing.T) {
|
||||
|
||||
func TestTrimStageDisabledCopiesNormalizedTranscript(t *testing.T) {
|
||||
env, m, scr, ser := setupTrimEnv(t)
|
||||
paths := env.ArtifactStore.SessionPaths(m.SessionID)
|
||||
paths := sessionPathsForEnv(env, m.SessionID)
|
||||
normalized := filepath.Join(paths.TranscriptsDir, "normalized.json")
|
||||
normalizedBody := `{"segments":[{"id":1,"text":"alpha"},{"id":2,"text":"beta"}]}`
|
||||
writeFile(t, normalized, normalizedBody)
|
||||
@@ -298,6 +298,39 @@ func TestTrimStageDisabledCopiesNormalizedTranscript(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
func TestTrimStageUsesRunLocalPathsAndPromotesCanonical(t *testing.T) {
|
||||
env, m, scr, ser := setupTrimEnv(t)
|
||||
env.Config.Session.Campaign = "sample-campaign"
|
||||
m.Campaign = "sample-campaign"
|
||||
m.RunID = "20260518T010203Z-abcdef12"
|
||||
paths := sessionPathsForEnv(env, m.SessionID)
|
||||
normalized := filepath.Join(paths.TranscriptsDir, "normalized.json")
|
||||
writeFile(t, normalized, `{"segments":[{"id":10},{"id":11}]}`)
|
||||
scr.BoundsBody = `{"trim_action":"trim","start_segment_id":10,"end_segment_id":11,"warnings":[]}`
|
||||
|
||||
result, err := (trimStage{}).Run(context.Background(), env, m)
|
||||
if err != nil {
|
||||
t.Fatalf("trim.Run() error = %v", err)
|
||||
}
|
||||
if len(scr.RunRequests) != 1 || len(ser.TrimRequests) != 1 {
|
||||
t.Fatalf("scriptorium run=%d seriatim trim=%d, want 1/1", len(scr.RunRequests), len(ser.TrimRequests))
|
||||
}
|
||||
if !strings.Contains(scr.RunRequests[0].OutputPath, filepath.Join("runs", m.RunID, "trim", "outputs")) {
|
||||
t.Fatalf("bounds run output path = %q, want run-local path", scr.RunRequests[0].OutputPath)
|
||||
}
|
||||
if !strings.Contains(ser.TrimRequests[0].OutputTrimmedPath, filepath.Join("runs", m.RunID, "trim", "outputs")) {
|
||||
t.Fatalf("trim output path = %q, want run-local path", ser.TrimRequests[0].OutputTrimmedPath)
|
||||
}
|
||||
if len(result.Outputs) < 2 {
|
||||
t.Fatalf("outputs = %#v, want promoted trimmed+bounds outputs", result.Outputs)
|
||||
}
|
||||
for _, out := range result.Outputs {
|
||||
if strings.Contains(out.AbsolutePath, string(filepath.Separator)+"runs"+string(filepath.Separator)) {
|
||||
t.Fatalf("promoted output path = %q, want canonical session path", out.AbsolutePath)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
type boundsScriptoriumRunner struct {
|
||||
RunRequests []scriptorium.RunArtifactRequest
|
||||
RenderRequests []scriptorium.RenderArtifactRequest
|
||||
@@ -430,7 +463,7 @@ func setupTrimEnv(t *testing.T) (*Env, *manifest.Manifest, *boundsScriptoriumRun
|
||||
}
|
||||
|
||||
store := artifacts.NewLocalStore(workspace)
|
||||
if _, err := store.EnsureLayout("2026-05-03"); err != nil {
|
||||
if _, err := store.EnsureLayoutFor("sample-campaign", "2026-05-03"); err != nil {
|
||||
t.Fatalf("EnsureLayout() error = %v", err)
|
||||
}
|
||||
|
||||
|
||||
Reference in New Issue
Block a user