17 Commits

Author SHA1 Message Date
37daab7857 Bugfix involving nested directory creation
All checks were successful
ci/woodpecker/tag/release Pipeline was successful
2026-05-18 11:52:40 +00:00
2356688cb9 Removed legacy interfaces and old documentation references to the previous on-disk layout 2026-05-18 03:02:22 +00:00
1054b64d9f Implement minimal downstream invalidation after forced upstream reruns 2026-05-18 01:50:51 +00:00
01fb02426c Update the analyze stage to utilize the new artifact package 2026-05-18 01:29:18 +00:00
7dc79e052f Aligned the archive stage with the new work directory layout 2026-05-18 01:13:51 +00:00
cb525c0f72 Implemented run-local stage execution + immediate promotion for core output-producing stages 2026-05-18 00:55:35 +00:00
622677d038 Added run manifest scaffolding and helpers 2026-05-17 21:15:51 +00:00
550288e008 Add campaign-aware workspace path foundation 2026-05-17 20:57:27 +00:00
e58e545686 Audit workspace architecture implementation plan
All checks were successful
ci/woodpecker/tag/release Pipeline was successful
2026-05-17 13:22:09 -05:00
6ff54c5a0f Documentation update and reorganization 2026-05-17 13:14:28 -05:00
924b5d15c6 Applied a more general bugfix to path-resolution issues in the archive stage 2026-05-17 11:08:11 -05:00
b065663180 Bugfix involving path resolution in the archive stage 2026-05-17 11:03:02 -05:00
3ba564b00f Bugfix involving directory creation during the merge stage 2026-05-17 08:18:57 -05:00
a3986cf0d6 Centralized defaults into internal/config/defaults.go 2026-05-17 07:53:51 -05:00
539601bd16 Updated the merge stage to normalize the per-speaker transcripts before merging them
All checks were successful
ci/woodpecker/tag/release Pipeline was successful
2026-05-16 23:30:13 -05:00
6ca1c8d6b0 The backend S3 client now resolves credentials from user-configurable environment variables 2026-05-16 23:22:21 -05:00
4b7b50981b Add .gocache to .gitignore and minor documentation cleanup 2026-05-16 23:21:45 -05:00
70 changed files with 4147 additions and 1817 deletions

5
.gitignore vendored
View File

@@ -2,6 +2,8 @@
.codex
AGENTS.md
.DS_Store
# ---> Go
# If you prefer the allow list template instead of the deny list, see community template:
# https://github.com/github/gitignore/blob/main/community/Golang/Go.AllowList.gitignore
@@ -22,6 +24,9 @@ AGENTS.md
# Dependency directories (remove the comment below to include it)
# vendor/
# Go cache
.gocache
# Go workspace file
go.work
go.work.sum

View File

@@ -65,13 +65,15 @@ Optional secrets-from-files config:
YAML decoding is strict (`KnownFields(true)`), so unknown fields fail fast.
Maintainer note: application defaults are centralized in [`internal/config/defaults.go`](internal/config/defaults.go).
## Storage And Archive Foundations
Narratio now includes configuration and path-model foundations for archive support, plus implemented prepare-stage S3 audio input.
Implemented foundations:
- `pipeline.storage.s3` config shape (`bucket`, `root_prefix`, `region`, `endpoint`, `force_path_style`)
- `pipeline.storage.s3` config shape (`bucket`, `root_prefix`, `region`, `endpoint`, `force_path_style`, `access_key_id_env`, `secret_access_key_env`)
- `pipeline.spool` config shape (`root`, `delete_audio_after_archive`)
- `pipeline.archive` config shape (`enabled`, `upload_run`, `promote_artifacts`)
- promotion-rule validation (`from`/`to` required, relative-only paths, traversal rejected)
@@ -83,6 +85,8 @@ Implemented foundations:
Current defaults:
- `pipeline.storage.s3.root_prefix`: `dnd`
- `pipeline.storage.s3.access_key_id_env`: `OBJECT_STORAGE_KEY_ID`
- `pipeline.storage.s3.secret_access_key_env`: `OBJECT_STORAGE_KEY`
- `pipeline.workspace.cleanup_after_archive`: `false`
- `pipeline.spool.root`: `/var/spool/narratio`
- `pipeline.spool.delete_audio_after_archive`: `false`
@@ -110,7 +114,8 @@ Current boundaries:
- `pipeline.workspace.cleanup_after_archive: true` removes only the run-scoped local workdir after successful archive commit
- cleanup executes only after all selected stages for the command invocation succeed
- cleanup does not run for failed, incomplete, skipped, or unarchived runs
- local development `audio_dir`/`audio_files` source inputs are never deleted by spool cleanup
- local development `audio_dir`/`audio_files` source inputs are never deleted by spool cleanup
- S3 credentials are resolved from configured env-var names when both are present; if either is missing, Narratio falls back to the AWS SDK default credential chain
S3 input details and current boundaries are documented in [docs/s3-audio-input.md](docs/s3-audio-input.md).
@@ -364,10 +369,16 @@ For the initial implementation, only `session_recap` generation is supported.
Analyze-stage session recap behavior:
- available transcript input sources for configured artifacts: `processed_transcript`, `normalized_transcript`, `trimmed_transcript`
- preferred transcript artifact source IDs:
- `narratio.transcript.polished`
- `narratio.transcript.full`
- `narratio.transcript.trimmed`
- backward-compatible aliases remain supported:
- `processed_transcript`
- `normalized_transcript`
- `trimmed_transcript`
- session recap should use gameplay-only transcript input (`source: trimmed_transcript`)
- Narratio resolves `trimmed_transcript` from trim manifest output (`transcript_trimmed`) or fallback `transcripts/trimmed.json`
- Narratio resolves `normalized_transcript` from normalize manifest output (`transcript_normalized`) or fallback `transcripts/normalized.json`
- Narratio resolves transcript inputs from the artifact resolver (manifest producer outputs first, then canonical session paths)
- missing trimmed transcript fails clearly and advises running trim stage first
- `normalized_transcript` is the preferred full-transcript source for future table/meta-analysis artifacts
- `processed_transcript` remains supported for advanced/debug use cases

View File

@@ -114,6 +114,7 @@ CLI pipeline config path resolution:
- when `--config` is omitted, Narratio searches defaults in order:
- `/usr/local/etc/narratio/pipeline.yml`
- `/etc/narratio/pipeline.yml`
- default values are centralized in `internal/config/defaults.go`
CLI session config path resolution:
@@ -151,6 +152,8 @@ Storage and archive foundations:
- `region`
- `endpoint`
- `force_path_style` (default `false`)
- `access_key_id_env` (default `OBJECT_STORAGE_KEY_ID`)
- `secret_access_key_env` (default `OBJECT_STORAGE_KEY`)
- `pipeline.spool.root` defaults to `/var/spool/narratio`
- `pipeline.workspace.cleanup_after_archive` defaults to `false`
- `pipeline.spool.delete_audio_after_archive` defaults to `false`
@@ -176,7 +179,9 @@ Session input foundations:
Cross-config validation scope:
- `pipeline.storage.s3.bucket` is required only when an S3-dependent feature is explicitly configured (for current foundations, that includes `session.inputs.audio_s3`, and archive upload intent when using `storage.backend: s3`)
- no AWS credentials are stored in Narratio config; credential resolution remains an external runtime concern
- no AWS credential values are stored in Narratio config; only env-var names are configured
- when both configured credential env vars resolve to non-empty values, the S3 backend uses them as static credentials
- when either configured credential value is missing, the S3 backend falls back to the AWS SDK default credential chain
Remote object-store backend scope:

View File

@@ -0,0 +1,130 @@
# Workspace Architecture Implementation Plan (Status)
This document tracks the implemented workspace architecture and remaining work for v1.0.
## Current Architecture (Implemented)
Narratio now uses a canonical campaign-aware local layout:
```text
{workspace.root}/work/{campaign_id}/{session_id}/
manifest.json
current/
manifest.json
run_id.txt
inputs/
transcripts/
artifacts/
reports/
logs/
config/
runs/
{run_id}/
manifest.json
{stage}/
outputs/
logs/
reports/
config/
scratch/
```
Core behavior:
- Session manifest remains the skip/resume source of truth.
- Each invocation creates a run manifest at `runs/{run_id}/manifest.json`.
- Stage execution writes run-local artifacts and promotes durable outputs to canonical session paths.
- Archive uploads run records under `runs/{run_id}/`, applies promotion rules, then publishes `current/manifest.json` and `current/run_id.txt`.
- Analyze input resolution uses centralized artifact IDs with alias support.
- Forced upstream reruns mark downstream succeeded stages `stale` so later runs do not skip stale outputs.
## Section 4 Sequence Status
### Step 1: Campaign-aware session path model
Status: complete.
Implemented:
- Campaign-aware session and run path helpers.
- Campaign-aware artifact-store layout APIs.
- Canonical session manifest pathing under `work/{campaign}/{session}`.
### Step 2: Session manifest + run manifest scaffolding
Status: complete.
Implemented:
- Invocation-scoped run manifest type and store methods.
- Runner creates/saves run manifests per invocation.
- Session manifest remains authoritative for idempotent stage skipping.
### Step 3: Run-local stage execution + promotion
Status: complete.
Implemented:
- Run-local stage directory layout under `runs/{run_id}/{stage}`.
- Shared helpers for run-local output mapping and promotion to canonical durable paths.
- Producer run provenance recorded on durable artifact outputs.
### Step 4: Archive alignment
Status: complete.
Implemented:
- Canonical run-root/session-root resolution.
- Deterministic run-file collection and promotion source resolution.
- Current-pointer publication ordering retained (`current/manifest.json` then `current/run_id.txt`).
### Step 5: Artifact registry/resolver (analyze first consumer)
Status: complete.
Implemented:
- Central artifact resolver with canonical IDs:
- `narratio.transcript.merged`
- `narratio.transcript.polished`
- `narratio.transcript.full`
- `narratio.transcript.trimmed`
- `narratio.bounds.session`
- `narratio.artifact.session_recap`
- Backward-compatible aliases:
- `processed_transcript`
- `normalized_transcript`
- `trimmed_transcript`
- Analyze stage switched to resolver-based source resolution.
### Step 6: Minimal downstream invalidation for forced reruns
Status: complete.
Implemented:
- Deterministic downstream invalidation based on canonical stage order.
- On forced successful rerun of stage `X`, downstream succeeded stages are marked `stale`.
- Resume and non-forced runs naturally re-execute stale stages.
### Step 7: Legacy layout migration strategy
Status: intentionally skipped.
Decision:
- Automatic migration and legacy fallback compatibility are intentionally not implemented.
- The codebase targets canonical-only local layout behavior.
- Legacy local workspace state, if present, should be recreated or migrated manually outside Narratio.
## Remaining Work (v1.0)
No required workspace/run-history migration steps remain from Section 4.
Possible future enhancements (non-blocking):
- Full checksum/input-graph stale detection.
- Optional retention-policy expansion for run-history cleanup.
- Broader artifact-resolver adoption across additional stage consumers.

View File

@@ -0,0 +1,744 @@
# Narratio Workspace, Run History, and Artifact Resolution Architecture
## 1. Purpose
This document defines the intended v1.0 architecture for Narratio's local workspace layout, run history model, durable session outputs, manifest responsibilities, and artifact resolution contract.
Narratio is an idempotent session orchestrator. The command:
```bash
narratio run --session-id 2026-05-07
```
means "bring the identified session to its desired completed state." It does **not** mean "always create an entirely new independent output tree and ignore prior session state."
This distinction drives the architecture:
* A **session** is the durable domain object and idempotency boundary.
* A **run** is an execution attempt that may update the session's durable state.
* Durable outputs live at the session level.
* Run-specific outputs, logs, generated configs, scratch files, and diagnostics live under `runs/{run_id}/`.
* Successful stage outputs are promoted from run-local locations into canonical session-level locations.
* The session manifest records current durable state.
* Run manifests record execution history and debugging/provenance details.
This model intentionally mirrors the S3 archive model: session-level current artifacts are distinct from run-record history.
## 2. Core Concepts
### 2.1 Session
A session is the stable unit of work identified by `campaign_id` and `session_id`.
Examples:
```text
campaign_id = dilfs
session_id = 2026-05-07
```
The session directory represents the current durable local state for that session. Re-running Narratio for the same session should consult this state, skip already-completed stages by default, and produce no changes unless work is incomplete, stale, forced, or explicitly selected.
### 2.2 Run
A run is a particular execution attempt identified by a generated `run_id`, for example:
```text
20260517T174748Z-abcd1234
```
A run may execute all stages or only a sparse subset of stages. Sparse runs are expected and desirable when the user invokes `--force`, `run-stage`, or a stage-limited command.
Run directories are provenance/debug records. They should reflect what actually happened during that invocation, not a synthetic complete pipeline layout.
### 2.3 Durable Output
A durable output is a canonical session-level artifact intended for later stages, user consumption, archive promotion, or future idempotency decisions.
Examples:
```text
transcripts/merged.json
transcripts/processed.json
transcripts/normalized.json
transcripts/trimmed.json
artifacts/session_recap.md
```
Durable outputs live directly under the session directory, not under a particular run directory.
### 2.4 Run-Local Output
A run-local output is the file initially produced by a stage during a specific run. After validation, durable outputs are promoted from run-local paths to session-level canonical paths.
Run-local outputs, logs, generated configs, reports, and scratch files should remain under:
```text
runs/{run_id}/{stage}/...
```
## 3. Local Workspace Layout
The canonical local workspace layout is:
```text
{workspace.root}/work/{campaign_id}/{session_id}/
manifest.json
current/
manifest.json
run_id.txt
inputs/
transcripts/
artifacts/
reports/
logs/
config/
runs/
{run_id}/
manifest.json
prepare/
transcribe/
merge/
polish/
normalize/
trim/
analyze/
archive/
notify/
```
Not every directory must exist at all times. Directories should be created idempotently when needed.
### 3.1 Session Root
The session root is:
```text
{workspace.root}/work/{campaign_id}/{session_id}/
```
The session root is the stable local home for the session. It is the default base for resolving canonical artifact paths.
The only files that should live directly in the session root are core session-state files, primarily:
```text
manifest.json
```
Lock files may also be session-root scoped if the implementation uses file locks there, but transient locks should not be treated as durable artifacts.
### 3.2 Session-Level Canonical Directories
The following directories contain current durable session state:
```text
inputs/
transcripts/
artifacts/
reports/
logs/
config/
current/
```
Recommended meanings:
| Directory | Purpose |
| -------------- | ----------------------------------------------------------------------------- |
| `inputs/` | Materialized or copied input files used by the current durable session state. |
| `transcripts/` | Canonical transcript tiers. |
| `artifacts/` | User-facing and machine-readable generated artifacts. |
| `reports/` | Canonical stage reports worth preserving at the session level. |
| `logs/` | Optional session-level logs or promoted/latest logs. |
| `config/` | Optional session-level generated config snapshots or promoted/latest configs. |
| `current/` | Current published session pointers, mirroring the archive backend. |
Canonical durable outputs should use stable paths under these directories.
### 3.3 Run History Directory
Run history lives under:
```text
{workspace.root}/work/{campaign_id}/{session_id}/runs/{run_id}/
```
Each run directory records what happened during that invocation. A run may contain all stage directories or only a sparse subset.
Example full run:
```text
runs/20260517T174748Z-abcd1234/
manifest.json
prepare/
transcribe/
merge/
polish/
normalize/
trim/
analyze/
archive/
notify/
```
Example sparse forced analyze run:
```text
runs/20260518T030000Z-efgh5678/
manifest.json
analyze/
```
Example sparse polish-through-analyze rerun:
```text
runs/20260518T041500Z-a1b2c3d4/
manifest.json
polish/
normalize/
trim/
analyze/
```
Run directories should not create stage folders for stages that were not selected, executed, skipped, or otherwise considered during that run unless there is a clear diagnostic reason to do so.
### 3.4 Stage Run-Local Directories
Each stage receives a run-local directory:
```text
runs/{run_id}/{stage}/
```
Within that stage directory, the stage may use subdirectories such as:
```text
outputs/
logs/
reports/
config/
scratch/
```
For example:
```text
runs/{run_id}/polish/
outputs/transcripts/processed.json
reports/audita.polish.report.json
logs/stdout.log
logs/stderr.log
config/audita.polish.generated.yml
scratch/
```
The exact internal layout of a stage directory may vary by stage, but it should be deterministic, documented, and generated through centralized path helpers rather than ad hoc path joins.
## 4. Promotion Model
Narratio uses stage-level promotion with immediate promotion after successful validation.
The stage lifecycle is:
1. Resolve required inputs from the current session state and/or run-local context.
2. Create the run-local stage directory.
3. Execute the stage, writing outputs under `runs/{run_id}/{stage}/...`.
4. Validate run-local outputs.
5. Promote durable outputs into session-level canonical paths.
6. Update the session manifest.
7. Update the run manifest.
Promotion means an atomic or effectively atomic copy/rename from a run-local path to a session-level canonical path.
Example:
```text
runs/{run_id}/polish/outputs/transcripts/processed.json
```
is promoted to:
```text
transcripts/processed.json
```
Promotion should be safe and deterministic:
* Validate before promotion.
* Write promoted files atomically where possible.
* Never leave partially written durable outputs.
* Record the producing `run_id` in the session manifest.
* Preserve run-local files for debugging unless retention policy deletes them.
## 5. Promotion Policy: Option A
Narratio uses immediate stage-level promotion.
If a selected stage succeeds, its durable outputs are promoted immediately, even if a later selected stage fails.
Example:
```bash
narratio run --session-id 2026-05-07 --force --stages polish,normalize,trim,analyze
```
If `polish` succeeds and `normalize` fails:
* `transcripts/processed.json` may be updated from the new run.
* `normalize`, `trim`, and `analyze` should not be marked succeeded for the new input state.
* Downstream outputs may now be stale relative to the newly promoted polished transcript.
This policy is simpler, transparent, and consistent with stage-level resumability. It does require explicit stale/invalidation handling.
## 6. Stale and Invalidation Semantics
Full checksum-based stale detection may be implemented later. Before that exists, Narratio should still use a simple deterministic invalidation rule for forced or explicit upstream reruns.
When a stage is successfully re-executed and promoted, downstream stages should be marked stale unless they are also re-executed successfully in the same command invocation.
Example stage order:
```text
prepare -> transcribe -> merge -> polish -> normalize -> trim -> analyze -> archive -> notify
```
If `polish` is forced and promoted, then the following downstream stages should be invalidated unless rerun successfully:
```text
normalize
trim
analyze
archive
notify
```
A stale stage is not equivalent to a failed stage. It means its current durable outputs may no longer correspond to current upstream inputs or configuration.
Minimum manifest state model:
```text
pending
running
succeeded
failed
skipped
stale
```
If adding a new `stale` state is too invasive for v1.0, the implementation should at least record stale metadata or clear downstream success markers in a way that prevents accidental idempotent skips based on obsolete outputs.
## 7. Manifest Responsibilities
Narratio should distinguish between session manifests and run manifests.
The same underlying Go types may be reused where practical, but the concepts should remain separate.
### 7.1 Session Manifest
Path:
```text
{workspace.root}/work/{campaign_id}/{session_id}/manifest.json
```
The session manifest answers:
```text
What is the current durable state of this session?
```
It should record:
* campaign ID
* session ID
* current or latest run ID
* current stage states
* canonical durable output refs
* artifact IDs and paths
* producing run ID for each current stage output
* relevant input/config checksums when available
* stale/invalidated stage information
* archive/current publication metadata
A session's durable state may be a composite of multiple runs.
For example:
```text
transcripts/merged.json produced by run A
transcripts/processed.json produced by run B
transcripts/normalized.json produced by run B
transcripts/trimmed.json produced by run B
artifacts/session_recap.md produced by run C
```
This is valid and expected.
### 7.2 Run Manifest
Path:
```text
{workspace.root}/work/{campaign_id}/{session_id}/runs/{run_id}/manifest.json
```
The run manifest answers:
```text
What happened during this specific execution attempt?
```
It should record:
* run ID
* campaign ID
* session ID
* command mode and selected stages
* force flags or stage selection flags
* stages considered during this run
* stages executed during this run
* stages skipped during this run and reasons
* run-local output paths
* promoted output paths
* logs
* reports
* generated configs
* timings
* errors
* non-secret subprocess invocation metadata
Run manifests are primarily for debugging, auditability, and archive history.
## 8. Idempotency and Resume Behavior
The idempotency boundary is the session, not the run.
By default:
```bash
narratio run --session-id 2026-05-07
```
should consult the session manifest and skip stages that are already succeeded and not stale.
If all stages are already complete, the command should execute zero stages and report that the session is already complete.
Forced execution creates a new run record but updates session-level durable state only for stages that actually succeed and promote outputs.
Examples:
```bash
narratio run --session-id 2026-05-07 --force
```
Creates a new run and attempts to re-execute the selected/default stage set.
```bash
narratio run-stage --session-id 2026-05-07 analyze --force
```
Creates a sparse run that executes only `analyze`, then promotes updated analysis artifacts if successful.
```bash
narratio resume --session-id 2026-05-07
```
Uses the session manifest to determine what remains incomplete or stale. Resume does not need to resume the same `run_id` unless the implementation explicitly supports resuming an interrupted active run.
## 9. Artifact Resolution Contract
Narratio should provide a first-class artifact registry and resolver.
The resolver maps symbolic artifact source names to canonical session-level paths and manifest output kinds.
Stages and adapters should not hardcode path fragments when resolving cross-stage inputs. They should ask the artifact resolver for the current durable artifact by ID.
### 9.1 Canonical Artifact IDs
Preferred artifact IDs should be namespaced:
```text
narratio.transcript.merged
narratio.transcript.polished
narratio.transcript.full
narratio.transcript.trimmed
narratio.bounds.session
narratio.artifact.session_recap
```
Recommended initial registry:
| Artifact ID | Canonical Path | Producer Stage | Output Kind | Meaning |
| --------------------------------- | ------------------------------- | -------------- | ------------------------ | ------------------------------------- |
| `narratio.transcript.merged` | `transcripts/merged.json` | `merge` | `transcript_merged` | Deterministic Seriatim merge. |
| `narratio.transcript.polished` | `transcripts/processed.json` | `polish` | `transcript_processed` | Full Audita-polished transcript. |
| `narratio.transcript.full` | `transcripts/normalized.json` | `normalize` | `transcript_normalized` | Preferred full normalized transcript. |
| `narratio.transcript.trimmed` | `transcripts/trimmed.json` | `trim` | `transcript_trimmed` | Gameplay-only transcript. |
| `narratio.bounds.session` | `artifacts/session_bounds.json` | `trim` | `session_bounds` | Trim bounds selected for the session. |
| `narratio.artifact.session_recap` | `artifacts/session_recap.md` | `analyze` | `artifact_session_recap` | Generated session recap. |
### 9.2 Backward-Compatible Aliases
Existing source names should remain supported:
| Legacy Source | Preferred Artifact ID |
| ----------------------- | ------------------------------ |
| `processed_transcript` | `narratio.transcript.polished` |
| `normalized_transcript` | `narratio.transcript.full` |
| `trimmed_transcript` | `narratio.transcript.trimmed` |
These aliases may be supported silently for v1.0. Documentation should prefer namespaced IDs.
### 9.3 Resolver Behavior
Artifact resolution should follow this order:
1. Normalize aliases to canonical artifact IDs.
2. Look for a current output reference in the session manifest.
3. Fall back to the canonical session-level path.
4. If the artifact is required, fail clearly if missing.
5. If the artifact is optional and missing, omit it from the downstream invocation.
6. Validate the artifact using the expected content validator.
7. Return a resolved artifact record containing ID, path, producer stage, output kind, and provenance.
Example conceptual result:
```json
{
"id": "narratio.transcript.trimmed",
"path": "/var/lib/narratio/work/dilfs/2026-05-07/transcripts/trimmed.json",
"producer_stage": "trim",
"producer_run_id": "20260517T174748Z-abcd1234",
"output_kind": "transcript_trimmed",
"content_type": "application/json"
}
```
### 9.4 Artifact Validation
Transcript artifacts must be valid JSON with a top-level `segments` array.
Markdown/text artifacts must exist and be non-empty when required.
Bounds artifacts must match the expected bounds schema and refer to segment IDs in the same transcript ID space used by the trim stage.
Validation should happen before a resolved artifact is passed to another stage or external subprocess.
## 10. Analyze Stage Implications
The analyze stage should consume artifacts through the artifact resolver.
Preferred Scriptorium config shape:
```yaml
scriptorium:
artifacts:
session_recap:
enabled: true
prompt_id: "dnd.session_recap"
output_path: "artifacts/session_recap.md"
inputs:
transcript:
source: "narratio.transcript.trimmed"
required: true
```
Additional artifacts can choose different transcript tiers:
```yaml
scriptorium:
artifacts:
table_summary:
enabled: true
prompt_id: "dnd.table_summary"
output_path: "artifacts/table_summary.md"
inputs:
transcript:
source: "narratio.transcript.full"
required: true
```
For v1.0, Narratio does not need a generic DAG engine. It may execute configured analyze artifacts in deterministic order and allow later artifacts to consume earlier artifacts only when that relationship is explicit and unambiguous.
Rules:
* Artifact inputs resolve from current session-level durable state.
* Outputs are first written run-locally.
* Successful analyze outputs are promoted to session-level `artifacts/` paths.
* Manifest output refs record the producing run ID.
* Optional inputs are omitted when unavailable.
* Required missing inputs fail before invoking Scriptorium.
## 11. Archive Alignment
Local workspace semantics should mirror archive semantics.
Local session-level durable paths:
```text
work/{campaign}/{session}/transcripts/trimmed.json
work/{campaign}/{session}/artifacts/session_recap.md
work/{campaign}/{session}/current/manifest.json
work/{campaign}/{session}/current/run_id.txt
work/{campaign}/{session}/runs/{run_id}/...
```
should map naturally to remote archive paths:
```text
{root_prefix}/campaigns/{campaign}/sessions/{session}/transcripts/trimmed.json
{root_prefix}/campaigns/{campaign}/sessions/{session}/artifacts/session_recap.md
{root_prefix}/campaigns/{campaign}/sessions/{session}/current/manifest.json
{root_prefix}/campaigns/{campaign}/sessions/{session}/current/run_id.txt
{root_prefix}/campaigns/{campaign}/sessions/{session}/runs/{run_id}/...
```
The archive stage should publish run records and promoted current artifacts consistently with the local model.
`current/run_id.txt` remains the effective commit marker for the archived current session state.
## 12. Path Helper Requirements
All code should use centralized path helpers for workspace paths.
Stage code should not manually assemble durable cross-stage paths using raw string joins except through the path model.
Recommended helper surface:
```text
SessionRoot(campaignID, sessionID)
SessionManifestPath(campaignID, sessionID)
SessionCurrentDir(campaignID, sessionID)
SessionTranscriptsDir(campaignID, sessionID)
SessionArtifactsDir(campaignID, sessionID)
SessionReportsDir(campaignID, sessionID)
SessionLogsDir(campaignID, sessionID)
SessionConfigDir(campaignID, sessionID)
RunsDir(campaignID, sessionID)
RunRoot(campaignID, sessionID, runID)
RunManifestPath(campaignID, sessionID, runID)
RunStageDir(campaignID, sessionID, runID, stage)
RunStageOutputsDir(campaignID, sessionID, runID, stage)
RunStageLogsDir(campaignID, sessionID, runID, stage)
RunStageReportsDir(campaignID, sessionID, runID, stage)
RunStageConfigDir(campaignID, sessionID, runID, stage)
CanonicalArtifactPath(campaignID, sessionID, artifactID)
```
Path helpers should enforce safe relative paths for configured output paths:
* reject absolute paths unless explicitly allowed for a particular config field
* reject `..` traversal
* normalize separators
* preserve deterministic output paths
## 13. Directory Creation Policy
Directory creation should be centralized and idempotent.
Recommended policy:
* `prepare` ensures the baseline session directory structure exists.
* Every stage also calls shared layout helpers to ensure its required run-local directories exist before writing.
* `run-stage` should not depend on a prior `prepare` invocation merely to create folders.
* Missing directories should be created with appropriate permissions.
* Directory creation should not imply stage success.
This provides consistent layout while keeping direct stage execution robust.
## 14. Cleanup and Retention
Cleanup must preserve the distinction between durable session state and run history.
Workspace cleanup after successful archive may remove selected local directories only according to explicit configuration.
Potential retention policies:
```text
keep_all_runs
keep_failed_runs
keep_last_n_runs
delete_run_after_success
```
For v1.0, conservative retention is preferred:
* Do not delete durable session-level outputs unless explicitly requested.
* Do not delete failed run directories by default.
* If cleanup is enabled, remove only documented run-scoped or spool-scoped paths.
* Local development audio inputs must never be deleted by workspace cleanup.
## 15. Canonical-Only Layout Policy
Narratio now supports only the canonical campaign-aware layout:
```text
{workspace.root}/work/{campaign_id}/{session_id}/manifest.json
{workspace.root}/work/{campaign_id}/{session_id}/runs/{run_id}/...
```
Legacy session-only layout compatibility is intentionally not implemented.
If legacy workspace data exists, operators should recreate or manually migrate that data outside Narratio before running v1.0 commands.
## 16. Documentation Updates Required
The following documentation should be updated to reflect this architecture:
* `README.md`
* `docs/architecture.md`
* a dedicated workspace/run-history document, such as this file
* S3/archive documentation
* analyze/artifact configuration documentation
* example pipeline files
Documentation should consistently use the following terms:
| Term | Meaning |
| ---------------- | ---------------------------------------------------------------------- |
| Session | Durable domain object and idempotency boundary. |
| Run | Execution attempt that may update session state. |
| Durable output | Canonical current session-level output. |
| Run-local output | Output produced inside a specific run directory before promotion. |
| Promotion | Validated copy/rename from run-local output to durable session output. |
| Session manifest | Current durable state of the session. |
| Run manifest | Execution record for a particular run. |
| Artifact ID | Symbolic source name resolved by the artifact registry. |
## 17. Architectural Invariants
The following invariants should hold after implementation:
1. `session_id` remains the idempotency boundary for normal operator commands.
2. `run_id` identifies an execution attempt, not the primary durable workspace.
3. Session-level canonical artifacts are the default inputs for downstream stages.
4. Run-local outputs are promoted only after validation.
5. A session's current durable state may be composed of outputs from multiple runs.
6. Sparse run directories are valid and expected.
7. The session manifest records current stage/artifact state and producer run IDs.
8. The run manifest records what happened during one invocation.
9. Artifact consumers resolve symbolic artifact IDs through a registry/resolver.
10. Local workspace semantics mirror S3 archive semantics.
11. Directory creation is centralized and idempotent.
12. Stage code uses path helpers rather than ad hoc path construction.
13. Forced upstream reruns invalidate downstream stage success unless downstream stages are rerun successfully.
14. Cleanup never removes durable session outputs or local development inputs unless explicitly configured to do so.
## 18. Implementation Guidance
A practical implementation sequence is:
1. Add this architecture document.
2. Add or revise path model helpers for session roots, run roots, stage directories, and canonical artifact paths.
3. Introduce session manifest versus run manifest concepts.
4. Route stage outputs through run-local directories.
5. Add promotion helpers with validation and atomic writes.
6. Update existing stages to promote durable outputs to session-level canonical paths.
7. Add artifact registry and resolver.
8. Update analyze to use artifact IDs and aliases.
9. Add simple downstream stale invalidation for forced upstream reruns.
10. Align archive/local path behavior and documentation.
11. Update examples and README.
12. Add tests for idempotency, sparse forced runs, promotion, manifest provenance, and artifact resolution.
This sequence intentionally avoids introducing a generic DAG engine. The v1.0 goal is a clear, deterministic, stage-oriented orchestrator with stable session-level outputs and inspectable run history.

View File

@@ -283,9 +283,9 @@ Session recap:
```bash
scriptorium run \
--prompt dnd.session_recap \
--input transcript=/work/session-42/transcript.polished.md \
--input glossary=/work/session-42/glossary.yml \
--out /work/session-42/artifacts/session_recap.md
--input transcript=/work/campaign-7/session-42/transcript.polished.md \
--input glossary=/work/campaign-7/session-42/glossary.yml \
--out /work/campaign-7/session-42/artifacts/session_recap.md
```
Structured events:
@@ -293,8 +293,8 @@ Structured events:
```bash
scriptorium run \
--prompt dnd.structured_events \
--input transcript=/work/session-42/transcript.polished.md \
--out /work/session-42/artifacts/structured_events.json
--input transcript=/work/campaign-7/session-42/transcript.polished.md \
--out /work/campaign-7/session-42/artifacts/structured_events.json
```
Glossary suggestions:
@@ -302,9 +302,9 @@ Glossary suggestions:
```bash
scriptorium run \
--prompt dnd.glossary_suggestions \
--input transcript=/work/session-42/transcript.polished.md \
--input previous_recap=/work/session-41/artifacts/session_recap.md \
--out /work/session-42/artifacts/glossary_suggestions.md
--input transcript=/work/campaign-7/session-42/transcript.polished.md \
--input previous_recap=/work/campaign-7/session-41/artifacts/session_recap.md \
--out /work/campaign-7/session-42/artifacts/glossary_suggestions.md
```
Player-facing summary:
@@ -312,9 +312,9 @@ Player-facing summary:
```bash
scriptorium run \
--prompt dnd.player_summary \
--input transcript=/work/session-42/transcript.polished.md \
--input structured_events=/work/session-42/artifacts/structured_events.json \
--out /work/session-42/artifacts/player_summary.md
--input transcript=/work/campaign-7/session-42/transcript.polished.md \
--input structured_events=/work/campaign-7/session-42/artifacts/structured_events.json \
--out /work/campaign-7/session-42/artifacts/player_summary.md
```
## 21. Non-Goals

File diff suppressed because it is too large Load Diff

View File

@@ -1,35 +0,0 @@
# S3 Archive Foundations Runbook
This runbook documents the currently implemented storage/archive foundations and the boundaries of current behavior.
## Implemented Now
- config modeling for:
- `pipeline.storage.s3`
- `pipeline.spool`
- `pipeline.archive`
- `session.inputs.audio_s3`
- promotion rule validation for safe relative paths
- run ID generation and path/key helper functions
- manifest run/path identity fields
- remote storage backend layer:
- object-store interface (`List`, `Download`, `Upload`, `Exists`)
- fake backend for deterministic tests
- S3-compatible backend using AWS SDK v2
- config-based backend construction helper
## Not Implemented Yet
- prepare-stage S3 object listing or download
- archive-stage S3 upload or promotion writes
- writing `current/manifest.json` or `current/run_id.txt` in S3
## Operational Notes
- local audio workflows remain the active development path (`audio_dir` or `audio_files`)
- `audio_s3` and local audio config are mutually exclusive
- do not place AWS credentials in Narratio config files
## Next Implementation Target
Use the storage backend layer in prepare-stage session audio discovery/download flow, while preserving local audio input support.

View File

@@ -20,6 +20,8 @@ Not implemented:
- `storage.s3.bucket` must be set when S3 audio input is used.
- `storage.s3.root_prefix` defaults to `dnd`.
- `storage.s3.access_key_id_env` defaults to `OBJECT_STORAGE_KEY_ID`.
- `storage.s3.secret_access_key_env` defaults to `OBJECT_STORAGE_KEY`.
- `spool.root` defaults to `/var/spool/narratio`.
`session.yml`:
@@ -54,7 +56,7 @@ When `inputs.audio_s3.prefix` is configured, `prepare`:
4. downloads selected objects to spool audio:
- `{spool.root}/{campaign}/{session_id}/{run_id}/audio/`
5. materializes audio into workdir audio:
- `{workspace.root}/work/{campaign}/{session_id}/{run_id}/audio/`
- `{workspace.root}/work/{campaign}/{session_id}/runs/{run_id}/audio/`
6. records input provenance in the manifest (bucket, key, metadata, local paths, checksum)
Notes:

View File

@@ -1,58 +0,0 @@
# Storage Backends
This document describes the currently implemented remote object storage backend layer used by Narratio, and its intended role in later prepare/archive work.
## Implemented
Remote object store abstraction:
- `List(ctx, prefix)`
- `Download(ctx, key, localPath)`
- `Upload(ctx, localPath, key, opts)`
- `Exists(ctx, key)`
Object metadata model includes:
- key
- size
- ETag (provider metadata only)
- last modified time when available
Backends:
- fake storage backend for deterministic tests
- S3-compatible backend implemented with AWS SDK for Go v2
Construction:
- config-based constructor builds S3 backend from `pipeline.storage.s3` values:
- bucket
- region
- endpoint
- force_path_style
## Key Invariant
- callers pass full bucket-relative object keys
- storage backends do not prepend `root_prefix`
- storage backends do not infer campaign/session/run paths
S3 session/run key builders remain separate and continue to live outside backend implementations.
## Security Boundary
- do not store AWS credentials in Narratio config
- AWS credentials are resolved through standard AWS SDK credential chains
- AWS SDK-specific types remain isolated to the storage adapter package
## Testing
- fake storage tests cover list/download/upload/exists and error paths
- S3 backend tests use injected fake S3 API clients
- tests do not require live S3 services, AWS credentials, or network access
## Not Implemented Yet
- prepare-stage S3 object listing or downloads
- archive-stage S3 uploads or promotion writes
- writing `current/manifest.json` or `current/run_id.txt` to S3

View File

@@ -8,6 +8,9 @@ storage:
bucket: "my-dnd-archive"
root_prefix: "dnd"
region: "us-east-1"
# Optional credential env-var names (defaulted when omitted):
# access_key_id_env: "OBJECT_STORAGE_KEY_ID"
# secret_access_key_env: "OBJECT_STORAGE_KEY"
spool:
root: "/var/spool/narratio"

View File

@@ -11,6 +11,7 @@ import (
"time"
awsconfig "github.com/aws/aws-sdk-go-v2/config"
"github.com/aws/aws-sdk-go-v2/credentials"
"github.com/aws/aws-sdk-go-v2/service/s3"
"github.com/aws/aws-sdk-go-v2/service/s3/types"
"github.com/aws/smithy-go"
@@ -35,6 +36,8 @@ type s3ClientOptions struct {
Region string
Endpoint string
ForcePathStyle bool
AccessKeyID string
SecretKey string
}
var newS3Client = func(ctx context.Context, opts s3ClientOptions) (s3API, error) {
@@ -42,6 +45,15 @@ var newS3Client = func(ctx context.Context, opts s3ClientOptions) (s3API, error)
if strings.TrimSpace(opts.Region) != "" {
loadOpts = append(loadOpts, awsconfig.WithRegion(strings.TrimSpace(opts.Region)))
}
if strings.TrimSpace(opts.AccessKeyID) != "" && strings.TrimSpace(opts.SecretKey) != "" {
loadOpts = append(loadOpts, awsconfig.WithCredentialsProvider(
credentials.NewStaticCredentialsProvider(
strings.TrimSpace(opts.AccessKeyID),
strings.TrimSpace(opts.SecretKey),
"",
),
))
}
awsCfg, err := awsconfig.LoadDefaultConfig(ctx, loadOpts...)
if err != nil {
return nil, fmt.Errorf("load aws config: %w", err)
@@ -67,6 +79,8 @@ func NewS3BackendFromConfig(ctx context.Context, cfg config.StorageS3Config) (*S
Region: cfg.Region,
Endpoint: cfg.Endpoint,
ForcePathStyle: cfg.ForcePathStyle,
AccessKeyID: s3CredentialFromEnv(orDefaultEnvName(cfg.AccessKeyIDEnv, config.DefaultS3AccessKeyIDEnv)),
SecretKey: s3CredentialFromEnv(orDefaultEnvName(cfg.SecretKeyEnv, config.DefaultS3SecretAccessKeyEnv)),
})
if err != nil {
return nil, fmt.Errorf("build s3 client: %w", err)
@@ -78,6 +92,26 @@ func NewS3BackendFromConfig(ctx context.Context, cfg config.StorageS3Config) (*S
}, nil
}
func s3CredentialFromEnv(envVarName string) string {
name := strings.TrimSpace(envVarName)
if name == "" {
return ""
}
value, ok := os.LookupEnv(name)
if !ok {
return ""
}
return strings.TrimSpace(value)
}
func orDefaultEnvName(name, fallback string) string {
trimmed := strings.TrimSpace(name)
if trimmed == "" {
return fallback
}
return trimmed
}
// List returns objects under prefix.
func (b *S3Backend) List(ctx context.Context, prefix string) ([]ObjectInfo, error) {
normalizedPrefix := normalizeObjectKey(prefix)

View File

@@ -187,6 +187,8 @@ func TestS3BackendExistsNotFound(t *testing.T) {
func TestNewS3BackendFromConfigUsesClientOptions(t *testing.T) {
original := newS3Client
t.Cleanup(func() { newS3Client = original })
t.Setenv("OBJECT_STORAGE_KEY_ID", "id-123")
t.Setenv("OBJECT_STORAGE_KEY", "secret-abc")
var got s3ClientOptions
newS3Client = func(_ context.Context, opts s3ClientOptions) (s3API, error) {
@@ -209,6 +211,9 @@ func TestNewS3BackendFromConfigUsesClientOptions(t *testing.T) {
if got.Region != "us-east-1" || got.Endpoint != "http://localhost:9000" || !got.ForcePathStyle {
t.Fatalf("client options = %#v, want region/endpoint/path-style values", got)
}
if got.AccessKeyID != "id-123" || got.SecretKey != "secret-abc" {
t.Fatalf("client options credentials = %#v, want env-resolved static credentials", got)
}
}
func TestNewS3BackendFromConfigRequiresBucket(t *testing.T) {
@@ -218,6 +223,30 @@ func TestNewS3BackendFromConfigRequiresBucket(t *testing.T) {
}
}
func TestNewS3BackendFromConfigFallsBackWhenCredentialEnvMissing(t *testing.T) {
original := newS3Client
t.Cleanup(func() { newS3Client = original })
var got s3ClientOptions
newS3Client = func(_ context.Context, opts s3ClientOptions) (s3API, error) {
got = opts
return &fakeS3API{}, nil
}
_, err := NewS3BackendFromConfig(context.Background(), config.StorageS3Config{
Bucket: "my-archive",
Region: "us-east-1",
AccessKeyIDEnv: "MISSING_ACCESS_KEY_ID",
SecretKeyEnv: "MISSING_SECRET_KEY",
})
if err != nil {
t.Fatalf("NewS3BackendFromConfig() error = %v", err)
}
if got.AccessKeyID != "" || got.SecretKey != "" {
t.Fatalf("client options credentials = %#v, want empty fallback values", got)
}
}
func strPtr(v string) *string { return &v }
func int64Ptr(v int64) *int64 { return &v }

View File

@@ -111,7 +111,7 @@ func TestExecuteRunStageUnknownFails(t *testing.T) {
func TestExecuteRunStageNormalizeIsAccepted(t *testing.T) {
workspaceRoot := t.TempDir()
pipelinePath, sessionPath := writeValidConfigFiles(t, workspaceRoot, "https://example.com/transcribe")
workRoot := filepath.Join(workspaceRoot, "work", "2026-05-03")
workRoot := filepath.Join(workspaceRoot, "work", "sample-campaign", "2026-05-03")
mustWriteTestFile(t, filepath.Join(workRoot, "transcripts", "processed.json"), `{"segments":[{"id":1}]}`)
var stdout bytes.Buffer
@@ -156,7 +156,7 @@ func TestExecuteRunStageTranscribeUsesConfiguredWhisperXServer(t *testing.T) {
t.Fatal("expected whisperx server to be called at least once")
}
outPath := filepath.Join(workspaceRoot, "work", "2026-05-03", "transcripts", "raw", "alice.json")
outPath := filepath.Join(workspaceRoot, "work", "sample-campaign", "2026-05-03", "transcripts", "raw", "alice.json")
data, err := os.ReadFile(outPath)
if err != nil {
t.Fatalf("ReadFile(%q): %v", outPath, err)
@@ -233,7 +233,7 @@ inputs:
_ = os.Chdir(originalWD)
})
workRoot := filepath.Join(workspaceRoot, "work", sessionID)
workRoot := filepath.Join(workspaceRoot, "work", "sample-campaign", sessionID)
mustWriteTestFile(t, filepath.Join(workRoot, "transcripts", "merged.json"), `{"schema":"seriatim-intermediate","segments":[]}`)
mustWriteTestFile(t, filepath.Join(workRoot, "inputs", "glossary.yml"), "[]\n")

View File

@@ -57,7 +57,7 @@ func Plan(ctx context.Context, args []string, out io.Writer) error {
}
store := artifacts.NewLocalStore(cfg.Pipeline.Workspace.Root)
paths, err := store.EnsureLayout(cfg.Session.SessionID)
paths, err := store.EnsureLayoutFor(cfg.Session.Campaign, cfg.Session.SessionID)
if err != nil {
return fmt.Errorf("plan: prepare workdir: %w", err)
}

View File

@@ -36,7 +36,7 @@ func TestPlanCreatesAndReusesWorkdir(t *testing.T) {
t.Fatalf("first output = %q, want totals", got)
}
sessionWorkdir := artifacts.SessionWorkDir(workspaceRoot, "2026-05-03")
sessionWorkdir := artifacts.SessionWorkDirForCampaign(workspaceRoot, "sample-campaign", "2026-05-03")
expectedDirs := []string{
sessionWorkdir,
filepath.Join(sessionWorkdir, "inputs"),
@@ -63,7 +63,7 @@ func TestPlanCreatesAndReusesWorkdir(t *testing.T) {
func TestPlanShowsRunAndSkipFromManifest(t *testing.T) {
workspaceRoot := t.TempDir()
pipelinePath, sessionPath := writeValidConfigFiles(t, workspaceRoot)
manifestPath := filepath.Join(workspaceRoot, "work", "2026-05-03", "manifest.json")
manifestPath := filepath.Join(workspaceRoot, "work", "sample-campaign", "2026-05-03", "manifest.json")
store := &manifest.LocalStore{}
m := manifest.New("2026-05-03", time.Date(2026, 5, 3, 10, 0, 0, 0, time.UTC))

View File

@@ -54,7 +54,7 @@ func runPostArchiveCleanup(ctx context.Context, env *Env, manifestPath string, m
}
workDir := strings.TrimSpace(m.LocalWorkDir)
if workDir == "" {
workDir = artifacts.SessionRunWorkDir(
workDir = artifacts.SessionRunRootForCampaign(
env.Config.Pipeline.Workspace.Root,
strings.TrimSpace(env.Config.Session.Campaign),
strings.TrimSpace(env.Config.Session.SessionID),

View File

@@ -210,7 +210,7 @@ func TestPostArchiveCleanupNotRunWhenPromotionIsMissing(t *testing.T) {
assertExists(t, seed.spoolAudioDir)
assertExists(t, seed.runWorkDir)
assertExists(t, filepath.Join(seed.runWorkDir, "manifest.json"))
assertExists(t, artifacts.SessionRunWorkDir(cfg.Pipeline.Workspace.Root, cfg.Session.Campaign, cfg.Session.SessionID, runID))
assertExists(t, artifacts.SessionRunRootForCampaign(cfg.Pipeline.Workspace.Root, cfg.Session.Campaign, cfg.Session.SessionID, runID))
}
func TestPostArchiveCleanupNotRunWhenCurrentManifestUploadFails(t *testing.T) {
@@ -271,8 +271,8 @@ func cleanupFixtureConfig(t *testing.T) (*config.Config, cleanupSeed) {
cfg.Pipeline.Spool.Root = filepath.Join(t.TempDir(), "spool")
runID := "20260516T010203Z-1a2b3c4d"
runWorkDir := artifacts.SessionRunWorkDir(cfg.Pipeline.Workspace.Root, cfg.Session.Campaign, cfg.Session.SessionID, runID)
otherRunDir := artifacts.SessionRunWorkDir(cfg.Pipeline.Workspace.Root, cfg.Session.Campaign, cfg.Session.SessionID, "20260516T010204Z-5e6f7a8b")
runWorkDir := artifacts.SessionRunRootForCampaign(cfg.Pipeline.Workspace.Root, cfg.Session.Campaign, cfg.Session.SessionID, runID)
otherRunDir := artifacts.SessionRunRootForCampaign(cfg.Pipeline.Workspace.Root, cfg.Session.Campaign, cfg.Session.SessionID, "20260516T010204Z-5e6f7a8b")
spoolAudioDir := artifacts.SessionSpoolAudioDir(cfg.Pipeline.Spool.Root, cfg.Session.Campaign, cfg.Session.SessionID, runID)
mustWriteFile(t, filepath.Join(runWorkDir, "manifest.json"), "{}\n")
@@ -326,7 +326,11 @@ func archiveStageCleanupFixture(t *testing.T) (*config.Config, cleanupSeed, stri
{From: "artifacts/session_recap.md", To: "artifacts/session_recap.md", Required: boolPtr(true)},
},
}
writeArchiveFixtureRunFiles(t, seed.runWorkDir)
writeArchiveFixtureRunFiles(
t,
seed.runWorkDir,
artifacts.SessionWorkDirForCampaign(cfg.Pipeline.Workspace.Root, cfg.Session.Campaign, cfg.Session.SessionID),
)
store := &manifest.LocalStore{}
seedManifest, err := store.Load(context.Background(), manifestPathFor(cfg))
@@ -345,16 +349,19 @@ func archiveStageCleanupFixture(t *testing.T) (*config.Config, cleanupSeed, stri
return cfg, seed, runID
}
func writeArchiveFixtureRunFiles(t *testing.T, runWorkDir string) {
func writeArchiveFixtureRunFiles(t *testing.T, runWorkDir, sessionRoot string) {
t.Helper()
mustWriteFile(t, filepath.Join(runWorkDir, "inputs", "session.yml"), "session_id: 2026-05-03\n")
mustWriteFile(t, filepath.Join(runWorkDir, "transcripts", "raw", "speaker.json"), "{}\n")
mustWriteFile(t, filepath.Join(runWorkDir, "transcripts", "trimmed.json"), "{}\n")
mustWriteFile(t, filepath.Join(runWorkDir, "artifacts", "session_recap.md"), "# recap\n")
mustWriteFile(t, filepath.Join(runWorkDir, "reports", "audita.report.json"), "{}\n")
mustWriteFile(t, filepath.Join(runWorkDir, "config", "audita.generated.yml"), "key: value\n")
mustWriteFile(t, filepath.Join(runWorkDir, "prepare", "inputs", "session.yml"), "session_id: 2026-05-03\n")
mustWriteFile(t, filepath.Join(runWorkDir, "transcribe", "outputs", "transcripts", "raw", "speaker.json"), "{}\n")
mustWriteFile(t, filepath.Join(runWorkDir, "trim", "outputs", "transcripts", "trimmed.json"), "{}\n")
mustWriteFile(t, filepath.Join(runWorkDir, "analyze", "outputs", "artifacts", "session_recap.md"), "# recap\n")
mustWriteFile(t, filepath.Join(runWorkDir, "polish", "reports", "audita.report.json"), "{}\n")
mustWriteFile(t, filepath.Join(runWorkDir, "merge", "config", "seriatim.generated.yml"), "key: value\n")
mustWriteFile(t, filepath.Join(runWorkDir, "logs", "audita.stderr.log"), "stderr\n")
mustWriteFile(t, filepath.Join(runWorkDir, "manifest.json"), "{}\n")
mustWriteFile(t, filepath.Join(sessionRoot, "transcripts", "trimmed.json"), "{}\n")
mustWriteFile(t, filepath.Join(sessionRoot, "artifacts", "session_recap.md"), "# recap\n")
}
type failKeyStore struct {

View File

@@ -6,6 +6,7 @@ import (
"fmt"
"io"
"gitea.maximumdirect.net/eric/narratio/internal/artifacts"
"gitea.maximumdirect.net/eric/narratio/internal/config"
"gitea.maximumdirect.net/eric/narratio/internal/manifest"
)
@@ -83,7 +84,11 @@ func Resume(ctx context.Context, args []string, out io.Writer) error {
}
func loadManifestIfPresent(ctx context.Context, cfg *config.Config) (*manifest.Manifest, error) {
path := manifestPathFor(cfg)
path := artifacts.SessionManifestPathForCampaign(
cfg.Pipeline.Workspace.Root,
cfg.Session.Campaign,
cfg.Session.SessionID,
)
exists, err := fileExists(path)
if err != nil {
return nil, fmt.Errorf("check manifest %q: %w", path, err)

View File

@@ -16,7 +16,7 @@ import (
func TestResumeStartsAfterCompletedStages(t *testing.T) {
workspaceRoot := t.TempDir()
pipelinePath, sessionPath := writeValidConfigFiles(t, workspaceRoot)
manifestPath := filepath.Join(workspaceRoot, "work", "2026-05-03", "manifest.json")
manifestPath := filepath.Join(workspaceRoot, "work", "sample-campaign", "2026-05-03", "manifest.json")
store := &manifest.LocalStore{}
m := manifest.New("2026-05-03", time.Date(2026, 5, 3, 10, 0, 0, 0, time.UTC))
@@ -25,7 +25,7 @@ func TestResumeStartsAfterCompletedStages(t *testing.T) {
if err := store.Save(context.Background(), manifestPath, m); err != nil {
t.Fatalf("save manifest: %v", err)
}
workRoot := filepath.Join(workspaceRoot, "work", "2026-05-03")
workRoot := filepath.Join(workspaceRoot, "work", "sample-campaign", "2026-05-03")
mustWriteTestFile(t, filepath.Join(workRoot, "transcripts", "raw", "alice.json"), `{"segments":[]}`)
mustWriteTestFile(t, filepath.Join(workRoot, "inputs", "speakers.yml"), "match:\n - speaker: Alice\n match: [\"alice\"]\n")
mustWriteTestFile(t, filepath.Join(workRoot, "inputs", "autocorrect.yml"), "[]\n")
@@ -52,7 +52,7 @@ func TestResumeStartsAfterCompletedStages(t *testing.T) {
func TestResumeNoRemainingStages(t *testing.T) {
workspaceRoot := t.TempDir()
pipelinePath, sessionPath := writeValidConfigFiles(t, workspaceRoot)
manifestPath := filepath.Join(workspaceRoot, "work", "2026-05-03", "manifest.json")
manifestPath := filepath.Join(workspaceRoot, "work", "sample-campaign", "2026-05-03", "manifest.json")
store := &manifest.LocalStore{}
m := manifest.New("2026-05-03", time.Date(2026, 5, 3, 10, 0, 0, 0, time.UTC))
@@ -81,7 +81,7 @@ func TestResumeForceRerunsSucceeded(t *testing.T) {
}))
defer srv.Close()
pipelinePath, sessionPath := writeValidConfigFiles(t, workspaceRoot, srv.URL)
manifestPath := filepath.Join(workspaceRoot, "work", "2026-05-03", "manifest.json")
manifestPath := filepath.Join(workspaceRoot, "work", "sample-campaign", "2026-05-03", "manifest.json")
store := &manifest.LocalStore{}
m := manifest.New("2026-05-03", time.Date(2026, 5, 3, 10, 0, 0, 0, time.UTC))
@@ -105,8 +105,8 @@ func TestResumeForceRerunsSucceeded(t *testing.T) {
func TestRunStageExecutesOnlySelectedStage(t *testing.T) {
workspaceRoot := t.TempDir()
pipelinePath, sessionPath := writeValidConfigFiles(t, workspaceRoot)
manifestPath := filepath.Join(workspaceRoot, "work", "2026-05-03", "manifest.json")
workRoot := filepath.Join(workspaceRoot, "work", "2026-05-03")
manifestPath := filepath.Join(workspaceRoot, "work", "sample-campaign", "2026-05-03", "manifest.json")
workRoot := filepath.Join(workspaceRoot, "work", "sample-campaign", "2026-05-03")
mustWriteTestFile(t, filepath.Join(workRoot, "transcripts", "merged.json"), `{"segments":[]}`)
mustWriteTestFile(t, filepath.Join(workRoot, "inputs", "glossary.yml"), "terms: []\n")
@@ -135,8 +135,8 @@ func TestRunStageExecutesOnlySelectedStage(t *testing.T) {
func TestRunStageSkipAndForce(t *testing.T) {
workspaceRoot := t.TempDir()
pipelinePath, sessionPath := writeValidConfigFiles(t, workspaceRoot)
manifestPath := filepath.Join(workspaceRoot, "work", "2026-05-03", "manifest.json")
workRoot := filepath.Join(workspaceRoot, "work", "2026-05-03")
manifestPath := filepath.Join(workspaceRoot, "work", "sample-campaign", "2026-05-03", "manifest.json")
workRoot := filepath.Join(workspaceRoot, "work", "sample-campaign", "2026-05-03")
mustWriteTestFile(t, filepath.Join(workRoot, "transcripts", "merged.json"), `{"segments":[]}`)
mustWriteTestFile(t, filepath.Join(workRoot, "inputs", "glossary.yml"), "terms: []\n")
@@ -166,11 +166,57 @@ func TestRunStageSkipAndForce(t *testing.T) {
}
}
func TestRunStageForceMarksDownstreamStaleAndResumeContinuesFromStale(t *testing.T) {
workspaceRoot := t.TempDir()
pipelinePath, sessionPath := writeValidConfigFiles(t, workspaceRoot)
manifestPath := filepath.Join(workspaceRoot, "work", "sample-campaign", "2026-05-03", "manifest.json")
workRoot := filepath.Join(workspaceRoot, "work", "sample-campaign", "2026-05-03")
mustWriteTestFile(t, filepath.Join(workRoot, "transcripts", "merged.json"), `{"segments":[]}`)
mustWriteTestFile(t, filepath.Join(workRoot, "inputs", "glossary.yml"), "terms: []\n")
store := &manifest.LocalStore{}
seed := manifest.New("2026-05-03", time.Date(2026, 5, 3, 10, 0, 0, 0, time.UTC))
for _, name := range []string{"prepare", "transcribe", "merge", "polish", "normalize", "trim", "analyze", "archive", "notify"} {
seed.MarkStageSucceeded(name, time.Date(2026, 5, 3, 10, 1, 0, 0, time.UTC), nil)
}
if err := store.Save(context.Background(), manifestPath, seed); err != nil {
t.Fatalf("save manifest: %v", err)
}
var out bytes.Buffer
err := RunStage(context.Background(), []string{"--config", pipelinePath, "--session", sessionPath, "--force", "polish"}, &out)
if err != nil {
t.Fatalf("RunStage(force) error = %v", err)
}
if !strings.Contains(out.String(), "stage=polish executed=1 skipped=0 force=true") {
t.Fatalf("output = %q, want forced polish rerun", out.String())
}
afterForce, err := store.Load(context.Background(), manifestPath)
if err != nil {
t.Fatalf("load manifest after force: %v", err)
}
for _, name := range []string{"normalize", "trim", "analyze", "archive", "notify"} {
if afterForce.Stages[name] == nil || afterForce.Stages[name].Status != manifest.StatusStale {
t.Fatalf("stage %q = %#v, want stale", name, afterForce.Stages[name])
}
}
out.Reset()
err = Resume(context.Background(), []string{"--config", pipelinePath, "--session", sessionPath}, &out)
if err != nil {
t.Fatalf("Resume() error = %v", err)
}
if !strings.Contains(out.String(), "executed=5 skipped=0") {
t.Fatalf("output = %q, want resume to execute normalize..notify", out.String())
}
}
func TestRunStageTrimExecutes(t *testing.T) {
workspaceRoot := t.TempDir()
pipelinePath, sessionPath := writeValidConfigFiles(t, workspaceRoot)
manifestPath := filepath.Join(workspaceRoot, "work", "2026-05-03", "manifest.json")
workRoot := filepath.Join(workspaceRoot, "work", "2026-05-03")
manifestPath := filepath.Join(workspaceRoot, "work", "sample-campaign", "2026-05-03", "manifest.json")
workRoot := filepath.Join(workspaceRoot, "work", "sample-campaign", "2026-05-03")
mustWriteTestFile(t, filepath.Join(workRoot, "transcripts", "normalized.json"), `{"segments":[{"id":1},{"id":2}]}`)
var out bytes.Buffer
@@ -198,8 +244,8 @@ func TestRunStageTrimExecutes(t *testing.T) {
func TestRunStageNormalizeExecutes(t *testing.T) {
workspaceRoot := t.TempDir()
pipelinePath, sessionPath := writeValidConfigFiles(t, workspaceRoot)
manifestPath := filepath.Join(workspaceRoot, "work", "2026-05-03", "manifest.json")
workRoot := filepath.Join(workspaceRoot, "work", "2026-05-03")
manifestPath := filepath.Join(workspaceRoot, "work", "sample-campaign", "2026-05-03", "manifest.json")
workRoot := filepath.Join(workspaceRoot, "work", "sample-campaign", "2026-05-03")
mustWriteTestFile(t, filepath.Join(workRoot, "transcripts", "processed.json"), `{"segments":[{"id":1},{"id":2}]}`)
var out bytes.Buffer

View File

@@ -1,6 +1,8 @@
package app
import (
"time"
"gitea.maximumdirect.net/eric/narratio/internal/manifest"
"gitea.maximumdirect.net/eric/narratio/internal/stage"
)
@@ -49,3 +51,43 @@ func firstNonSucceededIndex(stages []stage.Stage, m *manifest.Manifest) int {
}
return len(stages)
}
func canonicalStageNames() []string {
all := stage.All()
out := make([]string, 0, len(all))
for _, s := range all {
if s == nil {
continue
}
out = append(out, s.Name())
}
return out
}
func downstreamStageNames(stageName string) []string {
names := canonicalStageNames()
for i, name := range names {
if name != stageName {
continue
}
return append([]string(nil), names[i+1:]...)
}
return nil
}
func invalidateDownstreamSucceededStages(m *manifest.Manifest, upstreamStage string, at time.Time) []string {
if m == nil || m.Stages == nil {
return nil
}
invalidated := make([]string, 0)
for _, downstream := range downstreamStageNames(upstreamStage) {
sr := m.Stages[downstream]
if sr == nil || sr.Status != manifest.StatusSucceeded {
continue
}
m.MarkStageStale(downstream, at, "upstream stage rerun with force")
invalidated = append(invalidated, downstream)
}
return invalidated
}

View File

@@ -1,6 +1,7 @@
package app
import (
"reflect"
"testing"
"time"
@@ -40,3 +41,48 @@ func TestDecideStageActions(t *testing.T) {
t.Fatalf("forced prepare action = %q, want %q", forced[0].Action, stageActionRun)
}
}
func TestDownstreamStageNames(t *testing.T) {
got := downstreamStageNames("polish")
want := []string{"normalize", "trim", "analyze", "archive", "notify"}
if !reflect.DeepEqual(got, want) {
t.Fatalf("downstreamStageNames(polish) = %#v, want %#v", got, want)
}
missing := downstreamStageNames("unknown")
if len(missing) != 0 {
t.Fatalf("downstreamStageNames(unknown) = %#v, want empty", missing)
}
}
func TestInvalidateDownstreamSucceededStages(t *testing.T) {
now := time.Now().UTC()
m := manifest.New("2026-05-03", now)
m.MarkStageSucceeded("prepare", now, nil)
m.MarkStageSucceeded("transcribe", now, nil)
m.MarkStageSucceeded("merge", now, nil)
m.MarkStageSucceeded("polish", now, nil)
m.MarkStageSucceeded("normalize", now, nil)
m.MarkStageSucceeded("trim", now, nil)
m.MarkStageFailed("analyze", now, "analysis failed")
m.MarkStageSucceeded("archive", now, nil)
m.MarkStageSucceeded("notify", now, nil)
got := invalidateDownstreamSucceededStages(m, "polish", now.Add(1*time.Second))
want := []string{"normalize", "trim", "archive", "notify"}
if !reflect.DeepEqual(got, want) {
t.Fatalf("invalidateDownstreamSucceededStages() = %#v, want %#v", got, want)
}
for _, stageName := range want {
if m.Stages[stageName].Status != manifest.StatusStale {
t.Fatalf("%s status = %q, want stale", stageName, m.Stages[stageName].Status)
}
}
if m.Stages["analyze"].Status != manifest.StatusFailed {
t.Fatalf("analyze status = %q, want failed", m.Stages["analyze"].Status)
}
if m.Stages["prepare"].Status != manifest.StatusSucceeded {
t.Fatalf("prepare status = %q, want succeeded", m.Stages["prepare"].Status)
}
}

View File

@@ -5,7 +5,6 @@ import (
"fmt"
"log/slog"
"os"
"path/filepath"
"strings"
"gitea.maximumdirect.net/eric/narratio/internal/adapters/audita"
@@ -27,11 +26,13 @@ type RunOptions struct {
}
type RunSummary struct {
SessionID string
ManifestPath string
StageNames []string
Executed []string
Skipped []string
SessionID string
RunID string
ManifestPath string
RunManifestPath string
StageNames []string
Executed []string
Skipped []string
}
func executeStages(ctx context.Context, cfg *config.Config, stages []stage.Stage, opts RunOptions) (*RunSummary, error) {
@@ -93,12 +94,12 @@ func executeStages(ctx context.Context, cfg *config.Config, stages []stage.Stage
}
artifactStore := env.ArtifactStore
paths, err := artifactStore.EnsureLayout(cfg.Session.SessionID)
paths, err := artifactStore.EnsureLayoutFor(cfg.Session.Campaign, cfg.Session.SessionID)
if err != nil {
return nil, fmt.Errorf("prepare workdir: %w", err)
}
lock, err := artifactStore.AcquireSessionLock(cfg.Session.SessionID)
lock, err := artifactStore.AcquireSessionLockFor(cfg.Session.Campaign, cfg.Session.SessionID)
if err != nil {
return nil, fmt.Errorf("acquire session lock: %w", err)
}
@@ -111,7 +112,11 @@ func executeStages(ctx context.Context, cfg *config.Config, stages []stage.Stage
if err != nil {
return nil, err
}
identityChanged, err := ensureManifestIdentity(cfg, m)
runID, err := artifacts.NewRunID()
if err != nil {
return nil, fmt.Errorf("generate run id: %w", err)
}
identityChanged, err := ensureManifestIdentity(cfg, m, runID)
if err != nil {
return nil, fmt.Errorf("initialize manifest identity: %w", err)
}
@@ -120,6 +125,29 @@ func executeStages(ctx context.Context, cfg *config.Config, stages []stage.Stage
return nil, fmt.Errorf("save manifest identity %q: %w", manifestPath, err)
}
}
runManifestPath := artifacts.SessionRunManifestPathForCampaign(
cfg.Pipeline.Workspace.Root,
cfg.Session.Campaign,
cfg.Session.SessionID,
runID,
)
runManifestStore := &manifest.LocalStore{}
runManifest, err := runManifestStore.CreateRun(
ctx,
cfg.Session.SessionID,
cfg.Session.Campaign,
runID,
opts.Force,
requestedStageNames(stages),
)
if err != nil {
return nil, fmt.Errorf("create run manifest: %w", err)
}
runManifest.SessionManifestPath = manifestPath
syncRunManifestIdentityFromSession(m, runManifest)
if err := runManifestStore.SaveRun(ctx, runManifestPath, runManifest); err != nil {
return nil, fmt.Errorf("save initial run manifest %q: %w", runManifestPath, err)
}
stageEnv := env
@@ -134,12 +162,23 @@ func executeStages(ctx context.Context, cfg *config.Config, stages []stage.Stage
if d.Action == stageActionSkip {
skipped = append(skipped, s.Name())
skipAt := nowUTC()
runManifest.SetStageAction(s.Name(), manifest.RunStageActionSkip, skipAt)
runManifest.MarkStageSkipped(s.Name(), skipAt, "already_succeeded")
if err := runManifestStore.SaveRun(ctx, runManifestPath, runManifest); err != nil {
return nil, fmt.Errorf("save run manifest after skip %q: %w", s.Name(), err)
}
env.Logger.Info("skipping stage", "stage", s.Name(), "reason", "already_succeeded", "force", opts.Force)
continue
}
executed = append(executed, s.Name())
now := nowUTC()
runManifest.SetStageAction(s.Name(), manifest.RunStageActionRun, now)
runManifest.MarkStageRunning(s.Name(), now)
if err := runManifestStore.SaveRun(ctx, runManifestPath, runManifest); err != nil {
return nil, fmt.Errorf("save run manifest before stage %q: %w", s.Name(), err)
}
m.MarkStageRunning(s.Name(), now)
env.Logger.Info("starting stage", "stage", s.Name())
if err := env.ManifestStore.Save(ctx, manifestPath, m); err != nil {
@@ -149,35 +188,65 @@ func executeStages(ctx context.Context, cfg *config.Config, stages []stage.Stage
result, err := s.Run(ctx, stageEnv, m)
if err != nil {
m.MarkStageFailed(s.Name(), nowUTC(), err.Error())
failedAt := nowUTC()
m.MarkStageFailed(s.Name(), failedAt, err.Error())
if saveErr := env.ManifestStore.Save(ctx, manifestPath, m); saveErr != nil {
return nil, fmt.Errorf("stage %q failed (%v) and manifest save failed (%v)", s.Name(), err, saveErr)
}
runManifest.MarkStageFailed(s.Name(), failedAt, err.Error())
syncRunManifestIdentityFromSession(m, runManifest)
if saveErr := runManifestStore.SaveRun(ctx, runManifestPath, runManifest); saveErr != nil {
return nil, fmt.Errorf("stage %q failed (%v) and run-manifest save failed (%v)", s.Name(), err, saveErr)
}
env.Logger.Info("stage failed", "stage", s.Name(), "error", err)
return nil, fmt.Errorf("stage %q failed: %w", s.Name(), err)
}
outputs := mapResultOutputs(result)
m.MarkStageSucceeded(s.Name(), nowUTC(), outputs)
outputs := mapResultOutputs(result, runID)
succeededAt := nowUTC()
m.MarkStageSucceeded(s.Name(), succeededAt, outputs)
applyStageResultToManifest(m, s.Name(), result)
if opts.Force {
invalidateDownstreamSucceededStages(m, s.Name(), succeededAt)
}
if err := env.ManifestStore.Save(ctx, manifestPath, m); err != nil {
return nil, fmt.Errorf("save manifest after stage %q: %w", s.Name(), err)
}
runManifest.MarkStageSucceeded(s.Name(), succeededAt, outputs)
applyStageResultToRunManifest(runManifest, s.Name(), result)
syncRunManifestIdentityFromSession(m, runManifest)
if err := runManifestStore.SaveRun(ctx, runManifestPath, runManifest); err != nil {
return nil, fmt.Errorf("save run manifest after stage %q: %w", s.Name(), err)
}
env.Logger.Debug("manifest saved", "stage", s.Name(), "transition", "succeeded", "path", manifestPath)
env.Logger.Info("stage succeeded", "stage", s.Name())
}
if err := runPostArchiveCleanup(ctx, env, manifestPath, m, executed); err != nil {
failedAt := nowUTC()
runManifest.MarkFailed(failedAt, err.Error())
syncRunManifestIdentityFromSession(m, runManifest)
if saveErr := runManifestStore.SaveRun(ctx, runManifestPath, runManifest); saveErr != nil {
return nil, fmt.Errorf("post-archive cleanup failed (%v) and run-manifest save failed (%v)", err, saveErr)
}
return nil, fmt.Errorf("post-archive cleanup: %w", err)
}
completedAt := nowUTC()
runManifest.MarkSucceeded(completedAt)
syncRunManifestIdentityFromSession(m, runManifest)
if err := runManifestStore.SaveRun(ctx, runManifestPath, runManifest); err != nil {
return nil, fmt.Errorf("save final run manifest %q: %w", runManifestPath, err)
}
return &RunSummary{
SessionID: cfg.Session.SessionID,
ManifestPath: manifestPath,
StageNames: runNames,
Executed: executed,
Skipped: skipped,
SessionID: cfg.Session.SessionID,
RunID: runID,
ManifestPath: manifestPath,
RunManifestPath: runManifestPath,
StageNames: runNames,
Executed: executed,
Skipped: skipped,
}, nil
}
@@ -317,11 +386,12 @@ func fileExists(path string) (bool, error) {
return false, err
}
func mapResultOutputs(result *stage.StageResult) []manifest.ArtifactRecord {
func mapResultOutputs(result *stage.StageResult, runID string) []manifest.ArtifactRecord {
if result == nil || len(result.Outputs) == 0 {
return nil
}
runID = strings.TrimSpace(runID)
out := make([]manifest.ArtifactRecord, 0, len(result.Outputs))
for _, ref := range result.Outputs {
localPath := ref.AbsolutePath
@@ -329,10 +399,11 @@ func mapResultOutputs(result *stage.StageResult) []manifest.ArtifactRecord {
localPath = ref.RelativePath
}
out = append(out, manifest.ArtifactRecord{
Kind: ref.Kind,
LocalPath: localPath,
RemoteKey: ref.RemoteKey,
Checksum: ref.Checksum,
Kind: ref.Kind,
LocalPath: localPath,
ProducerRunID: runID,
RemoteKey: ref.RemoteKey,
Checksum: ref.Checksum,
})
}
@@ -358,7 +429,7 @@ func applyStageResultToManifest(m *manifest.Manifest, stageName string, result *
}
}
func ensureManifestIdentity(cfg *config.Config, m *manifest.Manifest) (bool, error) {
func ensureManifestIdentity(cfg *config.Config, m *manifest.Manifest, runID string) (bool, error) {
if cfg == nil || cfg.Pipeline == nil || cfg.Session == nil || m == nil {
return false, nil
}
@@ -374,16 +445,13 @@ func ensureManifestIdentity(cfg *config.Config, m *manifest.Manifest) (bool, err
m.Campaign = campaign
changed = true
}
if m.RunID == "" {
runID, err := artifacts.NewRunID()
if err != nil {
return false, err
}
runID = strings.TrimSpace(runID)
if runID != "" && m.RunID != runID {
m.RunID = runID
changed = true
}
if m.LocalWorkDir == "" && campaign != "" && sessionID != "" && m.RunID != "" {
m.LocalWorkDir = artifacts.SessionRunWorkDir(cfg.Pipeline.Workspace.Root, campaign, sessionID, m.RunID)
m.LocalWorkDir = artifacts.SessionRunRootForCampaign(cfg.Pipeline.Workspace.Root, campaign, sessionID, m.RunID)
changed = true
}
if m.LocalSpoolDir == "" && campaign != "" && sessionID != "" && m.RunID != "" && strings.TrimSpace(cfg.Pipeline.Spool.Root) != "" {
@@ -410,8 +478,54 @@ func ensureManifestIdentity(cfg *config.Config, m *manifest.Manifest) (bool, err
return changed, nil
}
func requestedStageNames(stages []stage.Stage) []string {
out := make([]string, 0, len(stages))
for _, s := range stages {
if s == nil {
continue
}
out = append(out, s.Name())
}
return out
}
func applyStageResultToRunManifest(m *manifest.RunManifest, stageName string, result *stage.StageResult) {
if m == nil || result == nil {
return
}
sr := m.Stages[stageName]
if sr == nil {
return
}
if len(result.Logs) > 0 {
sr.Logs = append([]string(nil), result.Logs...)
}
if len(result.GeneratedConfigs) > 0 {
sr.GeneratedConfigs = append([]string(nil), result.GeneratedConfigs...)
}
if len(result.Metadata) > 0 {
sr.Metadata = result.Metadata
}
}
func syncRunManifestIdentityFromSession(session *manifest.Manifest, run *manifest.RunManifest) {
if session == nil || run == nil {
return
}
run.Campaign = session.Campaign
run.LocalWorkDir = session.LocalWorkDir
run.LocalSpoolDir = session.LocalSpoolDir
run.S3Bucket = session.S3Bucket
run.S3SessionPrefix = session.S3SessionPrefix
run.S3RunPrefix = session.S3RunPrefix
}
func manifestPathFor(cfg *config.Config) string {
return filepath.Join(cfg.Pipeline.Workspace.Root, "work", cfg.Session.SessionID, "manifest.json")
return artifacts.SessionManifestPathForCampaign(
cfg.Pipeline.Workspace.Root,
cfg.Session.Campaign,
cfg.Session.SessionID,
)
}
func needsObjectStoreForRun(cfg *config.Config, stages []stage.Stage) bool {

View File

@@ -235,6 +235,56 @@ func TestExecuteStagesForceRerunsSucceeded(t *testing.T) {
}
}
func TestExecuteStagesForceSuccessInvalidatesDownstreamSucceededStages(t *testing.T) {
cfg := testConfig(t)
manifestPath := manifestPathFor(cfg)
store := &manifest.LocalStore{}
existing := manifest.New(cfg.Session.SessionID, time.Date(2026, 5, 3, 1, 0, 0, 0, time.UTC))
for _, stageName := range []string{"prepare", "transcribe", "merge", "polish", "normalize", "trim", "archive", "notify"} {
existing.MarkStageSucceeded(stageName, time.Date(2026, 5, 3, 1, 1, 0, 0, time.UTC), nil)
}
existing.MarkStageFailed("analyze", time.Date(2026, 5, 3, 1, 1, 0, 0, time.UTC), "previous analyze failure")
if err := os.MkdirAll(filepath.Dir(manifestPath), 0o755); err != nil {
t.Fatalf("MkdirAll() error = %v", err)
}
if err := store.Save(context.Background(), manifestPath, existing); err != nil {
t.Fatalf("Save manifest error = %v", err)
}
runs := 0
stageToRun := countingStage{name: "polish", runs: &runs}
summary, err := executeStages(context.Background(), cfg, []stage.Stage{stageToRun}, RunOptions{Force: true})
if err != nil {
t.Fatalf("executeStages() error = %v", err)
}
if runs != 1 {
t.Fatalf("runs = %d, want 1 with force", runs)
}
if len(summary.Executed) != 1 || summary.Executed[0] != "polish" || len(summary.Skipped) != 0 {
t.Fatalf("summary = %#v, want executed polish", summary)
}
loaded, err := store.Load(context.Background(), manifestPath)
if err != nil {
t.Fatalf("Load manifest error = %v", err)
}
if loaded.Stages["polish"] == nil || loaded.Stages["polish"].Status != manifest.StatusSucceeded {
t.Fatalf("polish status = %#v, want succeeded", loaded.Stages["polish"])
}
for _, stageName := range []string{"normalize", "trim", "archive", "notify"} {
if loaded.Stages[stageName] == nil || loaded.Stages[stageName].Status != manifest.StatusStale {
t.Fatalf("%s status = %#v, want stale", stageName, loaded.Stages[stageName])
}
}
if loaded.Stages["analyze"] == nil || loaded.Stages["analyze"].Status != manifest.StatusFailed {
t.Fatalf("analyze status = %#v, want preserved failed", loaded.Stages["analyze"])
}
if loaded.Stages["transcribe"] == nil || loaded.Stages["transcribe"].Status != manifest.StatusSucceeded {
t.Fatalf("transcribe status = %#v, want preserved succeeded", loaded.Stages["transcribe"])
}
}
func TestExecuteStagesFailureUpdatesManifest(t *testing.T) {
cfg := testConfig(t)
@@ -280,7 +330,14 @@ func TestExecuteStagesLoadsExistingManifest(t *testing.T) {
existing := manifest.New(cfg.Session.SessionID, time.Date(2026, 5, 3, 1, 0, 0, 0, time.UTC))
existing.MarkStageSucceeded("prepare", time.Date(2026, 5, 3, 1, 1, 0, 0, time.UTC), nil)
audioPath := filepath.Join(cfg.Pipeline.Workspace.Root, "work", cfg.Session.SessionID, "audio", "alice.flac")
audioPath := filepath.Join(
cfg.Pipeline.Workspace.Root,
"work",
cfg.Session.Campaign,
cfg.Session.SessionID,
"audio",
"alice.flac",
)
if err := os.MkdirAll(filepath.Dir(audioPath), 0o755); err != nil {
t.Fatalf("MkdirAll() error = %v", err)
}
@@ -315,6 +372,208 @@ func TestExecuteStagesLoadsExistingManifest(t *testing.T) {
}
}
func TestExecuteStagesCreatesRunManifestPerInvocation(t *testing.T) {
cfg := testConfig(t)
run1, err := executeStages(context.Background(), cfg, []stage.Stage{BuildFullPlan()[0]}, RunOptions{})
if err != nil {
t.Fatalf("first executeStages() error = %v", err)
}
run2, err := executeStages(context.Background(), cfg, []stage.Stage{BuildFullPlan()[0]}, RunOptions{Force: true})
if err != nil {
t.Fatalf("second executeStages() error = %v", err)
}
if run1.RunID == "" || run2.RunID == "" {
t.Fatalf("run ids must be set, got %q and %q", run1.RunID, run2.RunID)
}
if run1.RunID == run2.RunID {
t.Fatalf("expected distinct run ids, got %q", run1.RunID)
}
if run1.RunManifestPath == "" || run2.RunManifestPath == "" {
t.Fatalf("run manifest paths must be set, got %q and %q", run1.RunManifestPath, run2.RunManifestPath)
}
if run1.RunManifestPath == run2.RunManifestPath {
t.Fatalf("expected distinct run manifest paths, got %q", run1.RunManifestPath)
}
for _, path := range []string{run1.RunManifestPath, run2.RunManifestPath} {
if _, statErr := os.Stat(path); statErr != nil {
t.Fatalf("run manifest missing at %q: %v", path, statErr)
}
}
store := &manifest.LocalStore{}
sessionManifest, err := store.Load(context.Background(), run2.ManifestPath)
if err != nil {
t.Fatalf("Load session manifest error = %v", err)
}
if sessionManifest.RunID != run2.RunID {
t.Fatalf("session manifest run_id = %q, want latest run id %q", sessionManifest.RunID, run2.RunID)
}
}
func TestExecuteStagesRunManifestRecordsSkippedStage(t *testing.T) {
cfg := testConfig(t)
store := &manifest.LocalStore{}
manifestPath := manifestPathFor(cfg)
existing := manifest.New(cfg.Session.SessionID, time.Date(2026, 5, 3, 1, 0, 0, 0, time.UTC))
existing.MarkStageSucceeded("transcribe", time.Date(2026, 5, 3, 1, 1, 0, 0, time.UTC), nil)
if err := os.MkdirAll(filepath.Dir(manifestPath), 0o755); err != nil {
t.Fatalf("MkdirAll() error = %v", err)
}
if err := store.Save(context.Background(), manifestPath, existing); err != nil {
t.Fatalf("Save manifest error = %v", err)
}
summary, err := executeStages(context.Background(), cfg, []stage.Stage{BuildFullPlan()[1]}, RunOptions{})
if err != nil {
t.Fatalf("executeStages() error = %v", err)
}
if len(summary.Skipped) != 1 || summary.Skipped[0] != "transcribe" {
t.Fatalf("summary = %#v, want skipped transcribe", summary)
}
runManifest, err := store.LoadRun(context.Background(), summary.RunManifestPath)
if err != nil {
t.Fatalf("LoadRun() error = %v", err)
}
sr := runManifest.Stages["transcribe"]
if sr == nil {
t.Fatal("run manifest transcribe stage missing")
}
if sr.Action != manifest.RunStageActionSkip {
t.Fatalf("action = %q, want %q", sr.Action, manifest.RunStageActionSkip)
}
if sr.Status != manifest.StatusSkipped {
t.Fatalf("status = %q, want %q", sr.Status, manifest.StatusSkipped)
}
}
func TestExecuteStagesRunLocalArtifactsAndCanonicalPromotion(t *testing.T) {
cfg := testConfig(t)
stages := []stage.Stage{
BuildFullPlan()[0], // prepare
BuildFullPlan()[1], // transcribe
BuildFullPlan()[2], // merge
BuildFullPlan()[3], // polish
BuildFullPlan()[4], // normalize
BuildFullPlan()[5], // trim
}
summary, err := executeStages(context.Background(), cfg, stages, RunOptions{})
if err != nil {
t.Fatalf("executeStages() error = %v", err)
}
if summary.RunID == "" {
t.Fatal("run id must be set")
}
runRoot := artifacts.SessionRunRootForCampaign(
cfg.Pipeline.Workspace.Root,
cfg.Session.Campaign,
cfg.Session.SessionID,
summary.RunID,
)
paths := artifacts.NewLocalStore(cfg.Pipeline.Workspace.Root).SessionPathsFor(cfg.Session.Campaign, cfg.Session.SessionID)
runLocalChecks := []string{
filepath.Join(runRoot, "transcribe", "outputs", "transcripts", "raw", "alice.json"),
filepath.Join(runRoot, "merge", "logs", "seriatim.stdout.log"),
filepath.Join(runRoot, "polish", "config", "audita.generated.yml"),
filepath.Join(runRoot, "normalize", "logs", "seriatim.normalize.stdout.log"),
filepath.Join(runRoot, "trim", "outputs", "transcripts", "trimmed.json"),
}
for _, p := range runLocalChecks {
if _, statErr := os.Stat(p); statErr != nil {
t.Fatalf("run-local artifact missing at %q: %v", p, statErr)
}
}
canonicalChecks := []string{
filepath.Join(paths.TranscriptsRawDir, "alice.json"),
filepath.Join(paths.TranscriptsDir, "merged.json"),
filepath.Join(paths.TranscriptsDir, "processed.json"),
filepath.Join(paths.TranscriptsDir, "normalized.json"),
filepath.Join(paths.TranscriptsDir, "trimmed.json"),
}
for _, p := range canonicalChecks {
if _, statErr := os.Stat(p); statErr != nil {
t.Fatalf("canonical promoted artifact missing at %q: %v", p, statErr)
}
}
store := &manifest.LocalStore{}
sessionManifest, err := store.Load(context.Background(), summary.ManifestPath)
if err != nil {
t.Fatalf("Load manifest error = %v", err)
}
if got := sessionManifest.Stages["trim"]; got == nil || len(got.Outputs) == 0 {
t.Fatalf("trim stage outputs missing in session manifest: %#v", got)
}
for _, out := range sessionManifest.Stages["trim"].Outputs {
if strings.Contains(out.LocalPath, string(filepath.Separator)+"runs"+string(filepath.Separator)) {
t.Fatalf("session manifest output should be canonical, got run-local path %q", out.LocalPath)
}
if out.ProducerRunID != summary.RunID {
t.Fatalf("producer_run_id = %q, want %q", out.ProducerRunID, summary.RunID)
}
}
runManifest, err := store.LoadRun(context.Background(), summary.RunManifestPath)
if err != nil {
t.Fatalf("LoadRun() error = %v", err)
}
mergeStage := runManifest.Stages["merge"]
if mergeStage == nil || len(mergeStage.Logs) == 0 {
t.Fatalf("merge logs missing in run manifest: %#v", mergeStage)
}
for _, logPath := range mergeStage.Logs {
if !strings.Contains(logPath, filepath.Join("runs", summary.RunID, "merge", "logs")) {
t.Fatalf("run manifest merge log path = %q, want run-local merge logs path", logPath)
}
}
}
func TestExecuteStagesSkippedStagePreservesExistingOutputsProvenance(t *testing.T) {
cfg := testConfig(t)
store := &manifest.LocalStore{}
manifestPath := manifestPathFor(cfg)
existing := manifest.New(cfg.Session.SessionID, time.Date(2026, 5, 3, 1, 0, 0, 0, time.UTC))
existing.MarkStageSucceeded("transcribe", time.Date(2026, 5, 3, 1, 1, 0, 0, time.UTC), []manifest.ArtifactRecord{
{
Kind: "transcript_raw",
LocalPath: "transcripts/raw/alice.json",
ProducerRunID: "20260501T000000Z-deadbeef",
},
})
if err := os.MkdirAll(filepath.Dir(manifestPath), 0o755); err != nil {
t.Fatalf("MkdirAll() error = %v", err)
}
if err := store.Save(context.Background(), manifestPath, existing); err != nil {
t.Fatalf("Save manifest error = %v", err)
}
summary, err := executeStages(context.Background(), cfg, []stage.Stage{BuildFullPlan()[1]}, RunOptions{})
if err != nil {
t.Fatalf("executeStages() error = %v", err)
}
if len(summary.Skipped) != 1 || summary.Skipped[0] != "transcribe" {
t.Fatalf("summary = %#v, want skipped transcribe", summary)
}
loaded, err := store.Load(context.Background(), manifestPath)
if err != nil {
t.Fatalf("Load manifest error = %v", err)
}
got := loaded.Stages["transcribe"]
if got == nil || len(got.Outputs) != 1 {
t.Fatalf("transcribe outputs = %#v, want one preserved output", got)
}
if got.Outputs[0].ProducerRunID != "20260501T000000Z-deadbeef" {
t.Fatalf("producer_run_id = %q, want preserved value", got.Outputs[0].ProducerRunID)
}
}
func TestAdapterBackedStageFailureMarksManifestFailed(t *testing.T) {
cases := []struct {
name string
@@ -341,7 +600,7 @@ func TestAdapterBackedStageFailureMarksManifestFailed(t *testing.T) {
tc.env.ArtifactStore = artifactStore
tc.env.ManifestStore = &manifest.LocalStore{}
if tc.name == "transcribe" {
paths, ensureErr := artifactStore.EnsureLayout(cfg.Session.SessionID)
paths, ensureErr := artifactStore.EnsureLayoutFor(cfg.Session.Campaign, cfg.Session.SessionID)
if ensureErr != nil {
t.Fatalf("EnsureLayout() error = %v", ensureErr)
}
@@ -356,7 +615,7 @@ func TestAdapterBackedStageFailureMarksManifestFailed(t *testing.T) {
}
}
if tc.name == "merge" {
paths, ensureErr := artifactStore.EnsureLayout(cfg.Session.SessionID)
paths, ensureErr := artifactStore.EnsureLayoutFor(cfg.Session.Campaign, cfg.Session.SessionID)
if ensureErr != nil {
t.Fatalf("EnsureLayout() error = %v", ensureErr)
}
@@ -375,7 +634,7 @@ func TestAdapterBackedStageFailureMarksManifestFailed(t *testing.T) {
}
}
if tc.name == "polish" {
paths, ensureErr := artifactStore.EnsureLayout(cfg.Session.SessionID)
paths, ensureErr := artifactStore.EnsureLayoutFor(cfg.Session.Campaign, cfg.Session.SessionID)
if ensureErr != nil {
t.Fatalf("EnsureLayout() error = %v", ensureErr)
}
@@ -387,7 +646,7 @@ func TestAdapterBackedStageFailureMarksManifestFailed(t *testing.T) {
}
}
if tc.name == "analyze" {
paths, ensureErr := artifactStore.EnsureLayout(cfg.Session.SessionID)
paths, ensureErr := artifactStore.EnsureLayoutFor(cfg.Session.Campaign, cfg.Session.SessionID)
if ensureErr != nil {
t.Fatalf("EnsureLayout() error = %v", ensureErr)
}

View File

@@ -63,7 +63,7 @@ func TestExecuteStagesDefaultWiringUsesWhisperXHTTPClient(t *testing.T) {
t.Fatal("audio file payload was empty")
}
outPath := filepath.Join(cfg.Pipeline.Workspace.Root, "work", cfg.Session.SessionID, "transcripts", "raw", "alice.json")
outPath := filepath.Join(cfg.Pipeline.Workspace.Root, "work", cfg.Session.Campaign, cfg.Session.SessionID, "transcripts", "raw", "alice.json")
data, err := os.ReadFile(outPath)
if err != nil {
t.Fatalf("ReadFile(%q) error = %v", outPath, err)

View File

@@ -0,0 +1,294 @@
package artifacts
import (
"encoding/json"
"errors"
"fmt"
"os"
"path/filepath"
"strings"
"gitea.maximumdirect.net/eric/narratio/internal/manifest"
)
const (
ArtifactTranscriptMerged = "narratio.transcript.merged"
ArtifactTranscriptPolished = "narratio.transcript.polished"
ArtifactTranscriptFull = "narratio.transcript.full"
ArtifactTranscriptTrimmed = "narratio.transcript.trimmed"
ArtifactBoundsSession = "narratio.bounds.session"
ArtifactSessionRecap = "narratio.artifact.session_recap"
)
// ErrSessionArtifactNotFound is returned when no readable artifact exists for a known ID.
var ErrSessionArtifactNotFound = errors.New("session artifact not found")
type artifactContentKind string
const (
contentTranscriptJSON artifactContentKind = "transcript_json"
contentJSON artifactContentKind = "json"
contentText artifactContentKind = "text"
)
type artifactSpec struct {
ID string
CanonicalRelPath string
ProducerStage string
OutputKind string
ContentKind artifactContentKind
}
var artifactRegistry = map[string]artifactSpec{
ArtifactTranscriptMerged: {
ID: ArtifactTranscriptMerged,
CanonicalRelPath: "transcripts/merged.json",
ProducerStage: "merge",
OutputKind: "transcript_merged",
ContentKind: contentTranscriptJSON,
},
ArtifactTranscriptPolished: {
ID: ArtifactTranscriptPolished,
CanonicalRelPath: "transcripts/processed.json",
ProducerStage: "polish",
OutputKind: "transcript_processed",
ContentKind: contentTranscriptJSON,
},
ArtifactTranscriptFull: {
ID: ArtifactTranscriptFull,
CanonicalRelPath: "transcripts/normalized.json",
ProducerStage: "normalize",
OutputKind: "transcript_normalized",
ContentKind: contentTranscriptJSON,
},
ArtifactTranscriptTrimmed: {
ID: ArtifactTranscriptTrimmed,
CanonicalRelPath: "transcripts/trimmed.json",
ProducerStage: "trim",
OutputKind: "transcript_trimmed",
ContentKind: contentTranscriptJSON,
},
ArtifactBoundsSession: {
ID: ArtifactBoundsSession,
CanonicalRelPath: "artifacts/session_bounds.json",
ProducerStage: "trim",
OutputKind: "session_bounds",
ContentKind: contentJSON,
},
ArtifactSessionRecap: {
ID: ArtifactSessionRecap,
CanonicalRelPath: "artifacts/session_recap.md",
ProducerStage: "analyze",
OutputKind: "session_recap",
ContentKind: contentText,
},
}
var artifactAliases = map[string]string{
"processed_transcript": ArtifactTranscriptPolished,
"normalized_transcript": ArtifactTranscriptFull,
"trimmed_transcript": ArtifactTranscriptTrimmed,
}
// ResolvedSessionArtifact describes one session-level artifact lookup result.
type ResolvedSessionArtifact struct {
ID string
Path string
ProducerStage string
OutputKind string
ProducerRunID string
Provenance string
}
// SessionArtifactNotFoundError includes context when a known artifact cannot be read.
type SessionArtifactNotFoundError struct {
ArtifactID string
}
func (e *SessionArtifactNotFoundError) Error() string {
return fmt.Sprintf("%s: %q", ErrSessionArtifactNotFound, e.ArtifactID)
}
func (e *SessionArtifactNotFoundError) Unwrap() error {
return ErrSessionArtifactNotFound
}
// NormalizeSessionArtifactSource maps legacy aliases to canonical IDs and validates IDs.
func NormalizeSessionArtifactSource(source string) (string, error) {
normalized := strings.TrimSpace(source)
if normalized == "" {
return "", fmt.Errorf("artifact source is required")
}
if alias, ok := artifactAliases[normalized]; ok {
normalized = alias
}
if _, ok := artifactRegistry[normalized]; !ok {
return "", fmt.Errorf("unsupported artifact source %q", source)
}
return normalized, nil
}
// ResolveSessionArtifact resolves a symbolic source to a readable local session artifact path.
// Resolution order is manifest producer outputs first, then canonical session path fallback.
func ResolveSessionArtifact(paths SessionPaths, m *manifest.Manifest, source string) (ResolvedSessionArtifact, error) {
id, err := NormalizeSessionArtifactSource(source)
if err != nil {
return ResolvedSessionArtifact{}, err
}
spec := artifactRegistry[id]
for _, candidate := range manifestArtifactCandidates(paths, m, spec) {
exists, isDir, statErr := pathExists(candidate.Path)
if statErr != nil {
return ResolvedSessionArtifact{}, fmt.Errorf("stat %q: %w", candidate.Path, statErr)
}
if !exists || isDir {
continue
}
resolved := candidate
resolved.ID = spec.ID
resolved.ProducerStage = spec.ProducerStage
resolved.OutputKind = spec.OutputKind
if err := validateResolvedContent(resolved.Path, spec.ContentKind); err != nil {
return ResolvedSessionArtifact{}, fmt.Errorf("validate %q: %w", resolved.ID, err)
}
return resolved, nil
}
fallbackPath := filepath.Join(paths.Root, filepath.FromSlash(spec.CanonicalRelPath))
exists, isDir, statErr := pathExists(fallbackPath)
if statErr != nil {
return ResolvedSessionArtifact{}, fmt.Errorf("stat %q: %w", fallbackPath, statErr)
}
if exists && !isDir {
if err := validateResolvedContent(fallbackPath, spec.ContentKind); err != nil {
return ResolvedSessionArtifact{}, fmt.Errorf("validate %q: %w", spec.ID, err)
}
return ResolvedSessionArtifact{
ID: spec.ID,
Path: filepath.Clean(fallbackPath),
ProducerStage: spec.ProducerStage,
OutputKind: spec.OutputKind,
Provenance: "fallback.canonical_path",
}, nil
}
return ResolvedSessionArtifact{}, &SessionArtifactNotFoundError{ArtifactID: spec.ID}
}
func manifestArtifactCandidates(paths SessionPaths, m *manifest.Manifest, spec artifactSpec) []ResolvedSessionArtifact {
if m == nil || len(m.Stages) == 0 || spec.ProducerStage == "" || spec.OutputKind == "" {
return nil
}
sr := m.Stages[spec.ProducerStage]
if sr == nil {
return nil
}
candidates := make([]ResolvedSessionArtifact, 0, len(sr.Outputs))
for _, out := range sr.Outputs {
if strings.TrimSpace(out.Kind) != spec.OutputKind {
continue
}
if strings.TrimSpace(out.LocalPath) == "" {
continue
}
resolved := filepath.Clean(ResolveSessionLocalPathForRead(paths, out.LocalPath))
if resolved == "" {
continue
}
candidates = append(candidates, ResolvedSessionArtifact{
Path: resolved,
ProducerRunID: strings.TrimSpace(out.ProducerRunID),
Provenance: "manifest." + spec.ProducerStage + ".outputs",
})
}
return dedupeResolvedArtifacts(candidates)
}
func dedupeResolvedArtifacts(values []ResolvedSessionArtifact) []ResolvedSessionArtifact {
seen := map[string]struct{}{}
out := make([]ResolvedSessionArtifact, 0, len(values))
for _, value := range values {
key := filepath.Clean(strings.TrimSpace(value.Path))
if key == "" {
continue
}
if _, ok := seen[key]; ok {
continue
}
seen[key] = struct{}{}
value.Path = key
out = append(out, value)
}
return out
}
func pathExists(path string) (exists bool, isDir bool, err error) {
info, err := os.Stat(path)
if err == nil {
return true, info.IsDir(), nil
}
if errors.Is(err, os.ErrNotExist) {
return false, false, nil
}
return false, false, err
}
func validateResolvedContent(path string, kind artifactContentKind) error {
switch kind {
case contentTranscriptJSON:
return validateTranscriptSegmentsJSON(path)
case contentJSON:
return validateJSONContent(path)
case contentText:
return validateNonEmptyContent(path)
default:
return fmt.Errorf("unsupported content kind %q", kind)
}
}
func validateTranscriptSegmentsJSON(path string) error {
data, err := os.ReadFile(path)
if err != nil {
return fmt.Errorf("read file: %w", err)
}
var payload map[string]any
if err := json.Unmarshal(data, &payload); err != nil {
return fmt.Errorf("decode json: %w", err)
}
segments, ok := payload["segments"]
if !ok {
return fmt.Errorf("top-level segments is required")
}
if _, ok := segments.([]any); !ok {
return fmt.Errorf("top-level segments must be an array")
}
return nil
}
func validateJSONContent(path string) error {
data, err := os.ReadFile(path)
if err != nil {
return fmt.Errorf("read file: %w", err)
}
var payload any
if err := json.Unmarshal(data, &payload); err != nil {
return fmt.Errorf("decode json: %w", err)
}
return nil
}
func validateNonEmptyContent(path string) error {
info, err := os.Stat(path)
if err != nil {
return fmt.Errorf("stat file: %w", err)
}
if info.IsDir() {
return fmt.Errorf("path is a directory")
}
if info.Size() <= 0 {
return fmt.Errorf("file is empty")
}
return nil
}

View File

@@ -0,0 +1,139 @@
package artifacts
import (
"errors"
"os"
"path/filepath"
"strings"
"testing"
"time"
"gitea.maximumdirect.net/eric/narratio/internal/manifest"
)
func TestNormalizeSessionArtifactSource(t *testing.T) {
tests := []struct {
name string
source string
wantID string
wantErr string
}{
{name: "legacy alias processed", source: "processed_transcript", wantID: ArtifactTranscriptPolished},
{name: "legacy alias normalized", source: "normalized_transcript", wantID: ArtifactTranscriptFull},
{name: "legacy alias trimmed", source: "trimmed_transcript", wantID: ArtifactTranscriptTrimmed},
{name: "canonical", source: ArtifactTranscriptTrimmed, wantID: ArtifactTranscriptTrimmed},
{name: "unsupported", source: "narratio.unknown", wantErr: "unsupported artifact source"},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
got, err := NormalizeSessionArtifactSource(tt.source)
if tt.wantErr != "" {
if err == nil || !strings.Contains(err.Error(), tt.wantErr) {
t.Fatalf("NormalizeSessionArtifactSource() error = %v, want contains %q", err, tt.wantErr)
}
return
}
if err != nil {
t.Fatalf("NormalizeSessionArtifactSource() error = %v", err)
}
if got != tt.wantID {
t.Fatalf("NormalizeSessionArtifactSource() = %q, want %q", got, tt.wantID)
}
})
}
}
func TestResolveSessionArtifactPrefersManifestOutput(t *testing.T) {
workspace := t.TempDir()
paths := buildSessionPaths(workspace, "campaign", "session")
if err := os.MkdirAll(paths.ArtifactsDir, 0o755); err != nil {
t.Fatalf("MkdirAll() error = %v", err)
}
manifestPath := filepath.Join(paths.ArtifactsDir, "normalized.from-manifest.json")
if err := os.WriteFile(manifestPath, []byte(`{"segments":[]}`), 0o644); err != nil {
t.Fatalf("WriteFile() error = %v", err)
}
canonicalPath := filepath.Join(paths.TranscriptsDir, "normalized.json")
if err := os.MkdirAll(filepath.Dir(canonicalPath), 0o755); err != nil {
t.Fatalf("MkdirAll() error = %v", err)
}
if err := os.WriteFile(canonicalPath, []byte(`{"segments":[{"id":123}]}`), 0o644); err != nil {
t.Fatalf("WriteFile() error = %v", err)
}
m := manifest.New("session", time.Now().UTC())
m.MarkStageSucceeded("normalize", time.Now().UTC(), []manifest.ArtifactRecord{
{Kind: "transcript_normalized", LocalPath: manifestPath, ProducerRunID: "run-123"},
})
resolved, err := ResolveSessionArtifact(paths, m, "normalized_transcript")
if err != nil {
t.Fatalf("ResolveSessionArtifact() error = %v", err)
}
if resolved.Path != manifestPath {
t.Fatalf("resolved path = %q, want %q", resolved.Path, manifestPath)
}
if resolved.Provenance != "manifest.normalize.outputs" {
t.Fatalf("provenance = %q, want manifest.normalize.outputs", resolved.Provenance)
}
if resolved.ProducerRunID != "run-123" {
t.Fatalf("producer run id = %q, want run-123", resolved.ProducerRunID)
}
}
func TestResolveSessionArtifactFallsBackToCanonicalPath(t *testing.T) {
workspace := t.TempDir()
paths := buildSessionPaths(workspace, "campaign", "session")
canonicalPath := filepath.Join(paths.TranscriptsDir, "trimmed.json")
if err := os.MkdirAll(filepath.Dir(canonicalPath), 0o755); err != nil {
t.Fatalf("MkdirAll() error = %v", err)
}
if err := os.WriteFile(canonicalPath, []byte(`{"segments":[]}`), 0o644); err != nil {
t.Fatalf("WriteFile() error = %v", err)
}
resolved, err := ResolveSessionArtifact(paths, nil, ArtifactTranscriptTrimmed)
if err != nil {
t.Fatalf("ResolveSessionArtifact() error = %v", err)
}
if resolved.Path != canonicalPath {
t.Fatalf("resolved path = %q, want %q", resolved.Path, canonicalPath)
}
if resolved.Provenance != "fallback.canonical_path" {
t.Fatalf("provenance = %q, want fallback.canonical_path", resolved.Provenance)
}
}
func TestResolveSessionArtifactMissingReturnsTypedError(t *testing.T) {
workspace := t.TempDir()
paths := buildSessionPaths(workspace, "campaign", "session")
_, err := ResolveSessionArtifact(paths, nil, ArtifactTranscriptTrimmed)
if err == nil {
t.Fatal("expected error, got nil")
}
if !errors.Is(err, ErrSessionArtifactNotFound) {
t.Fatalf("errors.Is(err, ErrSessionArtifactNotFound) = false; err=%v", err)
}
}
func TestResolveSessionArtifactValidatesTranscriptShape(t *testing.T) {
workspace := t.TempDir()
paths := buildSessionPaths(workspace, "campaign", "session")
canonicalPath := filepath.Join(paths.TranscriptsDir, "processed.json")
if err := os.MkdirAll(filepath.Dir(canonicalPath), 0o755); err != nil {
t.Fatalf("MkdirAll() error = %v", err)
}
if err := os.WriteFile(canonicalPath, []byte(`{"not_segments":[]}`), 0o644); err != nil {
t.Fatalf("WriteFile() error = %v", err)
}
_, err := ResolveSessionArtifact(paths, nil, ArtifactTranscriptPolished)
if err == nil {
t.Fatal("expected error, got nil")
}
if !strings.Contains(err.Error(), "top-level segments is required") {
t.Fatalf("error = %q, want segments validation error", err.Error())
}
}

View File

@@ -30,13 +30,13 @@ func NewLocalStore(workspaceRoot string) *LocalStore {
return &LocalStore{WorkspaceRoot: workspaceRoot}
}
// SessionPaths resolves canonical paths for a session workdir.
func (s *LocalStore) SessionPaths(sessionID string) SessionPaths {
return buildSessionPaths(s.WorkspaceRoot, sessionID)
// SessionPathsFor resolves canonical campaign-aware paths for a session workdir.
func (s *LocalStore) SessionPathsFor(campaign, sessionID string) SessionPaths {
return buildSessionPaths(s.WorkspaceRoot, campaign, sessionID)
}
// EnsureLayout creates and verifies the canonical session workdir directory layout.
func (s *LocalStore) EnsureLayout(sessionID string) (SessionPaths, error) {
// EnsureLayoutFor creates and verifies campaign-aware session layout.
func (s *LocalStore) EnsureLayoutFor(campaign, sessionID string) (SessionPaths, error) {
if strings.TrimSpace(s.WorkspaceRoot) == "" {
return SessionPaths{}, fmt.Errorf("workspace root is required")
}
@@ -44,7 +44,22 @@ func (s *LocalStore) EnsureLayout(sessionID string) (SessionPaths, error) {
return SessionPaths{}, fmt.Errorf("sessionID is required")
}
paths := s.SessionPaths(sessionID)
campaign = strings.TrimSpace(campaign)
if campaign == "" {
return SessionPaths{}, fmt.Errorf("campaign is required")
}
return s.ensureLayout(s.SessionPathsFor(campaign, sessionID))
}
func (s *LocalStore) ensureLayout(paths SessionPaths) (SessionPaths, error) {
if strings.TrimSpace(s.WorkspaceRoot) == "" {
return SessionPaths{}, fmt.Errorf("workspace root is required")
}
if strings.TrimSpace(paths.SessionID) == "" {
return SessionPaths{}, fmt.Errorf("sessionID is required")
}
dirs := []string{
paths.Root,
paths.InputsDir,
@@ -53,8 +68,11 @@ func (s *LocalStore) EnsureLayout(sessionID string) (SessionPaths, error) {
paths.TranscriptsRawDir,
paths.TranscriptsTrimmedDir,
paths.ArtifactsDir,
paths.ReportsDir,
paths.ConfigDir,
paths.LogsDir,
paths.CurrentDir,
paths.RunsDir,
}
for _, dir := range dirs {
@@ -66,13 +84,16 @@ func (s *LocalStore) EnsureLayout(sessionID string) (SessionPaths, error) {
return paths, nil
}
// CopyInput copies an input file into the session workdir under destRelativePath.
func (s *LocalStore) CopyInput(sessionID, srcPath, destRelativePath string) (Ref, error) {
paths, err := s.EnsureLayout(sessionID)
// CopyInputFor copies an input file into the campaign-aware session workdir under destRelativePath.
func (s *LocalStore) CopyInputFor(campaign, sessionID, srcPath, destRelativePath string) (Ref, error) {
paths, err := s.EnsureLayoutFor(campaign, sessionID)
if err != nil {
return Ref{}, err
}
return s.copyInputWithPaths(paths, sessionID, srcPath, destRelativePath)
}
func (s *LocalStore) copyInputWithPaths(paths SessionPaths, sessionID, srcPath, destRelativePath string) (Ref, error) {
destAbs, err := resolveInRoot(paths.Root, destRelativePath)
if err != nil {
return Ref{}, fmt.Errorf("copy input: %w", err)
@@ -173,13 +194,16 @@ func (s *LocalStore) Checksum(path string) (string, error) {
return digest, nil
}
// AcquireSessionLock acquires an exclusive lock file for a session workdir.
func (s *LocalStore) AcquireSessionLock(sessionID string) (*LockHandle, error) {
paths, err := s.EnsureLayout(sessionID)
// AcquireSessionLockFor acquires an exclusive lock file for a campaign/session workdir.
func (s *LocalStore) AcquireSessionLockFor(campaign, sessionID string) (*LockHandle, error) {
paths, err := s.EnsureLayoutFor(campaign, sessionID)
if err != nil {
return nil, err
}
return s.acquireSessionLockForPaths(paths)
}
func (s *LocalStore) acquireSessionLockForPaths(paths SessionPaths) (*LockHandle, error) {
f, err := os.OpenFile(paths.LockPath, os.O_CREATE|os.O_EXCL|os.O_WRONLY, 0o644)
if err != nil {
if errors.Is(err, os.ErrExist) {

View File

@@ -10,9 +10,9 @@ import (
func TestEnsureLayoutCreatesExpectedDirectories(t *testing.T) {
store := NewLocalStore(t.TempDir())
paths, err := store.EnsureLayout("session-1")
paths, err := store.EnsureLayoutFor("sample-campaign", "session-1")
if err != nil {
t.Fatalf("EnsureLayout() error = %v", err)
t.Fatalf("EnsureLayoutFor() error = %v", err)
}
checkDirExists(t, paths.Root)
@@ -22,8 +22,11 @@ func TestEnsureLayoutCreatesExpectedDirectories(t *testing.T) {
checkDirExists(t, paths.TranscriptsRawDir)
checkDirExists(t, paths.TranscriptsTrimmedDir)
checkDirExists(t, paths.ArtifactsDir)
checkDirExists(t, paths.ReportsDir)
checkDirExists(t, paths.ConfigDir)
checkDirExists(t, paths.LogsDir)
checkDirExists(t, paths.CurrentDir)
checkDirExists(t, paths.RunsDir)
if filepath.Base(paths.ManifestPath) != "manifest.json" {
t.Fatalf("ManifestPath = %q, want basename manifest.json", paths.ManifestPath)
@@ -33,6 +36,17 @@ func TestEnsureLayoutCreatesExpectedDirectories(t *testing.T) {
}
}
func TestEnsureLayoutForRequiresCampaign(t *testing.T) {
store := NewLocalStore(t.TempDir())
_, err := store.EnsureLayoutFor("", "session-1")
if err == nil {
t.Fatal("expected campaign-required error, got nil")
}
if !strings.Contains(err.Error(), "campaign is required") {
t.Fatalf("error = %v, want campaign-required error", err)
}
}
func TestChecksumCalculation(t *testing.T) {
store := NewLocalStore(t.TempDir())
path := filepath.Join(t.TempDir(), "sample.txt")
@@ -53,9 +67,9 @@ func TestChecksumCalculation(t *testing.T) {
func TestLockAcquireRelease(t *testing.T) {
store := NewLocalStore(t.TempDir())
lock, err := store.AcquireSessionLock("session-1")
lock, err := store.AcquireSessionLockFor("sample-campaign", "session-1")
if err != nil {
t.Fatalf("AcquireSessionLock() error = %v", err)
t.Fatalf("AcquireSessionLockFor() error = %v", err)
}
exists, err := store.Exists(lock.path)
@@ -81,15 +95,15 @@ func TestLockAcquireRelease(t *testing.T) {
func TestLockConflict(t *testing.T) {
store := NewLocalStore(t.TempDir())
lock1, err := store.AcquireSessionLock("session-1")
lock1, err := store.AcquireSessionLockFor("sample-campaign", "session-1")
if err != nil {
t.Fatalf("first AcquireSessionLock() error = %v", err)
t.Fatalf("first AcquireSessionLockFor() error = %v", err)
}
defer func() {
_ = store.ReleaseSessionLock(lock1)
}()
_, err = store.AcquireSessionLock("session-1")
_, err = store.AcquireSessionLockFor("sample-campaign", "session-1")
if err == nil {
t.Fatal("expected lock conflict error, got nil")
}
@@ -138,9 +152,9 @@ func TestCopyInput(t *testing.T) {
t.Fatalf("WriteFile() error = %v", err)
}
ref, err := store.CopyInput("session-1", srcPath, "inputs/speakers.yml")
ref, err := store.CopyInputFor("sample-campaign", "session-1", srcPath, "inputs/speakers.yml")
if err != nil {
t.Fatalf("CopyInput() error = %v", err)
t.Fatalf("CopyInputFor() error = %v", err)
}
if ref.Kind != "input" {

View File

@@ -2,11 +2,15 @@ package artifacts
import (
"path/filepath"
"gitea.maximumdirect.net/eric/narratio/internal/config"
)
// SessionPaths contains canonical local paths for one session work directory.
type SessionPaths struct {
WorkspaceRoot string
CampaignID string
SessionID string
Root string
InputsDir string
AudioDir string
@@ -14,42 +18,73 @@ type SessionPaths struct {
TranscriptsRawDir string
TranscriptsTrimmedDir string
ArtifactsDir string
ReportsDir string
ConfigDir string
LogsDir string
CurrentDir string
RunsDir string
ManifestPath string
LockPath string
}
// SessionWorkDir returns the work directory for one session.
func SessionWorkDir(rootDir, sessionID string) string {
return filepath.Join(rootDir, "work", sessionID)
// SessionWorkDirForCampaign returns the canonical campaign-aware work directory for one session.
func SessionWorkDirForCampaign(rootDir, campaign, sessionID string) string {
return filepath.Join(rootDir, config.PathWorkDirSegment, campaign, sessionID)
}
// SessionRunWorkDir returns the campaign/session/run scoped local work directory.
func SessionRunWorkDir(rootDir, campaign, sessionID, runID string) string {
return filepath.Join(rootDir, "work", campaign, sessionID, runID)
// SessionManifestPathForCampaign returns the canonical session manifest path.
func SessionManifestPathForCampaign(rootDir, campaign, sessionID string) string {
return filepath.Join(SessionWorkDirForCampaign(rootDir, campaign, sessionID), config.PathManifestFile)
}
// SessionRunsDirForCampaign returns the canonical runs directory for one session.
func SessionRunsDirForCampaign(rootDir, campaign, sessionID string) string {
return filepath.Join(SessionWorkDirForCampaign(rootDir, campaign, sessionID), config.PathRunsDirSegment)
}
// SessionRunRootForCampaign returns the canonical run root under runs/{run_id}.
func SessionRunRootForCampaign(rootDir, campaign, sessionID, runID string) string {
return filepath.Join(SessionRunsDirForCampaign(rootDir, campaign, sessionID), runID)
}
// SessionRunManifestPathForCampaign returns the canonical run manifest path under runs/{run_id}/manifest.json.
func SessionRunManifestPathForCampaign(rootDir, campaign, sessionID, runID string) string {
return filepath.Join(SessionRunRootForCampaign(rootDir, campaign, sessionID, runID), config.PathManifestFile)
}
// SessionRunStageDirForCampaign returns the canonical stage directory under runs/{run_id}/{stage}.
func SessionRunStageDirForCampaign(rootDir, campaign, sessionID, runID, stageName string) string {
return filepath.Join(SessionRunRootForCampaign(rootDir, campaign, sessionID, runID), stageName)
}
// SessionSpoolAudioDir returns the campaign/session/run scoped local spool audio path.
func SessionSpoolAudioDir(spoolRoot, campaign, sessionID, runID string) string {
return filepath.Join(spoolRoot, campaign, sessionID, runID, "audio")
return filepath.Join(spoolRoot, campaign, sessionID, runID, config.PathAudioDirSegment)
}
func buildSessionPaths(workspaceRoot, sessionID string) SessionPaths {
root := SessionWorkDir(workspaceRoot, sessionID)
transcripts := filepath.Join(root, "transcripts")
func buildSessionPaths(workspaceRoot, campaign, sessionID string) SessionPaths {
root := SessionWorkDirForCampaign(workspaceRoot, campaign, sessionID)
return buildSessionPathsFromRoot(workspaceRoot, campaign, sessionID, root)
}
func buildSessionPathsFromRoot(workspaceRoot, campaign, sessionID, root string) SessionPaths {
return SessionPaths{
WorkspaceRoot: workspaceRoot,
CampaignID: campaign,
SessionID: sessionID,
Root: root,
InputsDir: filepath.Join(root, "inputs"),
AudioDir: filepath.Join(root, "audio"),
TranscriptsDir: transcripts,
TranscriptsRawDir: filepath.Join(transcripts, "raw"),
TranscriptsTrimmedDir: filepath.Join(transcripts, "trimmed"),
ArtifactsDir: filepath.Join(root, "artifacts"),
ConfigDir: filepath.Join(root, "config"),
LogsDir: filepath.Join(root, "logs"),
ManifestPath: filepath.Join(root, "manifest.json"),
LockPath: filepath.Join(root, ".lock"),
InputsDir: filepath.Join(root, config.PathInputsDirSegment),
AudioDir: filepath.Join(root, config.PathAudioDirSegment),
TranscriptsDir: filepath.Join(root, config.PathTranscriptsSegment),
TranscriptsRawDir: filepath.Join(root, filepath.FromSlash(config.PathTranscriptsRaw)),
TranscriptsTrimmedDir: filepath.Join(root, filepath.FromSlash(config.PathTranscriptsTrimmed)),
ArtifactsDir: filepath.Join(root, config.PathArtifactsDirSegment),
ReportsDir: filepath.Join(root, config.PathReportsDirSegment),
ConfigDir: filepath.Join(root, config.PathConfigDirSegment),
LogsDir: filepath.Join(root, config.PathLogsDirSegment),
CurrentDir: filepath.Join(root, config.PathCurrentDirSegment),
RunsDir: filepath.Join(root, config.PathRunsDirSegment),
ManifestPath: filepath.Join(root, config.PathManifestFile),
LockPath: filepath.Join(root, config.PathLockFile),
}
}

View File

@@ -5,12 +5,47 @@ import (
"testing"
)
func TestSessionRunWorkDir(t *testing.T) {
func TestSessionWorkDirForCampaign(t *testing.T) {
root := "/tmp/workspace"
got := SessionRunWorkDir(root, "forsaken", "2026-04-19", "20260515T031522Z-a1b2c3d4")
want := filepath.Join(root, "work", "forsaken", "2026-04-19", "20260515T031522Z-a1b2c3d4")
got := SessionWorkDirForCampaign(root, "forsaken", "2026-04-19")
want := filepath.Join(root, "work", "forsaken", "2026-04-19")
if got != want {
t.Fatalf("SessionRunWorkDir() = %q, want %q", got, want)
t.Fatalf("SessionWorkDirForCampaign() = %q, want %q", got, want)
}
}
func TestSessionManifestPathForCampaign(t *testing.T) {
root := "/tmp/workspace"
got := SessionManifestPathForCampaign(root, "forsaken", "2026-04-19")
want := filepath.Join(root, "work", "forsaken", "2026-04-19", "manifest.json")
if got != want {
t.Fatalf("SessionManifestPathForCampaign() = %q, want %q", got, want)
}
}
func TestSessionRunRootAndStageDirForCampaign(t *testing.T) {
root := "/tmp/workspace"
runID := "20260515T031522Z-a1b2c3d4"
runRoot := SessionRunRootForCampaign(root, "forsaken", "2026-04-19", runID)
wantRoot := filepath.Join(root, "work", "forsaken", "2026-04-19", "runs", runID)
if runRoot != wantRoot {
t.Fatalf("SessionRunRootForCampaign() = %q, want %q", runRoot, wantRoot)
}
stageDir := SessionRunStageDirForCampaign(root, "forsaken", "2026-04-19", runID, "transcribe")
wantStage := filepath.Join(root, "work", "forsaken", "2026-04-19", "runs", runID, "transcribe")
if stageDir != wantStage {
t.Fatalf("SessionRunStageDirForCampaign() = %q, want %q", stageDir, wantStage)
}
}
func TestSessionRunManifestPathForCampaign(t *testing.T) {
root := "/tmp/workspace"
runID := "20260515T031522Z-a1b2c3d4"
got := SessionRunManifestPathForCampaign(root, "forsaken", "2026-04-19", runID)
want := filepath.Join(root, "work", "forsaken", "2026-04-19", "runs", runID, "manifest.json")
if got != want {
t.Fatalf("SessionRunManifestPathForCampaign() = %q, want %q", got, want)
}
}

View File

@@ -8,7 +8,7 @@ import (
func TestResolveSessionLocalPathForRead(t *testing.T) {
workspace := t.TempDir()
paths := buildSessionPaths(workspace, "s-1")
paths := buildSessionPaths(workspace, "sample-campaign", "s-1")
if err := os.MkdirAll(paths.TranscriptsRawDir, 0o755); err != nil {
t.Fatalf("MkdirAll() error = %v", err)
}
@@ -41,7 +41,7 @@ func TestResolveSessionLocalPathForReadRelativeWorkspaceRootQualifiedPath(t *tes
t.Fatalf("Rel() error = %v", err)
}
paths := buildSessionPaths(workspaceRel, "s-1")
paths := buildSessionPaths(workspaceRel, "sample-campaign", "s-1")
target := filepath.Join(paths.TranscriptsRawDir, "alice.json")
if err := os.MkdirAll(filepath.Dir(target), 0o755); err != nil {
t.Fatalf("MkdirAll() error = %v", err)
@@ -50,7 +50,7 @@ func TestResolveSessionLocalPathForReadRelativeWorkspaceRootQualifiedPath(t *tes
t.Fatalf("WriteFile() error = %v", err)
}
manifestPath := filepath.Join(workspaceRel, "work", "s-1", "transcripts", "raw", "alice.json")
manifestPath := filepath.Join(workspaceRel, "work", "sample-campaign", "s-1", "transcripts", "raw", "alice.json")
got := ResolveSessionLocalPathForRead(paths, manifestPath)
if got != filepath.Clean(manifestPath) {
t.Fatalf("resolution = %q, want %q", got, filepath.Clean(manifestPath))

View File

@@ -11,19 +11,19 @@ type S3Store struct {
Prefix string
}
// SessionPaths is not implemented for S3-backed storage.
func (s *S3Store) SessionPaths(_ string) SessionPaths {
// SessionPathsFor is not implemented for S3-backed storage.
func (s *S3Store) SessionPathsFor(_, _ string) SessionPaths {
return SessionPaths{}
}
// EnsureLayout returns a not-yet-implemented error in the scaffold.
func (s *S3Store) EnsureLayout(_ string) (SessionPaths, error) {
return SessionPaths{}, fmt.Errorf("artifacts s3 ensure layout: not yet implemented")
// EnsureLayoutFor returns a not-yet-implemented error in the scaffold.
func (s *S3Store) EnsureLayoutFor(_, _ string) (SessionPaths, error) {
return SessionPaths{}, fmt.Errorf("artifacts s3 ensure layout for campaign/session: not yet implemented")
}
// CopyInput returns a not-yet-implemented error in the scaffold.
func (s *S3Store) CopyInput(_, _, _ string) (Ref, error) {
return Ref{}, fmt.Errorf("artifacts s3 copy input: not yet implemented")
// CopyInputFor returns a not-yet-implemented error in the scaffold.
func (s *S3Store) CopyInputFor(_, _, _, _ string) (Ref, error) {
return Ref{}, fmt.Errorf("artifacts s3 copy input for campaign/session: not yet implemented")
}
// Exists returns a not-yet-implemented error in the scaffold.
@@ -46,9 +46,9 @@ func (s *S3Store) Checksum(_ string) (string, error) {
return "", fmt.Errorf("artifacts s3 checksum: not yet implemented")
}
// AcquireSessionLock returns a not-yet-implemented error in the scaffold.
func (s *S3Store) AcquireSessionLock(_ string) (*LockHandle, error) {
return nil, fmt.Errorf("artifacts s3 acquire lock: not yet implemented")
// AcquireSessionLockFor returns a not-yet-implemented error in the scaffold.
func (s *S3Store) AcquireSessionLockFor(_, _ string) (*LockHandle, error) {
return nil, fmt.Errorf("artifacts s3 acquire lock for campaign/session: not yet implemented")
}
// ReleaseSessionLock returns a not-yet-implemented error in the scaffold.

View File

@@ -3,6 +3,8 @@ package artifacts
import (
"path"
"strings"
"gitea.maximumdirect.net/eric/narratio/internal/config"
)
// S3SessionPrefix builds the canonical S3 session prefix.
@@ -10,9 +12,9 @@ import (
func S3SessionPrefix(rootPrefix, campaign, sessionID string) string {
prefix := path.Join(
cleanS3PathPart(rootPrefix),
"campaigns",
config.S3CampaignsSegment,
cleanS3PathPart(campaign),
"sessions",
config.S3SessionsSegment,
cleanS3PathPart(sessionID),
)
return ensureS3TrailingSlash(prefix)
@@ -21,7 +23,7 @@ func S3SessionPrefix(rootPrefix, campaign, sessionID string) string {
// S3RunPrefix builds the canonical S3 run prefix.
// Format: {session_prefix}/runs/{run_id}/
func S3RunPrefix(sessionPrefix, runID string) string {
prefix := path.Join(strings.TrimSuffix(cleanS3Key(sessionPrefix), "/"), "runs", cleanS3PathPart(runID))
prefix := path.Join(strings.TrimSuffix(cleanS3Key(sessionPrefix), "/"), config.S3RunsSegment, cleanS3PathPart(runID))
return ensureS3TrailingSlash(prefix)
}
@@ -35,13 +37,13 @@ func S3AudioPrefix(sessionPrefix, audioPrefix string) string {
// S3CurrentManifestKey returns the current manifest pointer key.
// Format: {session_prefix}/current/manifest.json
func S3CurrentManifestKey(sessionPrefix string) string {
return path.Join(strings.TrimSuffix(cleanS3Key(sessionPrefix), "/"), "current", "manifest.json")
return path.Join(strings.TrimSuffix(cleanS3Key(sessionPrefix), "/"), config.S3CurrentSegment, config.S3ManifestFile)
}
// S3CurrentRunPointerKey returns the current run pointer key.
// Format: {session_prefix}/current/run_id.txt
func S3CurrentRunPointerKey(sessionPrefix string) string {
return path.Join(strings.TrimSuffix(cleanS3Key(sessionPrefix), "/"), "current", "run_id.txt")
return path.Join(strings.TrimSuffix(cleanS3Key(sessionPrefix), "/"), config.S3CurrentSegment, config.S3RunIDFile)
}
// S3PromotedArtifactKey returns the destination key for one promoted artifact.

View File

@@ -15,13 +15,13 @@ type Ref struct {
// Store is the local artifact/workdir abstraction used by orchestration code.
type Store interface {
SessionPaths(sessionID string) SessionPaths
EnsureLayout(sessionID string) (SessionPaths, error)
CopyInput(sessionID, srcPath, destRelativePath string) (Ref, error)
SessionPathsFor(campaign, sessionID string) SessionPaths
EnsureLayoutFor(campaign, sessionID string) (SessionPaths, error)
CopyInputFor(campaign, sessionID, srcPath, destRelativePath string) (Ref, error)
Exists(path string) (bool, error)
ExistsRef(ref Ref) (bool, error)
WriteFileAtomic(path string, data []byte, perm os.FileMode) error
Checksum(path string) (string, error)
AcquireSessionLock(sessionID string) (*LockHandle, error)
AcquireSessionLockFor(campaign, sessionID string) (*LockHandle, error)
ReleaseSessionLock(lock *LockHandle) error
}

View File

@@ -60,6 +60,8 @@ type StorageS3Config struct {
Region string `yaml:"region"`
Endpoint string `yaml:"endpoint"`
ForcePathStyle bool `yaml:"force_path_style"`
AccessKeyIDEnv string `yaml:"access_key_id_env"`
SecretKeyEnv string `yaml:"secret_access_key_env"`
}
// SpoolConfig configures local spool storage for staged data.

View File

@@ -8,8 +8,74 @@ const (
DefaultSessionConfigPathLocal = "./session.yml"
DefaultSessionConfigPathUsrLocal = "/usr/local/etc/narratio/session.yml"
DefaultSessionConfigPathEtc = "/etc/narratio/session.yml"
DefaultS3AccessKeyIDEnv = "OBJECT_STORAGE_KEY_ID"
DefaultS3SecretAccessKeyEnv = "OBJECT_STORAGE_KEY"
DefaultStorageS3RootPrefix = "dnd"
DefaultSpoolRoot = "/var/spool/narratio"
DefaultWhisperXLanguage = "en"
DefaultWhisperXTimeout = "30m"
DefaultWhisperXRetryDelay = "2s"
DefaultWhisperXConcurrency = 2
DefaultWhisperXRetries = 3
DefaultSeriatimBinary = "seriatim"
DefaultSeriatimTimeout = "10m"
DefaultSeriatimOutputSchema = "seriatim-intermediate"
DefaultSeriatimCoalesceGap = 3.0
DefaultSeriatimReport = true
DefaultAuditaBinary = "audita"
DefaultAuditaTimeout = "3h"
DefaultAuditaReport = true
DefaultScriptoriumBinary = "scriptorium"
DefaultScriptoriumTimeout = "10m"
DefaultTrimBoundsTimeout = "10m"
DefaultTrimSeriatimReport = false
DefaultNormalizeOutputPath = "transcripts/normalized.json"
DefaultNormalizeOutputSchema = "seriatim-intermediate"
DefaultNormalizeReport = true
DefaultArchiveEnabled = true
DefaultArchiveUploadRun = true
PathWorkDirSegment = "work"
PathInputsDirSegment = "inputs"
PathAudioDirSegment = "audio"
PathTranscriptsSegment = "transcripts"
PathTranscriptsRaw = "transcripts/raw"
PathTranscriptsTrimmed = "transcripts/trimmed"
PathArtifactsDirSegment = "artifacts"
PathReportsDirSegment = "reports"
PathConfigDirSegment = "config"
PathLogsDirSegment = "logs"
PathCurrentDirSegment = "current"
PathRunsDirSegment = "runs"
PathManifestFile = "manifest.json"
PathLockFile = ".lock"
PathTranscriptMerged = "transcripts/merged.json"
PathTranscriptProcessed = "transcripts/processed.json"
PathTranscriptNormalized = "transcripts/normalized.json"
PathTranscriptTrimmed = "transcripts/trimmed.json"
S3CampaignsSegment = "campaigns"
S3SessionsSegment = "sessions"
S3RunsSegment = "runs"
S3CurrentSegment = "current"
S3ManifestFile = "manifest.json"
S3RunIDFile = "run_id.txt"
)
// DefaultArchivePromoteArtifacts defines the default archive promotion rules.
// Callers should copy this slice before mutating.
var DefaultArchivePromoteArtifacts = []ArchivePromotionRule{
{From: PathTranscriptTrimmed, To: PathTranscriptTrimmed},
{From: "artifacts/session_recap.md", To: "artifacts/session_recap.md"},
}
// DefaultPipelineConfigSearchPaths defines the default search order for
// pipeline.yml when callers do not provide an explicit path.
//

View File

@@ -174,7 +174,13 @@ func applyStorageDefaults(cfg *StorageConfig) {
cfg.S3 = &StorageS3Config{}
}
if cfg.S3.RootPrefix == "" {
cfg.S3.RootPrefix = "dnd"
cfg.S3.RootPrefix = DefaultStorageS3RootPrefix
}
if cfg.S3.AccessKeyIDEnv == "" {
cfg.S3.AccessKeyIDEnv = DefaultS3AccessKeyIDEnv
}
if cfg.S3.SecretKeyEnv == "" {
cfg.S3.SecretKeyEnv = DefaultS3SecretAccessKeyEnv
}
}
@@ -183,7 +189,7 @@ func applySpoolDefaults(cfg *SpoolConfig) {
return
}
if cfg.Root == "" {
cfg.Root = "/var/spool/narratio"
cfg.Root = DefaultSpoolRoot
}
}
@@ -196,16 +202,13 @@ func applyArchiveDefaults(cfg **ArchiveConfig) {
}
if (*cfg).Enabled == nil {
(*cfg).Enabled = boolPtr(true)
(*cfg).Enabled = boolPtr(DefaultArchiveEnabled)
}
if (*cfg).UploadRun == nil {
(*cfg).UploadRun = boolPtr(true)
(*cfg).UploadRun = boolPtr(DefaultArchiveUploadRun)
}
if len((*cfg).PromoteArtifacts) == 0 {
(*cfg).PromoteArtifacts = []ArchivePromotionRule{
{From: "transcripts/trimmed.json", To: "transcripts/trimmed.json", Required: boolPtr(true)},
{From: "artifacts/session_recap.md", To: "artifacts/session_recap.md", Required: boolPtr(true)},
}
(*cfg).PromoteArtifacts = append([]ArchivePromotionRule(nil), DefaultArchivePromoteArtifacts...)
}
for i := range (*cfg).PromoteArtifacts {
if (*cfg).PromoteArtifacts[i].Required == nil {
@@ -219,19 +222,19 @@ func applyWhisperXDefaults(cfg *WhisperXConfig) {
return
}
if cfg.Language == "" {
cfg.Language = "en"
cfg.Language = DefaultWhisperXLanguage
}
if cfg.Timeout == "" {
cfg.Timeout = "30m"
cfg.Timeout = DefaultWhisperXTimeout
}
if cfg.RetryDelay == "" {
cfg.RetryDelay = "2s"
cfg.RetryDelay = DefaultWhisperXRetryDelay
}
if cfg.Concurrency == nil {
cfg.Concurrency = intPtr(2)
cfg.Concurrency = intPtr(DefaultWhisperXConcurrency)
}
if cfg.Retries == nil {
cfg.Retries = intPtr(3)
cfg.Retries = intPtr(DefaultWhisperXRetries)
}
}
@@ -245,19 +248,19 @@ func applySeriatimDefaults(cfg *SeriatimConfig) {
return
}
if cfg.Binary == "" {
cfg.Binary = "seriatim"
cfg.Binary = DefaultSeriatimBinary
}
if cfg.Timeout == "" {
cfg.Timeout = "10m"
cfg.Timeout = DefaultSeriatimTimeout
}
if cfg.OutputSchema == "" {
cfg.OutputSchema = "seriatim-intermediate"
cfg.OutputSchema = DefaultSeriatimOutputSchema
}
if cfg.CoalesceGap == nil {
cfg.CoalesceGap = float64Ptr(3.0)
cfg.CoalesceGap = float64Ptr(DefaultSeriatimCoalesceGap)
}
if cfg.Report == nil {
cfg.Report = boolPtr(true)
cfg.Report = boolPtr(DefaultSeriatimReport)
}
}
@@ -266,13 +269,13 @@ func applyAuditaDefaults(cfg *AuditaConfig) {
return
}
if cfg.Binary == "" {
cfg.Binary = "audita"
cfg.Binary = DefaultAuditaBinary
}
if cfg.Timeout == "" {
cfg.Timeout = "3h"
cfg.Timeout = DefaultAuditaTimeout
}
if cfg.Report == nil {
cfg.Report = boolPtr(true)
cfg.Report = boolPtr(DefaultAuditaReport)
}
}
@@ -281,10 +284,10 @@ func applyScriptoriumDefaults(cfg *ScriptoriumConfig) {
return
}
if cfg.Binary == "" {
cfg.Binary = "scriptorium"
cfg.Binary = DefaultScriptoriumBinary
}
if cfg.Timeout == "" {
cfg.Timeout = "10m"
cfg.Timeout = DefaultScriptoriumTimeout
}
}
@@ -293,10 +296,10 @@ func applyTrimDefaults(cfg *TrimConfig) {
return
}
if cfg.Bounds.Timeout == "" {
cfg.Bounds.Timeout = "10m"
cfg.Bounds.Timeout = DefaultTrimBoundsTimeout
}
if cfg.Seriatim.Report == nil {
cfg.Seriatim.Report = boolPtr(false)
cfg.Seriatim.Report = boolPtr(DefaultTrimSeriatimReport)
}
}
@@ -305,13 +308,13 @@ func applyNormalizeDefaults(cfg *NormalizeConfig) {
return
}
if cfg.OutputSchema == "" {
cfg.OutputSchema = defaultNormalizeOutputSchema
cfg.OutputSchema = DefaultNormalizeOutputSchema
}
if cfg.OutputPath == "" && !cfg.outputPathWasSet() {
cfg.OutputPath = defaultNormalizeOutputPath
cfg.OutputPath = DefaultNormalizeOutputPath
}
if cfg.Report == nil {
cfg.Report = boolPtr(true)
cfg.Report = boolPtr(DefaultNormalizeReport)
}
}

View File

@@ -6,11 +6,6 @@ import (
"gopkg.in/yaml.v3"
)
const (
defaultNormalizeOutputPath = "transcripts/normalized.json"
defaultNormalizeOutputSchema = "seriatim-intermediate"
)
// UnmarshalYAML tracks explicit normalize.output_path presence so validation can
// distinguish omitted vs explicitly empty values.
func (cfg *NormalizeConfig) UnmarshalYAML(node *yaml.Node) error {

View File

@@ -116,6 +116,37 @@ func TestScriptoriumLoadAndValidate(t *testing.T) {
output_kind: session_recap
`,
},
{
name: "canonical artifact source is accepted",
scriptoriumYAML: `scriptorium:
binary: scriptorium
artifacts:
session_recap:
enabled: true
prompt_id: dnd.session_recap
output_path: artifacts/session_recap.md
inputs:
transcript:
source: narratio.transcript.trimmed
required: true
`,
},
{
name: "unknown artifact source fails validation",
scriptoriumYAML: `scriptorium:
binary: scriptorium
artifacts:
session_recap:
enabled: true
prompt_id: dnd.session_recap
output_path: artifacts/session_recap.md
inputs:
transcript:
source: narratio.unknown
required: true
`,
wantValidateErr: `pipeline.scriptorium.artifacts.session_recap.inputs.transcript.source "narratio.unknown" is unsupported`,
},
{
name: "artifact render_debug override is accepted",
scriptoriumYAML: `scriptorium:

View File

@@ -24,6 +24,12 @@ storage:
if cfg.Pipeline.Storage.S3.RootPrefix != "dnd" {
t.Fatalf("storage.s3.root_prefix = %q, want dnd", cfg.Pipeline.Storage.S3.RootPrefix)
}
if cfg.Pipeline.Storage.S3.AccessKeyIDEnv != DefaultS3AccessKeyIDEnv {
t.Fatalf("storage.s3.access_key_id_env = %q, want %q", cfg.Pipeline.Storage.S3.AccessKeyIDEnv, DefaultS3AccessKeyIDEnv)
}
if cfg.Pipeline.Storage.S3.SecretKeyEnv != DefaultS3SecretAccessKeyEnv {
t.Fatalf("storage.s3.secret_access_key_env = %q, want %q", cfg.Pipeline.Storage.S3.SecretKeyEnv, DefaultS3SecretAccessKeyEnv)
}
if cfg.Pipeline.Storage.S3.ForcePathStyle {
t.Fatalf("storage.s3.force_path_style = true, want false default")
}
@@ -33,6 +39,77 @@ storage:
}
}
func TestStorageS3CredentialEnvNamesLoadAndValidate(t *testing.T) {
pipelineYAML := testPipelineBaseYAML + `
storage:
backend: s3
s3:
bucket: my-dnd-archive
access_key_id_env: CUSTOM_KEY_ID
secret_access_key_env: CUSTOM_SECRET
`
pipelinePath, sessionPath := writeConfigFiles(t, pipelineYAML, testSessionBaseYAML)
cfg, err := Load(pipelinePath, sessionPath)
if err != nil {
t.Fatalf("Load() error = %v", err)
}
if cfg.Pipeline.Storage.S3.AccessKeyIDEnv != "CUSTOM_KEY_ID" {
t.Fatalf("storage.s3.access_key_id_env = %q, want CUSTOM_KEY_ID", cfg.Pipeline.Storage.S3.AccessKeyIDEnv)
}
if cfg.Pipeline.Storage.S3.SecretKeyEnv != "CUSTOM_SECRET" {
t.Fatalf("storage.s3.secret_access_key_env = %q, want CUSTOM_SECRET", cfg.Pipeline.Storage.S3.SecretKeyEnv)
}
if err := Validate(cfg); err != nil {
t.Fatalf("Validate() error = %v", err)
}
}
func TestStorageS3CredentialEnvValidation(t *testing.T) {
tests := []struct {
name string
pipelineYML string
wantErr string
}{
{
name: "invalid access key env name",
pipelineYML: testPipelineBaseYAML + `
storage:
backend: s3
s3:
bucket: my-dnd-archive
access_key_id_env: "123BAD"
`,
wantErr: "pipeline.storage.s3.access_key_id_env must be a valid environment variable name",
},
{
name: "invalid secret key env name",
pipelineYML: testPipelineBaseYAML + `
storage:
backend: s3
s3:
bucket: my-dnd-archive
secret_access_key_env: "bad-name"
`,
wantErr: "pipeline.storage.s3.secret_access_key_env must be a valid environment variable name",
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
pipelinePath, sessionPath := writeConfigFiles(t, tt.pipelineYML, testSessionBaseYAML)
cfg, err := Load(pipelinePath, sessionPath)
if err != nil {
t.Fatalf("Load() error = %v", err)
}
err = Validate(cfg)
if err == nil || !strings.Contains(err.Error(), tt.wantErr) {
t.Fatalf("Validate() error = %v, want to contain %q", err, tt.wantErr)
}
})
}
}
func TestSpoolAndArchiveDefaults(t *testing.T) {
pipelinePath, sessionPath := writeConfigFiles(t, testPipelineBaseYAML, testSessionBaseYAML)

View File

@@ -91,6 +91,12 @@ func validateStorage(cfg StorageConfig) error {
if cfg.S3.Endpoint != "" && strings.TrimSpace(cfg.S3.Endpoint) == "" {
return fmt.Errorf("pipeline.storage.s3.endpoint must be non-empty when provided")
}
if err := validateEnvVarNameField("pipeline.storage.s3.access_key_id_env", cfg.S3.AccessKeyIDEnv); err != nil {
return err
}
if err := validateEnvVarNameField("pipeline.storage.s3.secret_access_key_env", cfg.S3.SecretKeyEnv); err != nil {
return err
}
return nil
}
@@ -337,9 +343,13 @@ func validateScriptorium(cfg *ScriptoriumConfig) error {
if trimmedInputName == "" {
return fmt.Errorf("pipeline.scriptorium.artifacts.%s.inputs keys must be non-empty", artifactName)
}
if strings.TrimSpace(inputCfg.Source) == "" {
source := strings.TrimSpace(inputCfg.Source)
if source == "" {
return fmt.Errorf("pipeline.scriptorium.artifacts.%s.inputs.%s.source is required", artifactName, inputName)
}
if !isSupportedScriptoriumInputSource(source) {
return fmt.Errorf("pipeline.scriptorium.artifacts.%s.inputs.%s.source %q is unsupported", artifactName, inputName, inputCfg.Source)
}
}
for varName, varValue := range artifactCfg.Vars {
if strings.TrimSpace(varName) == "" {
@@ -429,7 +439,46 @@ func archiveUploadConfiguredForS3(pipeline *PipelineConfig) bool {
return enabled && upload
}
func isSupportedScriptoriumInputSource(source string) bool {
switch strings.TrimSpace(source) {
case "previous_session_artifact":
return true
case "processed_transcript":
return true
case "normalized_transcript":
return true
case "trimmed_transcript":
return true
case "narratio.transcript.merged":
return true
case "narratio.transcript.polished":
return true
case "narratio.transcript.full":
return true
case "narratio.transcript.trimmed":
return true
case "narratio.bounds.session":
return true
case "narratio.artifact.session_recap":
return true
default:
return false
}
}
var windowsAbsPathRE = regexp.MustCompile(`^[A-Za-z]:[\\/].*`)
var envVarNameRE = regexp.MustCompile(`^[A-Za-z_][A-Za-z0-9_]*$`)
func validateEnvVarNameField(fieldName, value string) error {
trimmed := strings.TrimSpace(value)
if trimmed == "" {
return fmt.Errorf("%s must be non-empty", fieldName)
}
if !envVarNameRE.MatchString(trimmed) {
return fmt.Errorf("%s must be a valid environment variable name", fieldName)
}
return nil
}
func validateRelativeSafePath(fieldName, value string) error {
trimmed := strings.TrimSpace(value)

View File

@@ -29,8 +29,10 @@ type InputRecord struct {
type ArtifactRecord struct {
Kind string `json:"kind"`
LocalPath string `json:"local_path"`
RemoteKey string `json:"remote_key,omitempty"`
Checksum string `json:"checksum,omitempty"`
// ProducerRunID identifies the run that produced this durable artifact.
ProducerRunID string `json:"producer_run_id,omitempty"`
RemoteKey string `json:"remote_key,omitempty"`
Checksum string `json:"checksum,omitempty"`
}
// StageRecord tracks lifecycle and provenance for one pipeline stage.
@@ -120,6 +122,15 @@ func (m *Manifest) MarkStageSkipped(name string, at time.Time, reason string) {
m.UpdatedAt = at
}
// MarkStageStale marks a stage as stale so it is not skipped as idempotently complete.
func (m *Manifest) MarkStageStale(name string, at time.Time, reason string) {
s := m.ensureStage(name, at)
s.Status = StatusStale
s.Error = &ErrorRecord{Message: strings.TrimSpace(reason), Code: "stale", At: timePtr(at)}
s.UpdatedAt = at
m.UpdatedAt = at
}
func (m *Manifest) ensureStage(name string, at time.Time) *StageRecord {
if m.Stages == nil {
m.Stages = map[string]*StageRecord{}

View File

@@ -0,0 +1,164 @@
package manifest
import (
"strings"
"time"
)
type RunManifestStatus string
const (
RunManifestStatusRunning RunManifestStatus = "running"
RunManifestStatusSucceeded RunManifestStatus = "succeeded"
RunManifestStatusFailed RunManifestStatus = "failed"
)
type RunStageAction string
const (
RunStageActionRun RunStageAction = "run"
RunStageActionSkip RunStageAction = "skip"
)
// RunStageRecord tracks lifecycle and provenance for one stage within a single invocation.
type RunStageRecord struct {
Name string `json:"name"`
Action RunStageAction `json:"action"`
Status StageStatus `json:"status"`
CreatedAt time.Time `json:"created_at"`
UpdatedAt time.Time `json:"updated_at"`
StartedAt *time.Time `json:"started_at,omitempty"`
CompletedAt *time.Time `json:"completed_at,omitempty"`
Outputs []ArtifactRecord `json:"outputs,omitempty"`
Logs []string `json:"logs,omitempty"`
GeneratedConfigs []string `json:"generated_configs,omitempty"`
Error *ErrorRecord `json:"error,omitempty"`
Metadata map[string]any `json:"metadata,omitempty"`
}
// RunManifest is the invocation-scoped execution record under runs/{run_id}/manifest.json.
type RunManifest struct {
SessionID string `json:"session_id"`
Campaign string `json:"campaign,omitempty"`
RunID string `json:"run_id"`
Force bool `json:"force"`
RequestedStages []string `json:"requested_stages,omitempty"`
SessionManifestPath string `json:"session_manifest_path,omitempty"`
LocalWorkDir string `json:"local_workdir,omitempty"`
LocalSpoolDir string `json:"local_spool_dir,omitempty"`
S3Bucket string `json:"s3_bucket,omitempty"`
S3SessionPrefix string `json:"s3_session_prefix,omitempty"`
S3RunPrefix string `json:"s3_run_prefix,omitempty"`
CreatedAt time.Time `json:"created_at"`
UpdatedAt time.Time `json:"updated_at"`
StartedAt *time.Time `json:"started_at,omitempty"`
CompletedAt *time.Time `json:"completed_at,omitempty"`
Status RunManifestStatus `json:"status"`
LastError *ErrorRecord `json:"last_error,omitempty"`
Stages map[string]*RunStageRecord `json:"stages"`
Metadata map[string]any `json:"metadata,omitempty"`
}
// NewRun constructs a new run manifest with deterministic timestamps.
func NewRun(sessionID, campaign, runID string, force bool, requestedStages []string, now time.Time) *RunManifest {
return &RunManifest{
SessionID: strings.TrimSpace(sessionID),
Campaign: strings.TrimSpace(campaign),
RunID: strings.TrimSpace(runID),
Force: force,
RequestedStages: append([]string(nil), requestedStages...),
CreatedAt: now,
UpdatedAt: now,
StartedAt: timePtr(now),
Status: RunManifestStatusRunning,
Stages: map[string]*RunStageRecord{},
}
}
func (m *RunManifest) SetStageAction(name string, action RunStageAction, at time.Time) {
s := m.ensureStage(name, at)
s.Action = action
s.UpdatedAt = at
m.UpdatedAt = at
}
func (m *RunManifest) MarkStageRunning(name string, at time.Time) {
s := m.ensureStage(name, at)
s.Status = StatusRunning
s.StartedAt = timePtr(at)
s.CompletedAt = nil
s.Error = nil
s.UpdatedAt = at
m.UpdatedAt = at
}
func (m *RunManifest) MarkStageSucceeded(name string, at time.Time, outputs []ArtifactRecord) {
s := m.ensureStage(name, at)
s.Status = StatusSucceeded
s.CompletedAt = timePtr(at)
s.Error = nil
s.Outputs = append([]ArtifactRecord(nil), outputs...)
s.UpdatedAt = at
m.UpdatedAt = at
}
func (m *RunManifest) MarkStageFailed(name string, at time.Time, message string) {
s := m.ensureStage(name, at)
s.Status = StatusFailed
s.CompletedAt = timePtr(at)
s.Error = &ErrorRecord{Message: strings.TrimSpace(message), At: timePtr(at)}
s.UpdatedAt = at
m.LastError = &ErrorRecord{Message: strings.TrimSpace(message), At: timePtr(at)}
m.UpdatedAt = at
m.Status = RunManifestStatusFailed
m.CompletedAt = timePtr(at)
}
func (m *RunManifest) MarkStageSkipped(name string, at time.Time, reason string) {
s := m.ensureStage(name, at)
s.Status = StatusSkipped
s.CompletedAt = timePtr(at)
s.Error = &ErrorRecord{Message: strings.TrimSpace(reason), Code: "skipped", At: timePtr(at)}
s.UpdatedAt = at
m.UpdatedAt = at
}
func (m *RunManifest) MarkSucceeded(at time.Time) {
m.Status = RunManifestStatusSucceeded
m.CompletedAt = timePtr(at)
m.UpdatedAt = at
}
func (m *RunManifest) MarkFailed(at time.Time, message string) {
m.Status = RunManifestStatusFailed
m.CompletedAt = timePtr(at)
m.LastError = &ErrorRecord{Message: strings.TrimSpace(message), At: timePtr(at)}
m.UpdatedAt = at
}
func (m *RunManifest) ensureStage(name string, at time.Time) *RunStageRecord {
if m.Stages == nil {
m.Stages = map[string]*RunStageRecord{}
}
stageName := strings.TrimSpace(name)
s, ok := m.Stages[stageName]
if !ok || s == nil {
s = &RunStageRecord{
Name: stageName,
Action: RunStageActionRun,
Status: StatusPending,
CreatedAt: at,
UpdatedAt: at,
}
m.Stages[stageName] = s
}
if s.Name == "" {
s.Name = stageName
}
if s.CreatedAt.IsZero() {
s.CreatedAt = at
}
return s
}

View File

@@ -0,0 +1,50 @@
package manifest
import (
"testing"
"time"
)
func TestRunManifestStageMarkHelpers(t *testing.T) {
rm := NewRun("2026-05-03", "forsaken", "20260517T000000Z-abcdef12", false, []string{"prepare"}, time.Date(2026, 5, 3, 10, 0, 0, 0, time.UTC))
if rm.Status != RunManifestStatusRunning {
t.Fatalf("status = %q, want %q", rm.Status, RunManifestStatusRunning)
}
runningAt := time.Date(2026, 5, 3, 10, 1, 0, 0, time.UTC)
rm.SetStageAction("prepare", RunStageActionRun, runningAt)
rm.MarkStageRunning("prepare", runningAt)
rm.MarkStageSucceeded("prepare", runningAt.Add(30*time.Second), []ArtifactRecord{
{Kind: "input", LocalPath: "inputs/session.yml"},
})
stage := rm.Stages["prepare"]
if stage == nil {
t.Fatal("prepare stage missing")
}
if stage.Action != RunStageActionRun {
t.Fatalf("action = %q, want %q", stage.Action, RunStageActionRun)
}
if stage.Status != StatusSucceeded {
t.Fatalf("status = %q, want %q", stage.Status, StatusSucceeded)
}
skippedAt := runningAt.Add(1 * time.Minute)
rm.SetStageAction("notify", RunStageActionSkip, skippedAt)
rm.MarkStageSkipped("notify", skippedAt, "already_succeeded")
skipped := rm.Stages["notify"]
if skipped == nil {
t.Fatal("notify stage missing")
}
if skipped.Action != RunStageActionSkip {
t.Fatalf("action = %q, want %q", skipped.Action, RunStageActionSkip)
}
if skipped.Status != StatusSkipped {
t.Fatalf("status = %q, want %q", skipped.Status, StatusSkipped)
}
rm.MarkSucceeded(skippedAt.Add(10 * time.Second))
if rm.Status != RunManifestStatusSucceeded {
t.Fatalf("status = %q, want %q", rm.Status, RunManifestStatusSucceeded)
}
}

View File

@@ -129,6 +129,89 @@ func (s *LocalStore) Save(ctx context.Context, path string, m *Manifest) error {
return nil
}
// CreateRun returns a new in-memory run manifest for one invocation.
func (s *LocalStore) CreateRun(
ctx context.Context,
sessionID, campaign, runID string,
force bool,
requestedStages []string,
) (*RunManifest, error) {
if err := checkContext(ctx); err != nil {
return nil, err
}
if strings.TrimSpace(sessionID) == "" {
return nil, fmt.Errorf("create run manifest: session_id is required")
}
if strings.TrimSpace(runID) == "" {
return nil, fmt.Errorf("create run manifest: run_id is required")
}
now := time.Now().UTC()
return NewRun(sessionID, campaign, runID, force, requestedStages, now), nil
}
// LoadRun reads and validates a local JSON run manifest from path.
func (s *LocalStore) LoadRun(ctx context.Context, path string) (*RunManifest, error) {
if err := checkContext(ctx); err != nil {
return nil, err
}
if strings.TrimSpace(path) == "" {
return nil, fmt.Errorf("load run manifest: path is required")
}
data, err := os.ReadFile(path)
if err != nil {
return nil, fmt.Errorf("load run manifest %q: %w", path, err)
}
var m RunManifest
if err := json.Unmarshal(data, &m); err != nil {
return nil, fmt.Errorf("decode run manifest %q: %w", path, err)
}
if err := validateLoadedRunManifest(&m); err != nil {
return nil, fmt.Errorf("run manifest %q invalid: %w", path, err)
}
normalizeRunManifest(&m)
return &m, nil
}
// SaveRun writes the run manifest to path atomically via temp file + rename.
func (s *LocalStore) SaveRun(ctx context.Context, path string, m *RunManifest) error {
if err := checkContext(ctx); err != nil {
return err
}
if strings.TrimSpace(path) == "" {
return fmt.Errorf("save run manifest: path is required")
}
if m == nil {
return fmt.Errorf("save run manifest: manifest is nil")
}
if strings.TrimSpace(m.SessionID) == "" {
return fmt.Errorf("save run manifest: session_id is required")
}
if strings.TrimSpace(m.RunID) == "" {
return fmt.Errorf("save run manifest: run_id is required")
}
if m.CreatedAt.IsZero() {
return fmt.Errorf("save run manifest: created_at is required")
}
m.UpdatedAt = time.Now().UTC()
if m.Stages == nil {
m.Stages = map[string]*RunStageRecord{}
}
data, err := json.MarshalIndent(m, "", " ")
if err != nil {
return fmt.Errorf("save run manifest: marshal: %w", err)
}
data = append(data, '\n')
return writeJSONAtomically(ctx, path, ".run-manifest.json.tmp-*", data)
}
func validateLoadedManifest(m *Manifest) error {
if m == nil {
return fmt.Errorf("manifest is nil")
@@ -161,6 +244,88 @@ func normalizeManifest(m *Manifest) {
}
}
func validateLoadedRunManifest(m *RunManifest) error {
if m == nil {
return fmt.Errorf("manifest is nil")
}
if strings.TrimSpace(m.SessionID) == "" {
return fmt.Errorf("session_id is required")
}
if strings.TrimSpace(m.RunID) == "" {
return fmt.Errorf("run_id is required")
}
if m.CreatedAt.IsZero() {
return fmt.Errorf("created_at is required")
}
if m.UpdatedAt.IsZero() {
return fmt.Errorf("updated_at is required")
}
return nil
}
func normalizeRunManifest(m *RunManifest) {
if m.Stages == nil {
m.Stages = map[string]*RunStageRecord{}
}
for name, stage := range m.Stages {
if stage == nil {
stage = &RunStageRecord{
Name: name,
Action: RunStageActionRun,
Status: StatusPending,
CreatedAt: m.CreatedAt,
UpdatedAt: m.UpdatedAt,
}
m.Stages[name] = stage
}
if stage.Name == "" {
stage.Name = name
}
}
}
func writeJSONAtomically(ctx context.Context, path, tempPattern string, data []byte) error {
dir := filepath.Dir(path)
if err := os.MkdirAll(dir, 0o755); err != nil {
return fmt.Errorf("create directory %q: %w", dir, err)
}
tmp, err := os.CreateTemp(dir, tempPattern)
if err != nil {
return fmt.Errorf("create temp file: %w", err)
}
tmpName := tmp.Name()
removeTmp := true
defer func() {
if removeTmp {
_ = os.Remove(tmpName)
}
}()
if _, err := tmp.Write(data); err != nil {
_ = tmp.Close()
return fmt.Errorf("write temp file: %w", err)
}
if err := tmp.Sync(); err != nil {
_ = tmp.Close()
return fmt.Errorf("sync temp file: %w", err)
}
if err := tmp.Close(); err != nil {
return fmt.Errorf("close temp file: %w", err)
}
if err := checkContext(ctx); err != nil {
return err
}
if err := os.Rename(tmpName, path); err != nil {
return fmt.Errorf("rename temp file: %w", err)
}
removeTmp = false
return nil
}
func checkContext(ctx context.Context) error {
if ctx == nil {
return nil

View File

@@ -150,3 +150,73 @@ func TestLoadRejectsInvalidManifest(t *testing.T) {
t.Fatalf("error = %q, want session_id validation", err.Error())
}
}
func TestLocalStoreCreateSaveLoadRunManifestRoundTrip(t *testing.T) {
store := &LocalStore{}
ctx := context.Background()
run, err := store.CreateRun(
ctx,
"2026-05-03",
"forsaken",
"20260517T000000Z-abcdef12",
true,
[]string{"prepare", "transcribe"},
)
if err != nil {
t.Fatalf("CreateRun() error = %v", err)
}
run.SessionManifestPath = "/var/lib/narratio/work/forsaken/2026-05-03/manifest.json"
run.MarkStageRunning("prepare", time.Date(2026, 5, 3, 12, 1, 0, 0, time.UTC))
run.MarkStageSucceeded("prepare", time.Date(2026, 5, 3, 12, 2, 0, 0, time.UTC), []ArtifactRecord{
{Kind: "input", LocalPath: "inputs/session.yml"},
})
run.MarkSucceeded(time.Date(2026, 5, 3, 12, 3, 0, 0, time.UTC))
path := filepath.Join(t.TempDir(), "run-manifest.json")
if err := store.SaveRun(ctx, path, run); err != nil {
t.Fatalf("SaveRun() error = %v", err)
}
loaded, err := store.LoadRun(ctx, path)
if err != nil {
t.Fatalf("LoadRun() error = %v", err)
}
if loaded.SessionID != "2026-05-03" {
t.Fatalf("SessionID = %q, want %q", loaded.SessionID, "2026-05-03")
}
if loaded.Campaign != "forsaken" {
t.Fatalf("Campaign = %q, want %q", loaded.Campaign, "forsaken")
}
if loaded.RunID != "20260517T000000Z-abcdef12" {
t.Fatalf("RunID = %q, want %q", loaded.RunID, "20260517T000000Z-abcdef12")
}
if loaded.Status != RunManifestStatusSucceeded {
t.Fatalf("Status = %q, want %q", loaded.Status, RunManifestStatusSucceeded)
}
if loaded.Stages["prepare"] == nil || loaded.Stages["prepare"].Status != StatusSucceeded {
t.Fatalf("prepare stage = %#v, want succeeded", loaded.Stages["prepare"])
}
if loaded.Stages["prepare"].Action != RunStageActionRun {
t.Fatalf("prepare action = %q, want %q", loaded.Stages["prepare"].Action, RunStageActionRun)
}
}
func TestLoadRunRejectsInvalidManifest(t *testing.T) {
store := &LocalStore{}
ctx := context.Background()
path := filepath.Join(t.TempDir(), "run-manifest.json")
if err := os.WriteFile(path, []byte(`{"session_id":"2026-05-03"}`), 0o644); err != nil {
t.Fatalf("WriteFile() error = %v", err)
}
_, err := store.LoadRun(ctx, path)
if err == nil {
t.Fatal("expected error, got nil")
}
if !strings.Contains(err.Error(), "run_id is required") {
t.Fatalf("error = %q, want run_id validation", err.Error())
}
}

View File

@@ -3,6 +3,7 @@ package stage
import (
"context"
"encoding/json"
"errors"
"fmt"
"os"
"path/filepath"
@@ -58,7 +59,11 @@ func (analyzeStage) Run(ctx context.Context, env *Env, m *manifest.Manifest) (*S
return nil, fmt.Errorf("analyze: session id is required")
}
paths := env.ArtifactStore.SessionPaths(sessionID)
paths := sessionPathsForEnv(env, sessionID)
runLayout, err := resolveRunStageLayout(env, m, paths, sessionID, "analyze")
if err != nil {
return nil, fmt.Errorf("analyze: resolve run-stage layout: %w", err)
}
if env.Config.Pipeline.Scriptorium == nil {
return &StageResult{
Metadata: map[string]any{
@@ -83,27 +88,7 @@ func (analyzeStage) Run(ctx context.Context, env *Env, m *manifest.Manifest) (*S
}, nil
}
processedTranscriptPath, processedSource, err := discoverProcessedTranscript(m, paths)
if err != nil {
return nil, fmt.Errorf("analyze: resolve processed transcript: %w", err)
}
normalizedTranscriptPath, normalizedSource, err := discoverNormalizedTranscript(m, paths)
if err != nil {
return nil, fmt.Errorf("analyze: resolve normalized transcript: %w", err)
}
trimmedTranscriptPath, trimmedSource, err := discoverTrimmedTranscript(m, paths)
if err != nil {
return nil, fmt.Errorf("analyze: resolve trimmed transcript: %w", err)
}
transcriptInputs := analyzeTranscriptInputs{
ProcessedPath: processedTranscriptPath,
ProcessedSource: processedSource,
NormalizedPath: normalizedTranscriptPath,
NormalizedSource: normalizedSource,
TrimmedPath: trimmedTranscriptPath,
TrimmedSource: trimmedSource,
}
transcriptRefs := discoverAnalyzeTranscriptRefs(m, paths)
inputPaths := map[string]string{}
omittedOptionalInputs := []string{}
@@ -111,7 +96,7 @@ func (analyzeStage) Run(ctx context.Context, env *Env, m *manifest.Manifest) (*S
inputNames := sortedScriptoriumInputNames(artifactCfg.Inputs)
for _, inputName := range inputNames {
inputCfg := artifactCfg.Inputs[inputName]
resolvedPath, resolved, resolveErr := resolveScriptoriumInput(inputName, inputCfg, transcriptInputs, paths, sessionDir)
resolvedPath, resolved, resolveErr := resolveScriptoriumInput(inputName, inputCfg, m, paths, sessionDir)
if resolveErr != nil {
return nil, fmt.Errorf("analyze: resolve input %q: %w", inputName, resolveErr)
}
@@ -130,13 +115,22 @@ func (analyzeStage) Run(ctx context.Context, env *Env, m *manifest.Manifest) (*S
return nil, fmt.Errorf("analyze: resolve vars: %w", err)
}
outputPath, err := resolveScriptoriumOutputPath(paths, artifactCfg.OutputPath)
canonicalOutputPath, err := resolveScriptoriumOutputPath(paths, artifactCfg.OutputPath)
if err != nil {
return nil, fmt.Errorf("analyze: resolve output path: %w", err)
}
outputPath, err := runLocalPathForCanonical(runLayout, paths, canonicalOutputPath)
if err != nil {
return nil, fmt.Errorf("analyze: resolve run-local output path: %w", err)
}
stdoutLogPath := filepath.Join(paths.LogsDir, "scriptorium."+artifactName+".stdout.log")
stderrLogPath := filepath.Join(paths.LogsDir, "scriptorium."+artifactName+".stderr.log")
generatedConfigPath := filepath.Join(paths.ConfigDir, "scriptorium."+artifactName+".generated.yml")
if runLayout.Enabled {
stdoutLogPath = filepath.Join(runLayout.LogsDir, "scriptorium."+artifactName+".stdout.log")
stderrLogPath = filepath.Join(runLayout.LogsDir, "scriptorium."+artifactName+".stderr.log")
generatedConfigPath = filepath.Join(runLayout.ConfigDir, "scriptorium."+artifactName+".generated.yml")
}
timeout, err := resolveScriptoriumTimeout(env.Config.Pipeline.Scriptorium.Timeout, artifactCfg.Timeout)
if err != nil {
@@ -157,20 +151,28 @@ func (analyzeStage) Run(ctx context.Context, env *Env, m *manifest.Manifest) (*S
"omitted_optional_inputs": omittedOptionalInputs,
"vars": vars,
"timeout": timeout.String(),
"processed_transcript_path": processedTranscriptPath,
"processed_transcript_source": processedSource,
"normalized_transcript_path": normalizedTranscriptPath,
"normalized_transcript_source": normalizedSource,
"trimmed_transcript_path": trimmedTranscriptPath,
"trimmed_transcript_source": trimmedSource,
"processed_transcript_path": transcriptRefs.ProcessedPath,
"processed_transcript_source": transcriptRefs.ProcessedSource,
"normalized_transcript_path": transcriptRefs.NormalizedPath,
"normalized_transcript_source": transcriptRefs.NormalizedSource,
"trimmed_transcript_path": transcriptRefs.TrimmedPath,
"trimmed_transcript_source": transcriptRefs.TrimmedSource,
"render_debug_enabled": resolveRenderDebugEnabled(env.Config.Pipeline.Scriptorium.RenderDebug, artifactCfg.RenderDebug),
}
if meta["render_debug_enabled"] == true {
renderOutputPath := filepath.Join(paths.ArtifactsDir, artifactName+".render.json")
if runLayout.Enabled {
renderOutputPath = filepath.Join(runLayout.ReportsDir, artifactName+".render.json")
}
renderStdoutPath := filepath.Join(paths.LogsDir, "scriptorium."+artifactName+".render.stdout.log")
renderStderrPath := filepath.Join(paths.LogsDir, "scriptorium."+artifactName+".render.stderr.log")
renderGeneratedConfigPath := filepath.Join(paths.ConfigDir, "scriptorium."+artifactName+".render.generated.yml")
if runLayout.Enabled {
renderStdoutPath = filepath.Join(runLayout.LogsDir, "scriptorium."+artifactName+".render.stdout.log")
renderStderrPath = filepath.Join(runLayout.LogsDir, "scriptorium."+artifactName+".render.stderr.log")
renderGeneratedConfigPath = filepath.Join(runLayout.ConfigDir, "scriptorium."+artifactName+".render.generated.yml")
}
renderReq := scriptorium.RenderArtifactRequest{
Binary: env.Config.Pipeline.Scriptorium.Binary,
@@ -257,17 +259,19 @@ func (analyzeStage) Run(ctx context.Context, env *Env, m *manifest.Manifest) (*S
if err := requireNonEmptyFile(finalOutputPath, artifactName+" output"); err != nil {
return nil, fmt.Errorf("analyze: %w", err)
}
artifactRef := artifacts.Ref{
Kind: artifactName,
Category: "artifacts",
SessionID: sessionID,
AbsolutePath: finalOutputPath,
promotedArtifact, err := promoteRunLocalOutput(env.ArtifactStore, finalOutputPath, canonicalOutputPath, artifacts.Ref{
Kind: artifactName,
Category: "artifacts",
SessionID: sessionID,
})
if err != nil {
return nil, fmt.Errorf("analyze: promote artifact output: %w", err)
}
logPaths = append(logPaths, stdoutLogPath, stderrLogPath)
generatedConfigs = append(generatedConfigs, generatedConfigPath)
meta["output_path"] = finalOutputPath
meta["run_output_path"] = finalOutputPath
meta["output_path"] = canonicalOutputPath
meta["generated_config_path"] = generatedConfigPath
meta["stdout_log_path"] = stdoutLogPath
meta["stderr_log_path"] = stderrLogPath
@@ -286,7 +290,7 @@ func (analyzeStage) Run(ctx context.Context, env *Env, m *manifest.Manifest) (*S
}
return &StageResult{
Outputs: []artifacts.Ref{artifactRef},
Outputs: []artifacts.Ref{promotedArtifact},
Logs: logPaths,
GeneratedConfigs: generatedConfigs,
Metadata: meta,
@@ -350,40 +354,6 @@ func discoverProcessedTranscript(m *manifest.Manifest, paths artifacts.SessionPa
return "", "", nil
}
func discoverTrimmedTranscript(m *manifest.Manifest, paths artifacts.SessionPaths) (string, string, error) {
candidates := []string{}
if m != nil && m.Stages != nil {
if sr := m.Stages["trim"]; sr != nil {
for _, out := range sr.Outputs {
if out.Kind != "transcript_trimmed" {
continue
}
p := strings.TrimSpace(out.LocalPath)
if p == "" {
continue
}
resolved := artifacts.ResolveSessionLocalPathForRead(paths, p)
candidates = append(candidates, filepath.Clean(resolved))
}
}
}
deduped := dedupeAndSortPaths(candidates)
for _, p := range deduped {
if info, err := os.Stat(p); err == nil && !info.IsDir() {
return p, "manifest.trim.outputs", nil
}
}
fallback := filepath.Join(paths.TranscriptsDir, "trimmed.json")
if info, err := os.Stat(fallback); err == nil && !info.IsDir() {
return filepath.Clean(fallback), "fallback.transcripts_dir", nil
}
if len(deduped) > 0 {
return deduped[0], "manifest.trim.outputs", nil
}
return "", "", nil
}
type analyzeTranscriptInputs struct {
ProcessedPath string
ProcessedSource string
@@ -393,38 +363,36 @@ type analyzeTranscriptInputs struct {
TrimmedSource string
}
func discoverAnalyzeTranscriptRefs(m *manifest.Manifest, paths artifacts.SessionPaths) analyzeTranscriptInputs {
processedPath, processedSource := discoverAnalyzeArtifactRef(m, paths, artifacts.ArtifactTranscriptPolished)
normalizedPath, normalizedSource := discoverAnalyzeArtifactRef(m, paths, artifacts.ArtifactTranscriptFull)
trimmedPath, trimmedSource := discoverAnalyzeArtifactRef(m, paths, artifacts.ArtifactTranscriptTrimmed)
return analyzeTranscriptInputs{
ProcessedPath: processedPath,
ProcessedSource: processedSource,
NormalizedPath: normalizedPath,
NormalizedSource: normalizedSource,
TrimmedPath: trimmedPath,
TrimmedSource: trimmedSource,
}
}
func discoverAnalyzeArtifactRef(m *manifest.Manifest, paths artifacts.SessionPaths, source string) (string, string) {
resolved, err := artifacts.ResolveSessionArtifact(paths, m, source)
if err != nil {
return "", ""
}
return resolved.Path, resolved.Provenance
}
func resolveScriptoriumInput(
inputName string,
inputCfg config.ScriptoriumInputConfig,
transcriptInputs analyzeTranscriptInputs,
m *manifest.Manifest,
paths artifacts.SessionPaths,
sessionDir string,
) (string, bool, error) {
switch strings.TrimSpace(inputCfg.Source) {
case "processed_transcript":
if strings.TrimSpace(transcriptInputs.ProcessedPath) == "" {
return "", false, nil
}
if err := validateProcessedTranscriptOutput(transcriptInputs.ProcessedPath); err != nil {
return "", false, fmt.Errorf("processed transcript %q invalid: %w", transcriptInputs.ProcessedPath, err)
}
return transcriptInputs.ProcessedPath, true, nil
case "normalized_transcript":
if strings.TrimSpace(transcriptInputs.NormalizedPath) == "" {
return "", false, fmt.Errorf("normalized transcript input is unavailable; run normalize stage first")
}
if err := validateProcessedTranscriptOutput(transcriptInputs.NormalizedPath); err != nil {
return "", false, fmt.Errorf("normalized transcript %q invalid: %w", transcriptInputs.NormalizedPath, err)
}
return transcriptInputs.NormalizedPath, true, nil
case "trimmed_transcript":
if strings.TrimSpace(transcriptInputs.TrimmedPath) == "" {
return "", false, fmt.Errorf("trimmed transcript input is unavailable; run trim stage first")
}
if err := validateProcessedTranscriptOutput(transcriptInputs.TrimmedPath); err != nil {
return "", false, fmt.Errorf("trimmed transcript %q invalid: %w", transcriptInputs.TrimmedPath, err)
}
return transcriptInputs.TrimmedPath, true, nil
case "previous_session_artifact":
if strings.TrimSpace(inputCfg.Path) == "" {
return "", false, nil
@@ -435,7 +403,27 @@ func resolveScriptoriumInput(
}
return resolved, true, nil
default:
return "", false, fmt.Errorf("unsupported source %q", inputCfg.Source)
resolved, err := artifacts.ResolveSessionArtifact(paths, m, inputCfg.Source)
if err == nil {
return resolved.Path, true, nil
}
if errors.Is(err, artifacts.ErrSessionArtifactNotFound) {
normalized, normalizeErr := artifacts.NormalizeSessionArtifactSource(inputCfg.Source)
if normalizeErr != nil {
return "", false, normalizeErr
}
switch normalized {
case artifacts.ArtifactTranscriptPolished:
return "", false, nil
case artifacts.ArtifactTranscriptFull:
return "", false, fmt.Errorf("normalized transcript input is unavailable; run normalize stage first")
case artifacts.ArtifactTranscriptTrimmed:
return "", false, fmt.Errorf("trimmed transcript input is unavailable; run trim stage first")
default:
return "", false, nil
}
}
return "", false, err
}
}

View File

@@ -17,7 +17,7 @@ import (
func TestAnalyzeGeneratesSessionRecapFromTrimmedTranscript(t *testing.T) {
env, m, fake := setupAnalyzeEnv(t)
paths := env.ArtifactStore.SessionPaths(m.SessionID)
paths := sessionPathsForEnv(env, m.SessionID)
writeAnalyzeFile(t, filepath.Join(paths.TranscriptsDir, "trimmed.json"), `{"segments":[]}`)
result, err := (analyzeStage{}).Run(context.Background(), env, m)
@@ -70,7 +70,7 @@ func TestAnalyzeGeneratesSessionRecapFromTrimmedTranscript(t *testing.T) {
func TestAnalyzeRenderDebugFalseDoesNotCallRenderArtifact(t *testing.T) {
env, m, fake := setupAnalyzeEnv(t)
paths := env.ArtifactStore.SessionPaths(m.SessionID)
paths := sessionPathsForEnv(env, m.SessionID)
writeAnalyzeFile(t, filepath.Join(paths.TranscriptsDir, "trimmed.json"), `{"segments":[]}`)
env.Config.Pipeline.Scriptorium.RenderDebug = false
@@ -86,7 +86,7 @@ func TestAnalyzeRenderDebugFalseDoesNotCallRenderArtifact(t *testing.T) {
func TestAnalyzeRenderDebugArtifactOverrideFalseWinsOverGlobalTrue(t *testing.T) {
env, m, fake := setupAnalyzeEnv(t)
paths := env.ArtifactStore.SessionPaths(m.SessionID)
paths := sessionPathsForEnv(env, m.SessionID)
writeAnalyzeFile(t, filepath.Join(paths.TranscriptsDir, "trimmed.json"), `{"segments":[]}`)
env.Config.Pipeline.Scriptorium.RenderDebug = true
@@ -109,7 +109,7 @@ func TestAnalyzeRenderDebugArtifactOverrideFalseWinsOverGlobalTrue(t *testing.T)
func TestAnalyzeRenderDebugTrueCallsRenderBeforeRun(t *testing.T) {
env, m, _ := setupAnalyzeEnv(t)
paths := env.ArtifactStore.SessionPaths(m.SessionID)
paths := sessionPathsForEnv(env, m.SessionID)
writeAnalyzeFile(t, filepath.Join(paths.TranscriptsDir, "trimmed.json"), `{"segments":[]}`)
env.Config.Pipeline.Scriptorium.RenderDebug = true
@@ -130,7 +130,7 @@ func TestAnalyzeRenderDebugTrueCallsRenderBeforeRun(t *testing.T) {
func TestAnalyzeRenderOutputPathIsRecorded(t *testing.T) {
env, m, fake := setupAnalyzeEnv(t)
paths := env.ArtifactStore.SessionPaths(m.SessionID)
paths := sessionPathsForEnv(env, m.SessionID)
writeAnalyzeFile(t, filepath.Join(paths.TranscriptsDir, "trimmed.json"), `{"segments":[]}`)
env.Config.Pipeline.Scriptorium.RenderDebug = true
@@ -154,7 +154,7 @@ func TestAnalyzeRenderOutputPathIsRecorded(t *testing.T) {
func TestAnalyzeRenderFailurePreventsRun(t *testing.T) {
env, m, fake := setupAnalyzeEnv(t)
paths := env.ArtifactStore.SessionPaths(m.SessionID)
paths := sessionPathsForEnv(env, m.SessionID)
writeAnalyzeFile(t, filepath.Join(paths.TranscriptsDir, "trimmed.json"), `{"segments":[]}`)
env.Config.Pipeline.Scriptorium.RenderDebug = true
fake.RenderErr = errors.New("render boom")
@@ -176,7 +176,7 @@ func TestAnalyzeRenderFailurePreventsRun(t *testing.T) {
func TestAnalyzeRenderInvalidJSONFailsClearly(t *testing.T) {
env, m, _ := setupAnalyzeEnv(t)
paths := env.ArtifactStore.SessionPaths(m.SessionID)
paths := sessionPathsForEnv(env, m.SessionID)
writeAnalyzeFile(t, filepath.Join(paths.TranscriptsDir, "trimmed.json"), `{"segments":[]}`)
env.Config.Pipeline.Scriptorium.RenderDebug = true
runner := &orderedScriptoriumRunner{
@@ -199,7 +199,7 @@ func TestAnalyzeRenderInvalidJSONFailsClearly(t *testing.T) {
func TestAnalyzeRunStillSucceedsWhenRenderSucceeds(t *testing.T) {
env, m, fake := setupAnalyzeEnv(t)
paths := env.ArtifactStore.SessionPaths(m.SessionID)
paths := sessionPathsForEnv(env, m.SessionID)
writeAnalyzeFile(t, filepath.Join(paths.TranscriptsDir, "trimmed.json"), `{"segments":[]}`)
env.Config.Pipeline.Scriptorium.RenderDebug = true
@@ -220,7 +220,7 @@ func TestAnalyzeRunStillSucceedsWhenRenderSucceeds(t *testing.T) {
func TestAnalyzeOmitsOptionalPreviousRecapWhenUnavailable(t *testing.T) {
env, m, fake := setupAnalyzeEnv(t)
paths := env.ArtifactStore.SessionPaths(m.SessionID)
paths := sessionPathsForEnv(env, m.SessionID)
writeAnalyzeFile(t, filepath.Join(paths.TranscriptsDir, "processed.json"), `{"segments":[]}`)
env.Config.Pipeline.Scriptorium.Artifacts["session_recap"] = config.ScriptoriumArtifactConfig{
@@ -256,6 +256,32 @@ func TestAnalyzeOmitsOptionalPreviousRecapWhenUnavailable(t *testing.T) {
}
}
func TestAnalyzeUsesRunLocalPathsAndPromotesCanonical(t *testing.T) {
env, m, fake := setupAnalyzeEnv(t)
env.Config.Session.Campaign = "sample-campaign"
m.Campaign = "sample-campaign"
m.RunID = "20260518T010203Z-abcdef12"
paths := sessionPathsForEnv(env, m.SessionID)
writeAnalyzeFile(t, filepath.Join(paths.TranscriptsDir, "trimmed.json"), `{"segments":[]}`)
result, err := (analyzeStage{}).Run(context.Background(), env, m)
if err != nil {
t.Fatalf("Run() error = %v", err)
}
if len(fake.RunRequests) != 1 {
t.Fatalf("run requests = %d, want 1", len(fake.RunRequests))
}
if !strings.Contains(fake.RunRequests[0].OutputPath, filepath.Join("runs", m.RunID, "analyze", "outputs")) {
t.Fatalf("run output path = %q, want run-local path", fake.RunRequests[0].OutputPath)
}
if len(result.Outputs) != 1 {
t.Fatalf("outputs len = %d, want 1", len(result.Outputs))
}
if strings.Contains(result.Outputs[0].AbsolutePath, string(filepath.Separator)+"runs"+string(filepath.Separator)) {
t.Fatalf("promoted output path = %q, want canonical session path", result.Outputs[0].AbsolutePath)
}
}
type orderedScriptoriumRunner struct {
Calls []string
RenderErr error
@@ -318,7 +344,7 @@ func (r *orderedScriptoriumRunner) RunArtifact(_ context.Context, req scriptoriu
func TestAnalyzeIncludesPreviousRecapWhenConfiguredAndAvailable(t *testing.T) {
env, m, fake := setupAnalyzeEnv(t)
paths := env.ArtifactStore.SessionPaths(m.SessionID)
paths := sessionPathsForEnv(env, m.SessionID)
writeAnalyzeFile(t, filepath.Join(paths.TranscriptsDir, "processed.json"), `{"segments":[]}`)
previousRecapPath := filepath.Join(filepath.Dir(env.Config.SessionPath), "previous", "session_recap.md")
@@ -359,7 +385,7 @@ func TestAnalyzeIncludesPreviousRecapWhenConfiguredAndAvailable(t *testing.T) {
func TestAnalyzeFailsWhenRequiredPreviousRecapMissing(t *testing.T) {
env, m, _ := setupAnalyzeEnv(t)
paths := env.ArtifactStore.SessionPaths(m.SessionID)
paths := sessionPathsForEnv(env, m.SessionID)
writeAnalyzeFile(t, filepath.Join(paths.TranscriptsDir, "processed.json"), `{"segments":[]}`)
env.Config.Pipeline.Scriptorium.Artifacts["session_recap"] = config.ScriptoriumArtifactConfig{
@@ -390,7 +416,7 @@ func TestAnalyzeFailsWhenRequiredPreviousRecapMissing(t *testing.T) {
func TestAnalyzeFailsWhenOutputPathMissing(t *testing.T) {
env, m, fake := setupAnalyzeEnv(t)
paths := env.ArtifactStore.SessionPaths(m.SessionID)
paths := sessionPathsForEnv(env, m.SessionID)
writeAnalyzeFile(t, filepath.Join(paths.TranscriptsDir, "trimmed.json"), `{"segments":[]}`)
artifact := env.Config.Pipeline.Scriptorium.Artifacts["session_recap"]
@@ -425,7 +451,7 @@ func TestAnalyzeFailsWhenTrimmedTranscriptMissing(t *testing.T) {
func TestAnalyzeSupportsProcessedTranscriptSourceWhenConfigured(t *testing.T) {
env, m, fake := setupAnalyzeEnv(t)
paths := env.ArtifactStore.SessionPaths(m.SessionID)
paths := sessionPathsForEnv(env, m.SessionID)
writeAnalyzeFile(t, filepath.Join(paths.TranscriptsDir, "processed.json"), `{"segments":[]}`)
artifact := env.Config.Pipeline.Scriptorium.Artifacts["session_recap"]
@@ -447,9 +473,33 @@ func TestAnalyzeSupportsProcessedTranscriptSourceWhenConfigured(t *testing.T) {
}
}
func TestAnalyzeSupportsCanonicalTrimmedTranscriptSourceWhenConfigured(t *testing.T) {
env, m, fake := setupAnalyzeEnv(t)
paths := sessionPathsForEnv(env, m.SessionID)
writeAnalyzeFile(t, filepath.Join(paths.TranscriptsDir, "trimmed.json"), `{"segments":[]}`)
artifact := env.Config.Pipeline.Scriptorium.Artifacts["session_recap"]
artifact.Inputs["transcript"] = config.ScriptoriumInputConfig{
Source: "narratio.transcript.trimmed",
Required: true,
}
env.Config.Pipeline.Scriptorium.Artifacts["session_recap"] = artifact
_, err := (analyzeStage{}).Run(context.Background(), env, m)
if err != nil {
t.Fatalf("Run() error = %v", err)
}
if len(fake.RunRequests) != 1 {
t.Fatalf("run requests = %d, want 1", len(fake.RunRequests))
}
if fake.RunRequests[0].InputPaths["transcript"] != filepath.Join(paths.TranscriptsDir, "trimmed.json") {
t.Fatalf("transcript input = %q, want trimmed transcript path", fake.RunRequests[0].InputPaths["transcript"])
}
}
func TestAnalyzeSupportsNormalizedTranscriptSourceWhenConfigured(t *testing.T) {
env, m, fake := setupAnalyzeEnv(t)
paths := env.ArtifactStore.SessionPaths(m.SessionID)
paths := sessionPathsForEnv(env, m.SessionID)
normalizedPath := filepath.Join(paths.TranscriptsDir, "normalized.json")
writeAnalyzeFile(t, normalizedPath, `{"segments":[{"id":1}]}`)
@@ -472,9 +522,39 @@ func TestAnalyzeSupportsNormalizedTranscriptSourceWhenConfigured(t *testing.T) {
}
}
func TestAnalyzeSupportsCanonicalNormalizedTranscriptSourceFromManifestOutput(t *testing.T) {
env, m, fake := setupAnalyzeEnv(t)
paths := sessionPathsForEnv(env, m.SessionID)
fallbackPath := filepath.Join(paths.TranscriptsDir, "normalized.json")
manifestPath := filepath.Join(paths.ArtifactsDir, "normalized.from-manifest.json")
writeAnalyzeFile(t, fallbackPath, `{"segments":[{"id":999}]}`)
writeAnalyzeFile(t, manifestPath, `{"segments":[{"id":10}]}`)
m.MarkStageSucceeded("normalize", time.Now().UTC(), []manifest.ArtifactRecord{
{Kind: "transcript_normalized", LocalPath: manifestPath},
})
artifact := env.Config.Pipeline.Scriptorium.Artifacts["session_recap"]
artifact.Inputs["transcript"] = config.ScriptoriumInputConfig{
Source: "narratio.transcript.full",
Required: true,
}
env.Config.Pipeline.Scriptorium.Artifacts["session_recap"] = artifact
_, err := (analyzeStage{}).Run(context.Background(), env, m)
if err != nil {
t.Fatalf("Run() error = %v", err)
}
if len(fake.RunRequests) != 1 {
t.Fatalf("run requests = %d, want 1", len(fake.RunRequests))
}
if fake.RunRequests[0].InputPaths["transcript"] != manifestPath {
t.Fatalf("transcript input = %q, want manifest normalized transcript path", fake.RunRequests[0].InputPaths["transcript"])
}
}
func TestAnalyzeSupportsNormalizedTranscriptSourceFromManifestOutput(t *testing.T) {
env, m, fake := setupAnalyzeEnv(t)
paths := env.ArtifactStore.SessionPaths(m.SessionID)
paths := sessionPathsForEnv(env, m.SessionID)
fallbackPath := filepath.Join(paths.TranscriptsDir, "normalized.json")
manifestPath := filepath.Join(paths.ArtifactsDir, "normalized.from-manifest.json")
writeAnalyzeFile(t, fallbackPath, `{"segments":[{"id":999}]}`)
@@ -528,7 +608,7 @@ func TestAnalyzeFailsWhenNormalizedTranscriptMissing(t *testing.T) {
func TestAnalyzeFailsWhenProcessedTranscriptInvalidJSON(t *testing.T) {
env, m, _ := setupAnalyzeEnv(t)
paths := env.ArtifactStore.SessionPaths(m.SessionID)
paths := sessionPathsForEnv(env, m.SessionID)
writeAnalyzeFile(t, filepath.Join(paths.TranscriptsDir, "trimmed.json"), `{not-json`)
_, err := (analyzeStage{}).Run(context.Background(), env, m)
@@ -542,7 +622,7 @@ func TestAnalyzeFailsWhenProcessedTranscriptInvalidJSON(t *testing.T) {
func TestAnalyzeFailsWhenProcessedTranscriptMissingSegmentsArray(t *testing.T) {
env, m, _ := setupAnalyzeEnv(t)
paths := env.ArtifactStore.SessionPaths(m.SessionID)
paths := sessionPathsForEnv(env, m.SessionID)
writeAnalyzeFile(t, filepath.Join(paths.TranscriptsDir, "trimmed.json"), `{"not_segments":[]}`)
_, err := (analyzeStage{}).Run(context.Background(), env, m)
@@ -556,7 +636,7 @@ func TestAnalyzeFailsWhenProcessedTranscriptMissingSegmentsArray(t *testing.T) {
func TestAnalyzeRecordsRefsAndMetadata(t *testing.T) {
env, m, _ := setupAnalyzeEnv(t)
paths := env.ArtifactStore.SessionPaths(m.SessionID)
paths := sessionPathsForEnv(env, m.SessionID)
writeAnalyzeFile(t, filepath.Join(paths.TranscriptsDir, "trimmed.json"), `{"segments":[]}`)
result, err := (analyzeStage{}).Run(context.Background(), env, m)
@@ -585,7 +665,7 @@ func TestAnalyzeRecordsRefsAndMetadata(t *testing.T) {
func TestAnalyzeHandlesAdapterError(t *testing.T) {
env, m, fake := setupAnalyzeEnv(t)
paths := env.ArtifactStore.SessionPaths(m.SessionID)
paths := sessionPathsForEnv(env, m.SessionID)
writeAnalyzeFile(t, filepath.Join(paths.TranscriptsDir, "trimmed.json"), `{"segments":[]}`)
fake.RunErr = errors.New("adapter boom")
@@ -600,7 +680,7 @@ func TestAnalyzeHandlesAdapterError(t *testing.T) {
func TestAnalyzeHandlesValidationFailedResultAsError(t *testing.T) {
env, m, fake := setupAnalyzeEnv(t)
paths := env.ArtifactStore.SessionPaths(m.SessionID)
paths := sessionPathsForEnv(env, m.SessionID)
writeAnalyzeFile(t, filepath.Join(paths.TranscriptsDir, "trimmed.json"), `{"segments":[]}`)
fake.RunResult = scriptorium.ArtifactResult{
ValidationFailed: true,
@@ -619,7 +699,7 @@ func TestAnalyzeHandlesValidationFailedResultAsError(t *testing.T) {
func TestAnalyzeSkipsWhenNoEnabledScriptoriumArtifactsConfigured(t *testing.T) {
env, m, _ := setupAnalyzeEnv(t)
paths := env.ArtifactStore.SessionPaths(m.SessionID)
paths := sessionPathsForEnv(env, m.SessionID)
writeAnalyzeFile(t, filepath.Join(paths.TranscriptsDir, "trimmed.json"), `{"segments":[]}`)
env.Config.Pipeline.Scriptorium.Artifacts["session_recap"] = config.ScriptoriumArtifactConfig{
@@ -692,7 +772,7 @@ func setupAnalyzeEnv(t *testing.T) (*Env, *manifest.Manifest, *scriptorium.FakeR
}
store := artifacts.NewLocalStore(workspace)
if _, err := store.EnsureLayout("2026-05-03"); err != nil {
if _, err := store.EnsureLayoutFor("sample-campaign", "2026-05-03"); err != nil {
t.Fatalf("EnsureLayout() error = %v", err)
}

View File

@@ -18,6 +18,10 @@ import (
)
type archiveStage struct{}
type archiveUploadFile struct {
RelativePath string
LocalPath string
}
var archivePrerequisiteStages = []string{
"prepare",
@@ -29,15 +33,6 @@ var archivePrerequisiteStages = []string{
"analyze",
}
var archiveRunUploadDirs = []string{
"inputs",
"transcripts",
"artifacts",
"reports",
"config",
"logs",
}
func (archiveStage) Name() string { return "archive" }
func (archiveStage) Declares() IODecl {
@@ -83,16 +78,16 @@ func (archiveStage) Run(ctx context.Context, env *Env, m *manifest.Manifest) (*S
return nil, fmt.Errorf("archive: remote object store backend is required when archive run upload is enabled")
}
workDir, err := archiveWorkDir(env, m)
runRoot, err := resolveArchiveRunRoot(env, m)
if err != nil {
return nil, fmt.Errorf("archive: resolve local workdir: %w", err)
return nil, fmt.Errorf("archive: resolve run root: %w", err)
}
workDirInfo, err := os.Stat(workDir)
runRootInfo, err := os.Stat(runRoot)
if err != nil {
return nil, fmt.Errorf("archive: local workdir %q: %w", workDir, err)
return nil, fmt.Errorf("archive: run root %q: %w", runRoot, err)
}
if !workDirInfo.IsDir() {
return nil, fmt.Errorf("archive: local workdir %q is not a directory", workDir)
if !runRootInfo.IsDir() {
return nil, fmt.Errorf("archive: run root %q is not a directory", runRoot)
}
runPrefix, err := archiveRunPrefix(env, m)
@@ -112,29 +107,30 @@ func (archiveStage) Run(ctx context.Context, env *Env, m *manifest.Manifest) (*S
return nil, fmt.Errorf("archive: run id is required")
}
runFiles, err := collectArchiveRunFiles(workDir)
manifestSource, err := resolveArchiveRunManifestSource(runRoot)
if err != nil {
return nil, fmt.Errorf("archive: resolve run manifest source: %w", err)
}
runFiles, err := collectArchiveRunFiles(runRoot, manifestSource)
if err != nil {
return nil, fmt.Errorf("archive: collect run files: %w", err)
}
promotions, err := resolveArchivePromotions(workDir, env.Config.Pipeline.Archive.PromoteArtifacts)
sessionRoot, err := resolveArchiveSessionRoot(env, m)
if err != nil {
return nil, fmt.Errorf("archive: resolve session root for promotions: %w", err)
}
promotions, err := resolveArchivePromotions(sessionRoot, env.Config.Pipeline.Archive.PromoteArtifacts)
if err != nil {
return nil, fmt.Errorf("archive: resolve promotion rules: %w", err)
}
currentManifestSource := filepath.Join(workDir, "manifest.json")
if info, err := os.Stat(currentManifestSource); err != nil {
return nil, fmt.Errorf("archive: current manifest source %q: %w", currentManifestSource, err)
} else if info.IsDir() {
return nil, fmt.Errorf("archive: current manifest source %q is a directory", currentManifestSource)
}
runUploaded := make([]string, 0, len(runFiles))
for _, rel := range runFiles {
localPath := filepath.Join(workDir, filepath.FromSlash(rel))
key := artifacts.S3RunRelativeDestinationKey(runPrefix, rel)
if _, err := env.ObjectStore.Upload(ctx, localPath, key, storage.UploadOptions{}); err != nil {
return nil, fmt.Errorf("archive: upload run file %q to %q: %w", rel, key, err)
for _, file := range runFiles {
key := artifacts.S3RunRelativeDestinationKey(runPrefix, file.RelativePath)
if _, err := env.ObjectStore.Upload(ctx, file.LocalPath, key, storage.UploadOptions{}); err != nil {
return nil, fmt.Errorf("archive: upload run file %q to %q: %w", file.RelativePath, key, err)
}
runUploaded = append(runUploaded, rel)
runUploaded = append(runUploaded, file.RelativePath)
}
promotedUploaded := make([]string, 0, len(promotions))
@@ -247,36 +243,53 @@ func validateArchivePrerequisites(m *manifest.Manifest) error {
return nil
}
func archiveWorkDir(env *Env, m *manifest.Manifest) (string, error) {
workDir := strings.TrimSpace(m.LocalWorkDir)
if workDir != "" {
cleaned := filepath.Clean(workDir)
if info, err := os.Stat(cleaned); err == nil && info.IsDir() {
return cleaned, nil
}
}
func resolveArchiveRunRoot(env *Env, m *manifest.Manifest) (string, error) {
sessionID := strings.TrimSpace(env.Config.Session.SessionID)
if sessionID == "" {
if sessionID == "" && m != nil {
sessionID = strings.TrimSpace(m.SessionID)
}
campaign := strings.TrimSpace(env.Config.Session.Campaign)
if campaign == "" {
if campaign == "" && m != nil {
campaign = strings.TrimSpace(m.Campaign)
}
runID := strings.TrimSpace(m.RunID)
runID := ""
if m != nil {
runID = strings.TrimSpace(m.RunID)
}
if sessionID == "" || campaign == "" {
return "", fmt.Errorf("campaign and session id are required")
}
if runID == "" {
return "", fmt.Errorf("run id is required")
}
if campaign == "" || sessionID == "" {
return "", fmt.Errorf("campaign and session id are required")
canonical := filepath.Clean(artifacts.SessionRunRootForCampaign(env.Config.Pipeline.Workspace.Root, campaign, sessionID, runID))
canonicalExists, err := directoryExists(canonical)
if err != nil {
return "", fmt.Errorf("check canonical run root %q: %w", canonical, err)
}
runScoped := artifacts.SessionRunWorkDir(env.Config.Pipeline.Workspace.Root, campaign, sessionID, runID)
if info, err := os.Stat(runScoped); err == nil && info.IsDir() {
return runScoped, nil
if !canonicalExists {
return "", fmt.Errorf("run root not found for campaign %q session %q run %q at canonical path %q", campaign, sessionID, runID, canonical)
}
legacy := artifacts.SessionWorkDir(env.Config.Pipeline.Workspace.Root, sessionID)
return legacy, nil
return canonical, nil
}
func resolveArchiveSessionRoot(env *Env, m *manifest.Manifest) (string, error) {
sessionID := strings.TrimSpace(env.Config.Session.SessionID)
if sessionID == "" && m != nil {
sessionID = strings.TrimSpace(m.SessionID)
}
campaign := strings.TrimSpace(env.Config.Session.Campaign)
if campaign == "" && m != nil {
campaign = strings.TrimSpace(m.Campaign)
}
if sessionID == "" {
return "", fmt.Errorf("session id is required")
}
if campaign == "" {
return "", fmt.Errorf("campaign is required")
}
return filepath.Clean(artifacts.SessionWorkDirForCampaign(env.Config.Pipeline.Workspace.Root, campaign, sessionID)), nil
}
func archiveRunPrefix(env *Env, m *manifest.Manifest) (string, error) {
@@ -329,22 +342,27 @@ func archiveBucket(env *Env, m *manifest.Manifest) string {
return strings.TrimSpace(env.Config.Pipeline.Storage.S3.Bucket)
}
func resolveArchivePromotions(workDir string, rules []config.ArchivePromotionRule) ([]archivePromotion, error) {
func resolveArchivePromotions(sessionRoot string, rules []config.ArchivePromotionRule) ([]archivePromotion, error) {
sessionRoot = filepath.Clean(strings.TrimSpace(sessionRoot))
if sessionRoot == "" {
return nil, fmt.Errorf("session root is required")
}
out := make([]archivePromotion, 0, len(rules))
for _, rule := range rules {
from := strings.TrimSpace(rule.From)
to := strings.TrimSpace(rule.To)
required := rule.Required == nil || *rule.Required
localPath, err := resolveWorkDirRelativePath(workDir, from)
resolvedPath, err := resolveWorkDirRelativePath(sessionRoot, from)
if err != nil {
return nil, fmt.Errorf("promotion from %q: %w", from, err)
}
info, err := os.Stat(localPath)
info, err := os.Stat(resolvedPath)
exists := err == nil && !info.IsDir()
if err != nil && !os.IsNotExist(err) {
return nil, fmt.Errorf("promotion source %q: %w", from, err)
}
localPath := resolvedPath
out = append(out, archivePromotion{
From: from,
@@ -375,58 +393,97 @@ func resolveWorkDirRelativePath(workDir, rel string) (string, error) {
return cleanedFull, nil
}
func collectArchiveRunFiles(workDir string) ([]string, error) {
files := make([]string, 0, 64)
for _, dirName := range archiveRunUploadDirs {
fullDir := filepath.Join(workDir, dirName)
info, err := os.Stat(fullDir)
if err != nil {
if os.IsNotExist(err) {
continue
}
return nil, fmt.Errorf("stat %q: %w", fullDir, err)
func collectArchiveRunFiles(runRoot, manifestPath string) ([]archiveUploadFile, error) {
files := make([]archiveUploadFile, 0, 64)
err := filepath.WalkDir(runRoot, func(path string, d fs.DirEntry, walkErr error) error {
if walkErr != nil {
return walkErr
}
if !info.IsDir() {
continue
}
if err := filepath.WalkDir(fullDir, func(path string, d fs.DirEntry, walkErr error) error {
if walkErr != nil {
return walkErr
}
if d.IsDir() {
if d.IsDir() {
if path == runRoot {
return nil
}
rel, err := filepath.Rel(workDir, path)
relDir, err := filepath.Rel(runRoot, path)
if err != nil {
return fmt.Errorf("relative path from %q to %q: %w", workDir, path, err)
return fmt.Errorf("relative dir from %q to %q: %w", runRoot, path, err)
}
relDir = filepath.ToSlash(relDir)
// Preserve existing behavior: audio is not uploaded in archive run record.
if relDir == "audio" || strings.HasPrefix(relDir, "audio/") {
return filepath.SkipDir
}
rel = filepath.ToSlash(rel)
files = append(files, rel)
return nil
}); err != nil {
return nil, fmt.Errorf("walk %q: %w", fullDir, err)
}
rel, err := filepath.Rel(runRoot, path)
if err != nil {
return fmt.Errorf("relative path from %q to %q: %w", runRoot, path, err)
}
rel = filepath.ToSlash(rel)
files = append(files, archiveUploadFile{
RelativePath: rel,
LocalPath: path,
})
return nil
})
if err != nil {
return nil, fmt.Errorf("walk %q: %w", runRoot, err)
}
manifestPath := filepath.Join(workDir, "manifest.json")
manifestInfo, err := os.Stat(manifestPath)
if err != nil {
if os.IsNotExist(err) {
return nil, fmt.Errorf("manifest.json not found in workdir %q", workDir)
return nil, fmt.Errorf("manifest.json not found (checked path %q)", manifestPath)
}
return nil, fmt.Errorf("stat %q: %w", manifestPath, err)
}
if manifestInfo.IsDir() {
return nil, fmt.Errorf("manifest path %q is a directory", manifestPath)
}
files = append(files, "manifest.json")
files = append(files, archiveUploadFile{
RelativePath: "manifest.json",
LocalPath: manifestPath,
})
seen := map[string]archiveUploadFile{}
for _, file := range files {
seen[file.RelativePath] = file
}
files = files[:0]
for _, file := range seen {
files = append(files, file)
}
sort.Strings(files)
sort.Slice(files, func(i, j int) bool {
return files[i].RelativePath < files[j].RelativePath
})
return files, nil
}
func resolveArchiveRunManifestSource(runRoot string) (string, error) {
path := filepath.Join(filepath.Clean(runRoot), "manifest.json")
info, err := os.Stat(path)
if err != nil {
if os.IsNotExist(err) {
return "", fmt.Errorf("manifest.json not found in run root %q", runRoot)
}
return "", fmt.Errorf("stat %q: %w", path, err)
}
if info.IsDir() {
return "", fmt.Errorf("manifest path %q is a directory", path)
}
return path, nil
}
func directoryExists(path string) (bool, error) {
info, err := os.Stat(path)
if err == nil {
return info.IsDir(), nil
}
if os.IsNotExist(err) {
return false, nil
}
return false, err
}
func writeCurrentManifestSnapshot(m *manifest.Manifest, archiveMetadata map[string]any) (string, error) {
if m == nil {
return "", fmt.Errorf("manifest is required")

View File

@@ -3,6 +3,7 @@ package stage
import (
"context"
"errors"
"os"
"path/filepath"
"reflect"
"strings"
@@ -72,14 +73,15 @@ func TestArchiveUploadsRunRecordPromotionsAndCurrentPointer(t *testing.T) {
runPrefix := m.S3RunPrefix
sessionPrefix := m.S3SessionPrefix
wantRunUploads := []string{
"artifacts/session_recap.md",
"config/audita.generated.yml",
"inputs/session.yml",
"analyze/outputs/artifacts/session_recap.md",
"merge/config/seriatim.generated.yml",
"prepare/inputs/session.yml",
"prepare/outputs/audio/speaker.flac",
"logs/audita.stderr.log",
"manifest.json",
"reports/audita.report.json",
"transcripts/raw/speaker.json",
"transcripts/trimmed.json",
"polish/reports/audita.report.json",
"transcribe/outputs/transcripts/raw/speaker.json",
"trim/outputs/transcripts/trimmed.json",
}
for _, rel := range wantRunUploads {
key := runPrefix + rel
@@ -131,12 +133,11 @@ func TestArchiveUploadsRunRecordPromotionsAndCurrentPointer(t *testing.T) {
}
func TestArchiveUsesCustomPromotionRules(t *testing.T) {
env, m, workDir := archiveFixture(t)
env, m, _ := archiveFixture(t)
env.Config.Pipeline.Archive.PromoteArtifacts = []config.ArchivePromotionRule{
{From: "transcripts/trimmed.json", To: "published/trimmed.json", Required: boolPtr(true)},
{From: "artifacts/session_recap.md", To: "published/recap.md", Required: boolPtr(true)},
}
writeStageTestFile(t, filepath.Join(workDir, "published", "ignored.txt"), "ignore\n")
_, err := archiveStage{}.Run(context.Background(), env, m)
if err != nil {
@@ -182,6 +183,21 @@ func TestArchiveFailsWhenRequiredPromotionMissing(t *testing.T) {
}
}
func TestArchiveFailsWhenCanonicalRunRootMissing(t *testing.T) {
env, m, runRoot := archiveFixture(t)
if err := os.RemoveAll(runRoot); err != nil {
t.Fatalf("remove run root: %v", err)
}
_, err := archiveStage{}.Run(context.Background(), env, m)
if err == nil {
t.Fatal("expected missing run-root error, got nil")
}
if !strings.Contains(err.Error(), "run root not found") {
t.Fatalf("error = %v, want missing run-root error", err)
}
}
func TestArchiveDoesNotWriteCurrentPointerWhenPromotionUploadFails(t *testing.T) {
env, m, _ := archiveFixture(t)
fake := env.ObjectStore.(*storage.FakeBackend)
@@ -240,22 +256,27 @@ func archiveFixture(t *testing.T) (*Env, *manifest.Manifest, string) {
runID := "20260516T010203Z-1a2b3c4d"
campaign := "forsaken"
sessionID := "2026-04-19"
workDir := filepath.Join(root, "work", campaign, sessionID, runID)
sessionRoot := artifacts.SessionWorkDirForCampaign(root, campaign, sessionID)
runRoot := artifacts.SessionRunRootForCampaign(root, campaign, sessionID, runID)
writeStageTestFile(t, filepath.Join(workDir, "inputs", "session.yml"), "session_id: 2026-04-19\n")
writeStageTestFile(t, filepath.Join(workDir, "transcripts", "raw", "speaker.json"), "{}\n")
writeStageTestFile(t, filepath.Join(workDir, "transcripts", "trimmed.json"), "{}\n")
writeStageTestFile(t, filepath.Join(workDir, "artifacts", "session_recap.md"), "# recap\n")
writeStageTestFile(t, filepath.Join(workDir, "reports", "audita.report.json"), "{}\n")
writeStageTestFile(t, filepath.Join(workDir, "config", "audita.generated.yml"), "key: value\n")
writeStageTestFile(t, filepath.Join(workDir, "logs", "audita.stderr.log"), "stderr\n")
writeStageTestFile(t, filepath.Join(workDir, "audio", "speaker.flac"), "flac")
writeStageTestFile(t, filepath.Join(workDir, "manifest.json"), "{}\n")
writeStageTestFile(t, filepath.Join(sessionRoot, "transcripts", "trimmed.json"), "{}\n")
writeStageTestFile(t, filepath.Join(sessionRoot, "artifacts", "session_recap.md"), "# recap\n")
writeStageTestFile(t, filepath.Join(runRoot, "prepare", "inputs", "session.yml"), "session_id: 2026-04-19\n")
writeStageTestFile(t, filepath.Join(runRoot, "prepare", "outputs", "audio", "speaker.flac"), "flac\n")
writeStageTestFile(t, filepath.Join(runRoot, "transcribe", "outputs", "transcripts", "raw", "speaker.json"), "{}\n")
writeStageTestFile(t, filepath.Join(runRoot, "trim", "outputs", "transcripts", "trimmed.json"), "{}\n")
writeStageTestFile(t, filepath.Join(runRoot, "analyze", "outputs", "artifacts", "session_recap.md"), "# recap\n")
writeStageTestFile(t, filepath.Join(runRoot, "polish", "reports", "audita.report.json"), "{}\n")
writeStageTestFile(t, filepath.Join(runRoot, "merge", "config", "seriatim.generated.yml"), "key: value\n")
writeStageTestFile(t, filepath.Join(runRoot, "logs", "audita.stderr.log"), "stderr\n")
writeStageTestFile(t, filepath.Join(runRoot, "audio", "speaker.flac"), "flac")
writeStageTestFile(t, filepath.Join(runRoot, "manifest.json"), "{}\n")
m := manifest.New(sessionID, time.Date(2026, 5, 16, 1, 2, 3, 0, time.UTC))
m.Campaign = campaign
m.RunID = runID
m.LocalWorkDir = workDir
m.LocalWorkDir = runRoot
m.S3Bucket = "my-dnd-archive"
m.S3SessionPrefix = artifacts.S3SessionPrefix("dnd", campaign, sessionID)
m.S3RunPrefix = artifacts.S3RunPrefix(m.S3SessionPrefix, runID)
@@ -289,7 +310,7 @@ func archiveFixture(t *testing.T) (*Env, *manifest.Manifest, string) {
},
ObjectStore: &storage.FakeBackend{},
}
return env, m, workDir
return env, m, runRoot
}
type promotionFailingStore struct {

View File

@@ -7,6 +7,7 @@ import (
"path/filepath"
"sort"
"strings"
"time"
"gitea.maximumdirect.net/eric/narratio/internal/adapters/seriatim"
"gitea.maximumdirect.net/eric/narratio/internal/artifacts"
@@ -56,7 +57,11 @@ func (mergeStage) Run(ctx context.Context, env *Env, m *manifest.Manifest) (*Sta
return nil, fmt.Errorf("merge: session id is required")
}
paths := env.ArtifactStore.SessionPaths(sessionID)
paths := sessionPathsForEnv(env, sessionID)
runLayout, err := resolveRunStageLayout(env, m, paths, sessionID, "merge")
if err != nil {
return nil, fmt.Errorf("merge: resolve run-stage layout: %w", err)
}
inputs, err := discoverRawTranscripts(m, paths)
if err != nil {
@@ -80,16 +85,37 @@ func (mergeStage) Run(ctx context.Context, env *Env, m *manifest.Manifest) (*Sta
return nil, fmt.Errorf("merge: %w", err)
}
mergedPath := filepath.Join(paths.TranscriptsDir, "merged.json")
reportPath := filepath.Join(paths.ArtifactsDir, "seriatim.report.json")
canonicalMergedPath := filepath.Join(paths.TranscriptsDir, "merged.json")
mergedPath, err := runLocalPathForCanonical(runLayout, paths, canonicalMergedPath)
if err != nil {
return nil, fmt.Errorf("merge: resolve run-local merged transcript path: %w", err)
}
canonicalReportPath := filepath.Join(paths.ArtifactsDir, "seriatim.report.json")
reportPath := canonicalReportPath
if runLayout.Enabled {
reportPath, err = runLocalPathForCanonical(runLayout, paths, canonicalReportPath)
if err != nil {
return nil, fmt.Errorf("merge: resolve run-local report path: %w", err)
}
}
stdoutPath := filepath.Join(paths.LogsDir, "seriatim.stdout.log")
stderrPath := filepath.Join(paths.LogsDir, "seriatim.stderr.log")
genCfgPath := filepath.Join(paths.ConfigDir, "seriatim.generated.yml")
if runLayout.Enabled {
stdoutPath = filepath.Join(runLayout.LogsDir, "seriatim.stdout.log")
stderrPath = filepath.Join(runLayout.LogsDir, "seriatim.stderr.log")
genCfgPath = filepath.Join(runLayout.ConfigDir, "seriatim.generated.yml")
}
normalizedInputs, normalizeLogs, normalizeConfigs, normalizeMeta, err := normalizeMergeInputs(ctx, env, inputs, paths, runLayout)
if err != nil {
return nil, err
}
reportEnabled := env.Config.Pipeline.Seriatim.Report != nil && *env.Config.Pipeline.Seriatim.Report
req := seriatim.MergeRequest{
GeneratedConfigPath: genCfgPath,
InputTranscriptPaths: inputs,
InputTranscriptPaths: normalizedInputs,
OutputMergedTranscriptPath: mergedPath,
ReportPath: "",
SpeakersPath: speakersPath,
@@ -124,35 +150,50 @@ func (mergeStage) Run(ctx context.Context, env *Env, m *manifest.Manifest) (*Sta
}
}
outputs := []artifacts.Ref{{
Kind: "transcript_merged",
Category: "transcripts",
SessionID: sessionID,
AbsolutePath: finalMergedPath,
}}
promotedMerged, err := promoteRunLocalOutput(env.ArtifactStore, finalMergedPath, canonicalMergedPath, artifacts.Ref{
Kind: "transcript_merged",
Category: "transcripts",
SessionID: sessionID,
})
if err != nil {
return nil, fmt.Errorf("merge: promote merged transcript: %w", err)
}
outputs := []artifacts.Ref{promotedMerged}
if reportEnabled {
outputs = append(outputs, artifacts.Ref{
Kind: "seriatim_report",
Category: "artifacts",
SessionID: sessionID,
AbsolutePath: finalReportPath,
promotedReport, err := promoteRunLocalOutput(env.ArtifactStore, finalReportPath, canonicalReportPath, artifacts.Ref{
Kind: "seriatim_report",
Category: "artifacts",
SessionID: sessionID,
})
if err != nil {
return nil, fmt.Errorf("merge: promote report: %w", err)
}
outputs = append(outputs, promotedReport)
}
coalesceGap := any(nil)
if env.Config.Pipeline.Seriatim.CoalesceGap != nil {
coalesceGap = *env.Config.Pipeline.Seriatim.CoalesceGap
}
reportCanonicalPath := ""
if reportEnabled {
reportCanonicalPath = canonicalReportPath
}
meta := map[string]any{
"stage": "merge",
"input_transcripts_count": len(inputs),
"input_transcripts_count": len(normalizedInputs),
"input_transcript_paths": inputs,
"normalized_inputs_count": len(normalizedInputs),
"normalized_input_paths": normalizedInputs,
"normalize_inputs": normalizeMeta,
"output_schema": env.Config.Pipeline.Seriatim.OutputSchema,
"coalesce_gap": coalesceGap,
"report_enabled": reportEnabled,
"output_path": finalMergedPath,
"report_path": finalReportPath,
"run_output_path": finalMergedPath,
"output_path": canonicalMergedPath,
"run_report_path": finalReportPath,
"report_path": reportCanonicalPath,
"timeout": env.Config.Pipeline.Seriatim.Timeout,
"binary": env.Config.Pipeline.Seriatim.Binary,
"adapter_duration_ms": res.Duration.Milliseconds(),
@@ -172,12 +213,111 @@ func (mergeStage) Run(ctx context.Context, env *Env, m *manifest.Manifest) (*Sta
return &StageResult{
Outputs: outputs,
Logs: []string{stdoutPath, stderrPath},
GeneratedConfigs: []string{genCfgPath},
Logs: append(normalizeLogs, stdoutPath, stderrPath),
GeneratedConfigs: append(normalizeConfigs, genCfgPath),
Metadata: meta,
}, nil
}
type normalizeMergeInputMeta struct {
InputPath string `json:"input_path"`
OutputPath string `json:"output_path"`
StdoutLogPath string `json:"stdout_log_path"`
StderrLogPath string `json:"stderr_log_path"`
GeneratedConfig string `json:"generated_config_path"`
DurationMs int64 `json:"duration_ms"`
ExitCode int `json:"exit_code"`
InvokedBinary string `json:"invoked_binary"`
OutputSchema string `json:"output_schema"`
AdapterReportPath string `json:"adapter_report_path,omitempty"`
AdapterOutputPath string `json:"adapter_output_path,omitempty"`
}
func normalizeMergeInputs(
ctx context.Context,
env *Env,
rawInputs []string,
paths artifacts.SessionPaths,
runLayout runStageLayout,
) ([]string, []string, []string, []normalizeMergeInputMeta, error) {
normalizedInputs := make([]string, 0, len(rawInputs))
logs := make([]string, 0, len(rawInputs)*2)
configs := make([]string, 0, len(rawInputs))
meta := make([]normalizeMergeInputMeta, 0, len(rawInputs))
normalizedDir := filepath.Join(paths.TranscriptsRawDir, "normalized")
if runLayout.Enabled {
normalizedDir = filepath.Join(runLayout.ScratchDir, "normalized")
}
if err := os.MkdirAll(normalizedDir, 0o755); err != nil {
return nil, nil, nil, nil, fmt.Errorf("merge: ensure normalized transcripts directory %q: %w", normalizedDir, err)
}
var timeout time.Duration
timeoutRaw := strings.TrimSpace(env.Config.Pipeline.Seriatim.Timeout)
if timeoutRaw != "" {
parsed, err := time.ParseDuration(timeoutRaw)
if err != nil {
return nil, nil, nil, nil, fmt.Errorf("merge: parse seriatim timeout %q: %w", env.Config.Pipeline.Seriatim.Timeout, err)
}
timeout = parsed
}
for _, input := range rawInputs {
base := strings.TrimSuffix(filepath.Base(input), filepath.Ext(input))
outPath := filepath.Join(normalizedDir, base+".normalized.json")
stdoutPath := filepath.Join(paths.LogsDir, "seriatim.normalize."+base+".stdout.log")
stderrPath := filepath.Join(paths.LogsDir, "seriatim.normalize."+base+".stderr.log")
cfgPath := filepath.Join(paths.ConfigDir, "seriatim.normalize."+base+".generated.yml")
if runLayout.Enabled {
stdoutPath = filepath.Join(runLayout.LogsDir, "seriatim.normalize."+base+".stdout.log")
stderrPath = filepath.Join(runLayout.LogsDir, "seriatim.normalize."+base+".stderr.log")
cfgPath = filepath.Join(runLayout.ConfigDir, "seriatim.normalize."+base+".generated.yml")
}
req := seriatim.NormalizeRequest{
Binary: env.Config.Pipeline.Seriatim.Binary,
InputTranscriptPath: input,
OutputNormalizedPath: outPath,
OutputSchema: env.Config.Pipeline.Seriatim.OutputSchema,
ReportPath: "",
StdoutLogPath: stdoutPath,
StderrLogPath: stderrPath,
GeneratedConfigPath: cfgPath,
Timeout: timeout,
}
res, err := env.Seriatim.Normalize(ctx, req)
if err != nil {
return nil, nil, nil, nil, fmt.Errorf("merge: normalize input %q failed: %w", input, err)
}
finalOutputPath := outPath
if strings.TrimSpace(res.OutputNormalizedPath) != "" {
finalOutputPath = strings.TrimSpace(res.OutputNormalizedPath)
}
if err := validateTranscriptJSONFile(finalOutputPath); err != nil {
return nil, nil, nil, nil, fmt.Errorf("merge: normalized transcript %q invalid (input %q): %w", finalOutputPath, input, err)
}
normalizedInputs = append(normalizedInputs, finalOutputPath)
logs = append(logs, stdoutPath, stderrPath)
configs = append(configs, cfgPath)
meta = append(meta, normalizeMergeInputMeta{
InputPath: input,
OutputPath: finalOutputPath,
StdoutLogPath: stdoutPath,
StderrLogPath: stderrPath,
GeneratedConfig: cfgPath,
DurationMs: res.Duration.Milliseconds(),
ExitCode: res.ExitCode,
InvokedBinary: res.InvokedBinary,
OutputSchema: res.OutputSchema,
AdapterReportPath: res.ReportPath,
AdapterOutputPath: res.OutputNormalizedPath,
})
}
return normalizedInputs, logs, configs, meta, nil
}
func discoverRawTranscripts(m *manifest.Manifest, paths artifacts.SessionPaths) ([]string, error) {
fromManifest := make([]string, 0)
if m != nil && m.Stages != nil {

View File

@@ -2,6 +2,7 @@ package stage
import (
"context"
"fmt"
"os"
"path/filepath"
"strings"
@@ -14,9 +15,25 @@ import (
"gitea.maximumdirect.net/eric/narratio/internal/manifest"
)
type normalizeDirAssertingRunner struct {
*seriatim.FakeRunner
ExpectedDir string
}
func (r *normalizeDirAssertingRunner) Normalize(ctx context.Context, req seriatim.NormalizeRequest) (seriatim.NormalizeResult, error) {
info, err := os.Stat(r.ExpectedDir)
if err != nil {
return seriatim.NormalizeResult{}, fmt.Errorf("normalized directory check failed for %q: %w", r.ExpectedDir, err)
}
if !info.IsDir() {
return seriatim.NormalizeResult{}, fmt.Errorf("normalized path %q exists but is not a directory", r.ExpectedDir)
}
return r.FakeRunner.Normalize(ctx, req)
}
func TestMergeStageMergesRawTranscriptsAndRecordsMetadata(t *testing.T) {
env, m := setupMergeEnv(t)
paths := env.ArtifactStore.SessionPaths(m.SessionID)
paths := sessionPathsForEnv(env, m.SessionID)
inA := filepath.Join(paths.TranscriptsRawDir, "alice.json")
inB := filepath.Join(paths.TranscriptsRawDir, "bob.json")
@@ -54,6 +71,17 @@ func TestMergeStageMergesRawTranscriptsAndRecordsMetadata(t *testing.T) {
if len(req.InputTranscriptPaths) != 2 {
t.Fatalf("input transcripts = %#v, want 2", req.InputTranscriptPaths)
}
if len(fake.NormalizeRequests) != 2 {
t.Fatalf("normalize requests = %#v, want 2", fake.NormalizeRequests)
}
if fake.NormalizeRequests[0].InputTranscriptPath != inA || fake.NormalizeRequests[1].InputTranscriptPath != inB {
t.Fatalf("normalize request inputs = %#v", fake.NormalizeRequests)
}
for _, mergeIn := range req.InputTranscriptPaths {
if !strings.Contains(mergeIn, filepath.Join("transcripts", "raw", "normalized")) {
t.Fatalf("merge input path = %q, want normalized input path", mergeIn)
}
}
if len(result.Outputs) != 2 {
t.Fatalf("outputs len = %d, want 2", len(result.Outputs))
@@ -64,11 +92,11 @@ func TestMergeStageMergesRawTranscriptsAndRecordsMetadata(t *testing.T) {
if result.Outputs[1].Kind != "seriatim_report" {
t.Fatalf("output[1] kind = %q, want seriatim_report", result.Outputs[1].Kind)
}
if len(result.Logs) != 2 {
t.Fatalf("logs = %#v, want 2 paths", result.Logs)
if len(result.Logs) != 6 {
t.Fatalf("logs = %#v, want 6 paths (4 normalize + 2 merge)", result.Logs)
}
if len(result.GeneratedConfigs) != 1 {
t.Fatalf("generated configs = %#v, want 1 path", result.GeneratedConfigs)
if len(result.GeneratedConfigs) != 3 {
t.Fatalf("generated configs = %#v, want 3 paths (2 normalize + 1 merge)", result.GeneratedConfigs)
}
meta := result.Metadata
@@ -84,11 +112,17 @@ func TestMergeStageMergesRawTranscriptsAndRecordsMetadata(t *testing.T) {
if meta["input_transcripts_count"] != 2 {
t.Fatalf("metadata input_transcripts_count = %#v, want 2", meta["input_transcripts_count"])
}
if meta["normalized_inputs_count"] != 2 {
t.Fatalf("metadata normalized_inputs_count = %#v, want 2", meta["normalized_inputs_count"])
}
if _, ok := meta["normalize_inputs"]; !ok {
t.Fatalf("metadata normalize_inputs missing: %#v", meta)
}
}
func TestMergeStageFailsWhenNoRawTranscripts(t *testing.T) {
env, m := setupMergeEnv(t)
paths := env.ArtifactStore.SessionPaths(m.SessionID)
paths := sessionPathsForEnv(env, m.SessionID)
writeFile(t, filepath.Join(paths.InputsDir, "speakers.yml"), "match: []\n")
writeFile(t, filepath.Join(paths.InputsDir, "autocorrect.yml"), "rules: []\n")
env.Seriatim = &seriatim.FakeRunner{}
@@ -104,7 +138,7 @@ func TestMergeStageFailsWhenNoRawTranscripts(t *testing.T) {
func TestMergeStageFailsOnInvalidInputJSON(t *testing.T) {
env, m := setupMergeEnv(t)
paths := env.ArtifactStore.SessionPaths(m.SessionID)
paths := sessionPathsForEnv(env, m.SessionID)
writeFile(t, filepath.Join(paths.TranscriptsRawDir, "alice.json"), "not-json")
writeFile(t, filepath.Join(paths.InputsDir, "speakers.yml"), "match: []\n")
writeFile(t, filepath.Join(paths.InputsDir, "autocorrect.yml"), "rules: []\n")
@@ -121,7 +155,7 @@ func TestMergeStageFailsOnInvalidInputJSON(t *testing.T) {
func TestMergeStageFailsWhenAdapterFails(t *testing.T) {
env, m := setupMergeEnv(t)
paths := env.ArtifactStore.SessionPaths(m.SessionID)
paths := sessionPathsForEnv(env, m.SessionID)
writeFile(t, filepath.Join(paths.TranscriptsRawDir, "alice.json"), `{"segments":[]}`)
writeFile(t, filepath.Join(paths.InputsDir, "speakers.yml"), "match: []\n")
writeFile(t, filepath.Join(paths.InputsDir, "autocorrect.yml"), "rules: []\n")
@@ -138,7 +172,7 @@ func TestMergeStageFailsWhenAdapterFails(t *testing.T) {
func TestMergeStageFallsBackToRawDirectoryWhenTranscribeOutputsMissing(t *testing.T) {
env, m := setupMergeEnv(t)
paths := env.ArtifactStore.SessionPaths(m.SessionID)
paths := sessionPathsForEnv(env, m.SessionID)
writeFile(t, filepath.Join(paths.TranscriptsRawDir, "alice.json"), `{"segments":[]}`)
writeFile(t, filepath.Join(paths.InputsDir, "speakers.yml"), "match: []\n")
writeFile(t, filepath.Join(paths.InputsDir, "autocorrect.yml"), "rules: []\n")
@@ -152,11 +186,14 @@ func TestMergeStageFallsBackToRawDirectoryWhenTranscribeOutputsMissing(t *testin
if len(fake.Requests) != 1 || len(fake.Requests[0].InputTranscriptPaths) != 1 {
t.Fatalf("fallback inputs = %#v", fake.Requests)
}
if len(fake.NormalizeRequests) != 1 {
t.Fatalf("normalize requests = %#v, want 1", fake.NormalizeRequests)
}
}
func TestMergeStageResolvesWorkspaceQualifiedManifestOutputsWithoutDuplication(t *testing.T) {
env, m := setupMergeEnvWithRelativeWorkspaceRoot(t)
paths := env.ArtifactStore.SessionPaths(m.SessionID)
paths := sessionPathsForEnv(env, m.SessionID)
rawPath := filepath.Join(paths.TranscriptsRawDir, "alice.json")
writeFile(t, rawPath, `{"segments":[]}`)
@@ -180,14 +217,114 @@ func TestMergeStageResolvesWorkspaceQualifiedManifestOutputsWithoutDuplication(t
if len(got) != 1 {
t.Fatalf("input transcript paths = %#v, want len 1", got)
}
if got[0] != filepath.Clean(rawPath) {
t.Fatalf("resolved transcript path = %q, want %q", got[0], filepath.Clean(rawPath))
if !strings.Contains(got[0], filepath.Join("transcripts", "raw", "normalized")) {
t.Fatalf("resolved transcript path = %q, want normalized path under transcripts/raw/normalized", got[0])
}
}
func TestMergeStageCreatesNormalizedRawDirectoryBeforeNormalize(t *testing.T) {
env, m := setupMergeEnv(t)
paths := sessionPathsForEnv(env, m.SessionID)
rawPath := filepath.Join(paths.TranscriptsRawDir, "alice.json")
writeFile(t, rawPath, `{"segments":[]}`)
writeFile(t, filepath.Join(paths.InputsDir, "speakers.yml"), "match: []\n")
writeFile(t, filepath.Join(paths.InputsDir, "autocorrect.yml"), "rules: []\n")
normalizedDir := filepath.Join(paths.TranscriptsRawDir, "normalized")
if err := os.RemoveAll(normalizedDir); err != nil {
t.Fatalf("remove normalized dir: %v", err)
}
env.Seriatim = &normalizeDirAssertingRunner{
FakeRunner: &seriatim.FakeRunner{},
ExpectedDir: normalizedDir,
}
if _, err := (mergeStage{}).Run(context.Background(), env, m); err != nil {
t.Fatalf("merge.Run() error = %v", err)
}
}
func TestMergeStageFailsWhenNormalizeAdapterFails(t *testing.T) {
env, m := setupMergeEnv(t)
paths := sessionPathsForEnv(env, m.SessionID)
in := filepath.Join(paths.TranscriptsRawDir, "alice.json")
writeFile(t, in, `{"segments":[]}`)
writeFile(t, filepath.Join(paths.InputsDir, "speakers.yml"), "match: []\n")
writeFile(t, filepath.Join(paths.InputsDir, "autocorrect.yml"), "rules: []\n")
env.Seriatim = &seriatim.FakeRunner{NormalizeErr: context.DeadlineExceeded}
_, err := (mergeStage{}).Run(context.Background(), env, m)
if err == nil {
t.Fatal("expected error, got nil")
}
if !strings.Contains(err.Error(), "normalize input") {
t.Fatalf("error = %q", err.Error())
}
}
func TestMergeStageFailsWhenNormalizedOutputInvalid(t *testing.T) {
env, m := setupMergeEnv(t)
paths := sessionPathsForEnv(env, m.SessionID)
in := filepath.Join(paths.TranscriptsRawDir, "alice.json")
writeFile(t, in, `{"segments":[]}`)
writeFile(t, filepath.Join(paths.InputsDir, "speakers.yml"), "match: []\n")
writeFile(t, filepath.Join(paths.InputsDir, "autocorrect.yml"), "rules: []\n")
badNormalized := filepath.Join(paths.ArtifactsDir, "bad.normalized.json")
writeFile(t, badNormalized, "not-json")
env.Seriatim = &seriatim.FakeRunner{
NormalizeResult: seriatim.NormalizeResult{
OutputNormalizedPath: badNormalized,
},
}
_, err := (mergeStage{}).Run(context.Background(), env, m)
if err == nil {
t.Fatal("expected error, got nil")
}
if !strings.Contains(err.Error(), "normalized transcript") {
t.Fatalf("error = %q", err.Error())
}
}
func TestMergeStageUsesRunLocalPathsAndPromotesCanonical(t *testing.T) {
env, m := setupMergeEnv(t)
env.Config.Session.Campaign = "sample-campaign"
m.Campaign = "sample-campaign"
m.RunID = "20260518T010203Z-abcdef12"
paths := sessionPathsForEnv(env, m.SessionID)
in := filepath.Join(paths.TranscriptsRawDir, "alice.json")
writeFile(t, in, `{"segments":[]}`)
writeFile(t, filepath.Join(paths.InputsDir, "speakers.yml"), "match: []\n")
writeFile(t, filepath.Join(paths.InputsDir, "autocorrect.yml"), "rules: []\n")
fake := &seriatim.FakeRunner{}
env.Seriatim = fake
result, err := (mergeStage{}).Run(context.Background(), env, m)
if err != nil {
t.Fatalf("merge.Run() error = %v", err)
}
if len(fake.Requests) != 1 {
t.Fatalf("requests = %d, want 1", len(fake.Requests))
}
req := fake.Requests[0]
if !strings.Contains(req.OutputMergedTranscriptPath, filepath.Join("runs", m.RunID, "merge", "outputs")) {
t.Fatalf("run output path = %q, want run-local path", req.OutputMergedTranscriptPath)
}
if len(result.Outputs) == 0 {
t.Fatalf("outputs = %#v, want promoted outputs", result.Outputs)
}
if strings.Contains(result.Outputs[0].AbsolutePath, string(filepath.Separator)+"runs"+string(filepath.Separator)) {
t.Fatalf("promoted output path = %q, want canonical session path", result.Outputs[0].AbsolutePath)
}
}
func TestMergeStageResolvesSessionRelativeManifestOutputs(t *testing.T) {
env, m := setupMergeEnv(t)
paths := env.ArtifactStore.SessionPaths(m.SessionID)
paths := sessionPathsForEnv(env, m.SessionID)
rawPath := filepath.Join(paths.TranscriptsRawDir, "alice.json")
writeFile(t, rawPath, `{"segments":[]}`)
@@ -211,8 +348,8 @@ func TestMergeStageResolvesSessionRelativeManifestOutputs(t *testing.T) {
if len(got) != 1 {
t.Fatalf("input transcript paths = %#v, want len 1", got)
}
if got[0] != filepath.Clean(rawPath) {
t.Fatalf("resolved transcript path = %q, want %q", got[0], filepath.Clean(rawPath))
if !strings.Contains(got[0], filepath.Join("transcripts", "raw", "normalized")) {
t.Fatalf("resolved transcript path = %q, want normalized path under transcripts/raw/normalized", got[0])
}
}
@@ -269,7 +406,7 @@ func setupMergeEnvWithWorkspace(t *testing.T, workspace string) (*Env, *manifest
}
store := artifacts.NewLocalStore(workspace)
if _, err := store.EnsureLayout("2026-05-03"); err != nil {
if _, err := store.EnsureLayoutFor("sample-campaign", "2026-05-03"); err != nil {
t.Fatalf("EnsureLayout() error = %v", err)
}
return &Env{

View File

@@ -54,7 +54,11 @@ func (normalizeStage) Run(ctx context.Context, env *Env, m *manifest.Manifest) (
return nil, fmt.Errorf("normalize: session id is required")
}
paths := env.ArtifactStore.SessionPaths(sessionID)
paths := sessionPathsForEnv(env, sessionID)
runLayout, err := resolveRunStageLayout(env, m, paths, sessionID, "normalize")
if err != nil {
return nil, fmt.Errorf("normalize: resolve run-stage layout: %w", err)
}
processedPath, processedSource, err := discoverProcessedTranscript(m, paths)
if err != nil {
return nil, fmt.Errorf("normalize: resolve processed transcript: %w", err)
@@ -67,18 +71,34 @@ func (normalizeStage) Run(ctx context.Context, env *Env, m *manifest.Manifest) (
}
normalizeCfg := normalizeConfigOrDefault(env.Config.Pipeline.Normalize)
normalizedPath, err := resolveScriptoriumOutputPath(paths, normalizeCfg.OutputPath)
canonicalNormalizedPath, err := resolveScriptoriumOutputPath(paths, normalizeCfg.OutputPath)
if err != nil {
return nil, fmt.Errorf("normalize: resolve normalized output path: %w", err)
}
normalizedPath, err := runLocalPathForCanonical(runLayout, paths, canonicalNormalizedPath)
if err != nil {
return nil, fmt.Errorf("normalize: resolve run-local normalized output path: %w", err)
}
reportEnabled := normalizeCfg.Report != nil && *normalizeCfg.Report
reportPath := ""
canonicalReportPath := filepath.Join(paths.ArtifactsDir, "seriatim.normalize.report.json")
if reportEnabled {
reportPath = filepath.Join(paths.ArtifactsDir, "seriatim.normalize.report.json")
reportPath = canonicalReportPath
if runLayout.Enabled {
reportPath, err = runLocalPathForCanonical(runLayout, paths, canonicalReportPath)
if err != nil {
return nil, fmt.Errorf("normalize: resolve run-local report path: %w", err)
}
}
}
stdoutPath := filepath.Join(paths.LogsDir, "seriatim.normalize.stdout.log")
stderrPath := filepath.Join(paths.LogsDir, "seriatim.normalize.stderr.log")
generatedConfigPath := filepath.Join(paths.ConfigDir, "seriatim.normalize.generated.yml")
if runLayout.Enabled {
stdoutPath = filepath.Join(runLayout.LogsDir, "seriatim.normalize.stdout.log")
stderrPath = filepath.Join(runLayout.LogsDir, "seriatim.normalize.stderr.log")
generatedConfigPath = filepath.Join(runLayout.ConfigDir, "seriatim.normalize.generated.yml")
}
timeout, err := resolveTrimSeriatimTimeout(env.Config.Pipeline.Seriatim.Timeout)
if err != nil {
return nil, fmt.Errorf("normalize: resolve seriatim timeout: %w", err)
@@ -113,44 +133,56 @@ func (normalizeStage) Run(ctx context.Context, env *Env, m *manifest.Manifest) (
}
}
outputs := []artifacts.Ref{{
Kind: "transcript_normalized",
Category: "transcripts",
SessionID: sessionID,
AbsolutePath: finalNormalizedPath,
}}
promotedNormalized, err := promoteRunLocalOutput(env.ArtifactStore, finalNormalizedPath, canonicalNormalizedPath, artifacts.Ref{
Kind: "transcript_normalized",
Category: "transcripts",
SessionID: sessionID,
})
if err != nil {
return nil, fmt.Errorf("normalize: promote normalized transcript: %w", err)
}
outputs := []artifacts.Ref{promotedNormalized}
if reportEnabled {
outputs = append(outputs, artifacts.Ref{
Kind: "seriatim_normalize_report",
Category: "artifacts",
SessionID: sessionID,
AbsolutePath: finalReportPath,
promotedReport, err := promoteRunLocalOutput(env.ArtifactStore, finalReportPath, canonicalReportPath, artifacts.Ref{
Kind: "seriatim_normalize_report",
Category: "artifacts",
SessionID: sessionID,
})
if err != nil {
return nil, fmt.Errorf("normalize: promote report: %w", err)
}
outputs = append(outputs, promotedReport)
}
reportCanonicalPath := ""
if reportEnabled {
reportCanonicalPath = canonicalReportPath
}
meta := map[string]any{
"stage": "normalize",
"processed_transcript_path": processedPath,
"processed_transcript_source": processedSource,
"normalized_transcript_path": finalNormalizedPath,
"normalized_transcript_source": "stage.normalize.output",
"output_schema": normalizeCfg.OutputSchema,
"report_enabled": reportEnabled,
"report_path": finalReportPath,
"timeout": env.Config.Pipeline.Seriatim.Timeout,
"binary": env.Config.Pipeline.Seriatim.Binary,
"stdout_log_path": stdoutPath,
"stderr_log_path": stderrPath,
"generated_config_path": generatedConfigPath,
"adapter_duration_ms": res.Duration.Milliseconds(),
"adapter_exit_code": res.ExitCode,
"adapter_invoked_binary": res.InvokedBinary,
"adapter_output_schema": res.OutputSchema,
"adapter_output_path": res.OutputNormalizedPath,
"adapter_report_path": res.ReportPath,
"adapter_generated_config": res.GeneratedConfigPath,
"adapter_stdout_log_path": res.StdoutLogPath,
"adapter_stderr_log_path": res.StderrLogPath,
"stage": "normalize",
"processed_transcript_path": processedPath,
"processed_transcript_source": processedSource,
"run_normalized_transcript_path": finalNormalizedPath,
"normalized_transcript_path": canonicalNormalizedPath,
"normalized_transcript_source": "stage.normalize.output",
"output_schema": normalizeCfg.OutputSchema,
"report_enabled": reportEnabled,
"run_report_path": finalReportPath,
"report_path": reportCanonicalPath,
"timeout": env.Config.Pipeline.Seriatim.Timeout,
"binary": env.Config.Pipeline.Seriatim.Binary,
"stdout_log_path": stdoutPath,
"stderr_log_path": stderrPath,
"generated_config_path": generatedConfigPath,
"adapter_duration_ms": res.Duration.Milliseconds(),
"adapter_exit_code": res.ExitCode,
"adapter_invoked_binary": res.InvokedBinary,
"adapter_output_schema": res.OutputSchema,
"adapter_output_path": res.OutputNormalizedPath,
"adapter_report_path": res.ReportPath,
"adapter_generated_config": res.GeneratedConfigPath,
"adapter_stdout_log_path": res.StdoutLogPath,
"adapter_stderr_log_path": res.StderrLogPath,
}
if res.Metadata != nil {
meta["adapter_metadata"] = res.Metadata

View File

@@ -16,7 +16,7 @@ import (
func TestNormalizeStageConsumesProcessedTranscriptFromManifest(t *testing.T) {
env, m, ser := setupNormalizeEnv(t)
paths := env.ArtifactStore.SessionPaths(m.SessionID)
paths := sessionPathsForEnv(env, m.SessionID)
manifestProcessed := filepath.Join(paths.ArtifactsDir, "processed.from-manifest.json")
writeFile(t, manifestProcessed, `{"segments":[{"id":10}]}`)
writeFile(t, filepath.Join(paths.TranscriptsDir, "processed.json"), `{"segments":[{"id":99}]}`)
@@ -43,7 +43,7 @@ func TestNormalizeStageConsumesProcessedTranscriptFromManifest(t *testing.T) {
func TestNormalizeStageFallsBackToProcessedTranscriptPath(t *testing.T) {
env, m, ser := setupNormalizeEnv(t)
paths := env.ArtifactStore.SessionPaths(m.SessionID)
paths := sessionPathsForEnv(env, m.SessionID)
fallback := filepath.Join(paths.TranscriptsDir, "processed.json")
writeFile(t, fallback, `{"segments":[{"id":1}]}`)
@@ -72,7 +72,7 @@ func TestNormalizeStageFailsWhenProcessedTranscriptMissing(t *testing.T) {
func TestNormalizeStageFailsWhenProcessedTranscriptInvalidJSON(t *testing.T) {
env, m, _ := setupNormalizeEnv(t)
paths := env.ArtifactStore.SessionPaths(m.SessionID)
paths := sessionPathsForEnv(env, m.SessionID)
writeFile(t, filepath.Join(paths.TranscriptsDir, "processed.json"), "not-json")
_, err := (normalizeStage{}).Run(context.Background(), env, m)
@@ -86,7 +86,7 @@ func TestNormalizeStageFailsWhenProcessedTranscriptInvalidJSON(t *testing.T) {
func TestNormalizeStageFailsWhenProcessedTranscriptMissingSegments(t *testing.T) {
env, m, _ := setupNormalizeEnv(t)
paths := env.ArtifactStore.SessionPaths(m.SessionID)
paths := sessionPathsForEnv(env, m.SessionID)
writeFile(t, filepath.Join(paths.TranscriptsDir, "processed.json"), `{"schema":"audita.processed.v1"}`)
_, err := (normalizeStage{}).Run(context.Background(), env, m)
@@ -100,7 +100,7 @@ func TestNormalizeStageFailsWhenProcessedTranscriptMissingSegments(t *testing.T)
func TestNormalizeStagePassesConfiguredOutputSchemaToAdapter(t *testing.T) {
env, m, ser := setupNormalizeEnv(t)
paths := env.ArtifactStore.SessionPaths(m.SessionID)
paths := sessionPathsForEnv(env, m.SessionID)
writeFile(t, filepath.Join(paths.TranscriptsDir, "processed.json"), `{"segments":[{"id":1}]}`)
env.Config.Pipeline.Normalize.OutputSchema = "seriatim-full"
@@ -121,7 +121,7 @@ func TestNormalizeStagePassesConfiguredOutputSchemaToAdapter(t *testing.T) {
func TestNormalizeStageRecordsNormalizedTranscriptOutputKind(t *testing.T) {
env, m, _ := setupNormalizeEnv(t)
paths := env.ArtifactStore.SessionPaths(m.SessionID)
paths := sessionPathsForEnv(env, m.SessionID)
writeFile(t, filepath.Join(paths.TranscriptsDir, "processed.json"), `{"segments":[{"id":1}]}`)
result, err := (normalizeStage{}).Run(context.Background(), env, m)
@@ -138,7 +138,7 @@ func TestNormalizeStageRecordsNormalizedTranscriptOutputKind(t *testing.T) {
func TestNormalizeStageRecordsReportLogAndGeneratedConfigRefs(t *testing.T) {
env, m, _ := setupNormalizeEnv(t)
paths := env.ArtifactStore.SessionPaths(m.SessionID)
paths := sessionPathsForEnv(env, m.SessionID)
writeFile(t, filepath.Join(paths.TranscriptsDir, "processed.json"), `{"segments":[{"id":1}]}`)
report := true
env.Config.Pipeline.Normalize.Report = &report
@@ -164,7 +164,7 @@ func TestNormalizeStageRecordsReportLogAndGeneratedConfigRefs(t *testing.T) {
func TestNormalizeStageFailsWhenAdapterReturnsError(t *testing.T) {
env, m, ser := setupNormalizeEnv(t)
paths := env.ArtifactStore.SessionPaths(m.SessionID)
paths := sessionPathsForEnv(env, m.SessionID)
writeFile(t, filepath.Join(paths.TranscriptsDir, "processed.json"), `{"segments":[{"id":1}]}`)
ser.NormalizeErr = errors.New("normalize failed")
@@ -179,7 +179,7 @@ func TestNormalizeStageFailsWhenAdapterReturnsError(t *testing.T) {
func TestNormalizeStageFailsWhenNormalizedOutputInvalid(t *testing.T) {
env, m, ser := setupNormalizeEnv(t)
paths := env.ArtifactStore.SessionPaths(m.SessionID)
paths := sessionPathsForEnv(env, m.SessionID)
writeFile(t, filepath.Join(paths.TranscriptsDir, "processed.json"), `{"segments":[{"id":1}]}`)
badOutput := filepath.Join(paths.TranscriptsDir, "normalized.bad.json")
writeFile(t, badOutput, "not-json")
@@ -196,7 +196,7 @@ func TestNormalizeStageFailsWhenNormalizedOutputInvalid(t *testing.T) {
func TestNormalizeStageReportEnabledFailsWhenReportMissing(t *testing.T) {
env, m, ser := setupNormalizeEnv(t)
paths := env.ArtifactStore.SessionPaths(m.SessionID)
paths := sessionPathsForEnv(env, m.SessionID)
writeFile(t, filepath.Join(paths.TranscriptsDir, "processed.json"), `{"segments":[{"id":1}]}`)
report := true
env.Config.Pipeline.Normalize.Report = &report
@@ -211,6 +211,33 @@ func TestNormalizeStageReportEnabledFailsWhenReportMissing(t *testing.T) {
}
}
func TestNormalizeStageUsesRunLocalPathsAndPromotesCanonical(t *testing.T) {
env, m, ser := setupNormalizeEnv(t)
env.Config.Session.Campaign = "sample-campaign"
m.Campaign = "sample-campaign"
m.RunID = "20260518T010203Z-abcdef12"
paths := sessionPathsForEnv(env, m.SessionID)
writeFile(t, filepath.Join(paths.TranscriptsDir, "processed.json"), `{"segments":[{"id":1}]}`)
result, err := (normalizeStage{}).Run(context.Background(), env, m)
if err != nil {
t.Fatalf("normalize.Run() error = %v", err)
}
if len(ser.NormalizeRequests) != 1 {
t.Fatalf("normalize requests = %d, want 1", len(ser.NormalizeRequests))
}
req := ser.NormalizeRequests[0]
if !strings.Contains(req.OutputNormalizedPath, filepath.Join("runs", m.RunID, "normalize", "outputs")) {
t.Fatalf("run output path = %q, want run-local path", req.OutputNormalizedPath)
}
if len(result.Outputs) == 0 {
t.Fatalf("outputs = %#v, want promoted outputs", result.Outputs)
}
if strings.Contains(result.Outputs[0].AbsolutePath, string(filepath.Separator)+"runs"+string(filepath.Separator)) {
t.Fatalf("promoted output path = %q, want canonical session path", result.Outputs[0].AbsolutePath)
}
}
func setupNormalizeEnv(t *testing.T) (*Env, *manifest.Manifest, *seriatim.FakeRunner) {
t.Helper()
workspace := t.TempDir()
@@ -243,7 +270,7 @@ func setupNormalizeEnv(t *testing.T) (*Env, *manifest.Manifest, *seriatim.FakeRu
}
store := artifacts.NewLocalStore(workspace)
if _, err := store.EnsureLayout("2026-05-03"); err != nil {
if _, err := store.EnsureLayoutFor("sample-campaign", "2026-05-03"); err != nil {
t.Fatalf("EnsureLayout() error = %v", err)
}

View File

@@ -64,6 +64,7 @@ func TestStagesReturnExpectedMetadata(t *testing.T) {
},
Session: &config.SessionConfig{
SessionID: "2026-05-03",
Campaign: "sample-campaign",
Inputs: config.SessionInputsConfig{
AudioDir: "./audio",
SpeakersFile: "./speakers.yml",
@@ -88,7 +89,7 @@ func TestStagesReturnExpectedMetadata(t *testing.T) {
}
m.RunID = "20260516T000000Z-abcdef12"
m.Campaign = "sample-campaign"
m.LocalWorkDir = filepath.Join(root, "work", "sample-campaign", "2026-05-03", m.RunID)
m.LocalWorkDir = artifacts.SessionRunRootForCampaign(root, "sample-campaign", "2026-05-03", m.RunID)
m.S3RunPrefix = "dnd/campaigns/sample-campaign/sessions/2026-05-03/runs/" + m.RunID + "/"
m.S3Bucket = "my-dnd-archive"
if err := os.MkdirAll(filepath.Join(m.LocalWorkDir, "inputs"), 0o755); err != nil {
@@ -224,7 +225,7 @@ func TestPlaceholderAdapterErrorPropagation(t *testing.T) {
root := t.TempDir()
store := artifacts.NewLocalStore(root)
_, err := store.EnsureLayout("2026-05-03")
_, err := store.EnsureLayoutFor("sample-campaign", "2026-05-03")
if err != nil {
t.Fatalf("EnsureLayout() error = %v", err)
}

View File

@@ -55,7 +55,11 @@ func (polishStage) Run(ctx context.Context, env *Env, m *manifest.Manifest) (*St
return nil, fmt.Errorf("polish: session id is required")
}
paths := env.ArtifactStore.SessionPaths(sessionID)
paths := sessionPathsForEnv(env, sessionID)
runLayout, err := resolveRunStageLayout(env, m, paths, sessionID, "polish")
if err != nil {
return nil, fmt.Errorf("polish: resolve run-stage layout: %w", err)
}
mergedPath, source, err := discoverMergedTranscript(m, paths)
if err != nil {
return nil, fmt.Errorf("polish: resolve merged transcript: %w", err)
@@ -72,12 +76,29 @@ func (polishStage) Run(ctx context.Context, env *Env, m *manifest.Manifest) (*St
return nil, fmt.Errorf("polish: %w", err)
}
processedPath := filepath.Join(paths.TranscriptsDir, "processed.json")
reportPath := filepath.Join(paths.ArtifactsDir, "audita.report.json")
canonicalProcessedPath := filepath.Join(paths.TranscriptsDir, "processed.json")
processedPath, err := runLocalPathForCanonical(runLayout, paths, canonicalProcessedPath)
if err != nil {
return nil, fmt.Errorf("polish: resolve run-local processed transcript path: %w", err)
}
canonicalReportPath := filepath.Join(paths.ArtifactsDir, "audita.report.json")
reportPath := canonicalReportPath
if runLayout.Enabled {
reportPath, err = runLocalPathForCanonical(runLayout, paths, canonicalReportPath)
if err != nil {
return nil, fmt.Errorf("polish: resolve run-local report path: %w", err)
}
}
workDir := filepath.Join(paths.ArtifactsDir, "audita-work")
stdoutPath := filepath.Join(paths.LogsDir, "audita.stdout.log")
stderrPath := filepath.Join(paths.LogsDir, "audita.stderr.log")
generatedConfigPath := filepath.Join(paths.ConfigDir, "audita.generated.yml")
if runLayout.Enabled {
workDir = filepath.Join(runLayout.ScratchDir, "audita-work")
stdoutPath = filepath.Join(runLayout.LogsDir, "audita.stdout.log")
stderrPath = filepath.Join(runLayout.LogsDir, "audita.stderr.log")
generatedConfigPath = filepath.Join(runLayout.ConfigDir, "audita.generated.yml")
}
reportEnabled := env.Config.Pipeline.Audita.Report != nil && *env.Config.Pipeline.Audita.Report
req := audita.PolishRequest{
@@ -128,19 +149,25 @@ func (polishStage) Run(ctx context.Context, env *Env, m *manifest.Manifest) (*St
}
}
outputs := []artifacts.Ref{{
Kind: "transcript_processed",
Category: "transcripts",
SessionID: sessionID,
AbsolutePath: finalProcessedPath,
}}
promotedProcessed, err := promoteRunLocalOutput(env.ArtifactStore, finalProcessedPath, canonicalProcessedPath, artifacts.Ref{
Kind: "transcript_processed",
Category: "transcripts",
SessionID: sessionID,
})
if err != nil {
return nil, fmt.Errorf("polish: promote processed transcript: %w", err)
}
outputs := []artifacts.Ref{promotedProcessed}
if reportEnabled {
outputs = append(outputs, artifacts.Ref{
Kind: "audita_report",
Category: "artifacts",
SessionID: sessionID,
AbsolutePath: finalReportPath,
promotedReport, err := promoteRunLocalOutput(env.ArtifactStore, finalReportPath, canonicalReportPath, artifacts.Ref{
Kind: "audita_report",
Category: "artifacts",
SessionID: sessionID,
})
if err != nil {
return nil, fmt.Errorf("polish: promote report: %w", err)
}
outputs = append(outputs, promotedReport)
}
var validationConcurrency any
@@ -155,14 +182,20 @@ func (polishStage) Run(ctx context.Context, env *Env, m *manifest.Manifest) (*St
if env.Config.Pipeline.Audita.ProposalLLMConcurrency != nil {
proposalLLMConcurrency = *env.Config.Pipeline.Audita.ProposalLLMConcurrency
}
reportCanonicalPath := ""
if reportEnabled {
reportCanonicalPath = canonicalReportPath
}
meta := map[string]any{
"stage": "polish",
"merged_transcript_path": mergedPath,
"merged_transcript_source": source,
"glossary_path": glossaryPath,
"output_path": finalProcessedPath,
"report_path": finalReportPath,
"run_output_path": finalProcessedPath,
"output_path": canonicalProcessedPath,
"run_report_path": finalReportPath,
"report_path": reportCanonicalPath,
"audita_work_dir": workDir,
"report_enabled": reportEnabled,
"modules": append([]string(nil), req.Modules...),

View File

@@ -16,7 +16,7 @@ import (
func TestPolishStagePolishesMergedTranscriptAndRecordsMetadata(t *testing.T) {
env, m := setupPolishEnv(t)
paths := env.ArtifactStore.SessionPaths(m.SessionID)
paths := sessionPathsForEnv(env, m.SessionID)
mergedPath := filepath.Join(paths.TranscriptsDir, "merged.json")
writeFile(t, mergedPath, `{"segments":[]}`)
@@ -120,7 +120,7 @@ func TestPolishStagePolishesMergedTranscriptAndRecordsMetadata(t *testing.T) {
func TestPolishStageFallsBackToMergedTranscriptPath(t *testing.T) {
env, m := setupPolishEnv(t)
paths := env.ArtifactStore.SessionPaths(m.SessionID)
paths := sessionPathsForEnv(env, m.SessionID)
mergedPath := filepath.Join(paths.TranscriptsDir, "merged.json")
writeFile(t, mergedPath, `{"segments":[]}`)
writeFile(t, filepath.Join(paths.InputsDir, "glossary.yml"), "terms: []\n")
@@ -142,7 +142,7 @@ func TestPolishStageFallsBackToMergedTranscriptPath(t *testing.T) {
func TestPolishStageFailsWhenMergedTranscriptMissing(t *testing.T) {
env, m := setupPolishEnv(t)
paths := env.ArtifactStore.SessionPaths(m.SessionID)
paths := sessionPathsForEnv(env, m.SessionID)
writeFile(t, filepath.Join(paths.InputsDir, "glossary.yml"), "terms: []\n")
env.Audita = &audita.FakeRunner{}
@@ -157,7 +157,7 @@ func TestPolishStageFailsWhenMergedTranscriptMissing(t *testing.T) {
func TestPolishStageFailsWhenMergedTranscriptInvalidJSON(t *testing.T) {
env, m := setupPolishEnv(t)
paths := env.ArtifactStore.SessionPaths(m.SessionID)
paths := sessionPathsForEnv(env, m.SessionID)
writeFile(t, filepath.Join(paths.TranscriptsDir, "merged.json"), "not-json")
writeFile(t, filepath.Join(paths.InputsDir, "glossary.yml"), "terms: []\n")
env.Audita = &audita.FakeRunner{}
@@ -173,7 +173,7 @@ func TestPolishStageFailsWhenMergedTranscriptInvalidJSON(t *testing.T) {
func TestPolishStageFailsWhenGlossaryMissing(t *testing.T) {
env, m := setupPolishEnv(t)
paths := env.ArtifactStore.SessionPaths(m.SessionID)
paths := sessionPathsForEnv(env, m.SessionID)
writeFile(t, filepath.Join(paths.TranscriptsDir, "merged.json"), `{"segments":[]}`)
env.Audita = &audita.FakeRunner{}
@@ -188,7 +188,7 @@ func TestPolishStageFailsWhenGlossaryMissing(t *testing.T) {
func TestPolishStageFailsWhenAdapterFails(t *testing.T) {
env, m := setupPolishEnv(t)
paths := env.ArtifactStore.SessionPaths(m.SessionID)
paths := sessionPathsForEnv(env, m.SessionID)
writeFile(t, filepath.Join(paths.TranscriptsDir, "merged.json"), `{"segments":[]}`)
writeFile(t, filepath.Join(paths.InputsDir, "glossary.yml"), "terms: []\n")
env.Audita = &audita.FakeRunner{Err: errors.New("audita failed")}
@@ -204,7 +204,7 @@ func TestPolishStageFailsWhenAdapterFails(t *testing.T) {
func TestPolishStageFailsWhenProcessedOutputInvalid(t *testing.T) {
env, m := setupPolishEnv(t)
paths := env.ArtifactStore.SessionPaths(m.SessionID)
paths := sessionPathsForEnv(env, m.SessionID)
writeFile(t, filepath.Join(paths.TranscriptsDir, "merged.json"), `{"segments":[]}`)
writeFile(t, filepath.Join(paths.InputsDir, "glossary.yml"), "terms: []\n")
badPath := filepath.Join(paths.TranscriptsDir, "processed.invalid.json")
@@ -222,7 +222,7 @@ func TestPolishStageFailsWhenProcessedOutputInvalid(t *testing.T) {
func TestPolishStageFailsWhenReportInvalid(t *testing.T) {
env, m := setupPolishEnv(t)
paths := env.ArtifactStore.SessionPaths(m.SessionID)
paths := sessionPathsForEnv(env, m.SessionID)
writeFile(t, filepath.Join(paths.TranscriptsDir, "merged.json"), `{"segments":[]}`)
writeFile(t, filepath.Join(paths.InputsDir, "glossary.yml"), "terms: []\n")
badReport := filepath.Join(paths.ArtifactsDir, "bad.report.json")
@@ -238,6 +238,36 @@ func TestPolishStageFailsWhenReportInvalid(t *testing.T) {
}
}
func TestPolishStageUsesRunLocalPathsAndPromotesCanonical(t *testing.T) {
env, m := setupPolishEnv(t)
env.Config.Session.Campaign = "sample-campaign"
m.Campaign = "sample-campaign"
m.RunID = "20260518T010203Z-abcdef12"
paths := sessionPathsForEnv(env, m.SessionID)
writeFile(t, filepath.Join(paths.TranscriptsDir, "merged.json"), `{"segments":[]}`)
writeFile(t, filepath.Join(paths.InputsDir, "glossary.yml"), "terms: []\n")
fake := &audita.FakeRunner{}
env.Audita = fake
result, err := (polishStage{}).Run(context.Background(), env, m)
if err != nil {
t.Fatalf("polish.Run() error = %v", err)
}
if len(fake.Requests) != 1 {
t.Fatalf("requests = %d, want 1", len(fake.Requests))
}
req := fake.Requests[0]
if !strings.Contains(req.OutputProcessedPath, filepath.Join("runs", m.RunID, "polish", "outputs")) {
t.Fatalf("run output path = %q, want run-local path", req.OutputProcessedPath)
}
if len(result.Outputs) == 0 {
t.Fatalf("outputs = %#v, want promoted outputs", result.Outputs)
}
if strings.Contains(result.Outputs[0].AbsolutePath, string(filepath.Separator)+"runs"+string(filepath.Separator)) {
t.Fatalf("promoted output path = %q, want canonical session path", result.Outputs[0].AbsolutePath)
}
}
func setupPolishEnv(t *testing.T) (*Env, *manifest.Manifest) {
t.Helper()
workspace := t.TempDir()
@@ -284,7 +314,7 @@ func setupPolishEnv(t *testing.T) (*Env, *manifest.Manifest) {
}
store := artifacts.NewLocalStore(workspace)
if _, err := store.EnsureLayout("2026-05-03"); err != nil {
if _, err := store.EnsureLayoutFor("sample-campaign", "2026-05-03"); err != nil {
t.Fatalf("EnsureLayout() error = %v", err)
}
return &Env{

View File

@@ -54,7 +54,7 @@ func (prepareStage) Run(ctx context.Context, env *Env, m *manifest.Manifest) (*S
return nil, fmt.Errorf("prepare: session id is required")
}
paths, err := env.ArtifactStore.EnsureLayout(sessionID)
paths, err := ensureLayoutForEnv(env, sessionID)
if err != nil {
return nil, fmt.Errorf("prepare: ensure workdir layout: %w", err)
}
@@ -358,6 +358,10 @@ func pathsWorkDirForManifest(env *Env, m *manifest.Manifest, sessionID string) s
if env == nil || env.Config == nil || env.Config.Pipeline == nil || env.Config.Session == nil {
return ""
}
campaign := strings.TrimSpace(env.Config.Session.Campaign)
if campaign == "" {
return ""
}
if m != nil && strings.TrimSpace(m.LocalWorkDir) != "" {
return strings.TrimSpace(m.LocalWorkDir)
}
@@ -366,9 +370,9 @@ func pathsWorkDirForManifest(env *Env, m *manifest.Manifest, sessionID string) s
runID = strings.TrimSpace(m.RunID)
}
if runID != "" {
return artifacts.SessionRunWorkDir(env.Config.Pipeline.Workspace.Root, env.Config.Session.Campaign, sessionID, runID)
return artifacts.SessionRunRootForCampaign(env.Config.Pipeline.Workspace.Root, campaign, sessionID, runID)
}
return artifacts.SessionWorkDir(env.Config.Pipeline.Workspace.Root, sessionID)
return artifacts.SessionWorkDirForCampaign(env.Config.Pipeline.Workspace.Root, campaign, sessionID)
}
func resolvePath(baseDir, p string) (string, error) {

View File

@@ -33,7 +33,7 @@ func TestPrepareStageExplicitAudioFiles(t *testing.T) {
t.Fatalf("result metadata = %#v, want prepared=true", result)
}
paths := env.ArtifactStore.SessionPaths(m.SessionID)
paths := sessionPathsForEnv(env, m.SessionID)
for _, p := range []string{
filepath.Join(paths.InputsDir, "session.yml"),
filepath.Join(paths.InputsDir, "pipeline.resolved.yml"),
@@ -76,7 +76,7 @@ func TestPrepareStageAudioDirEnumeration(t *testing.T) {
t.Fatalf("audio_files_resolved = %#v, want 1", got)
}
paths := env.ArtifactStore.SessionPaths(m.SessionID)
paths := sessionPathsForEnv(env, m.SessionID)
if _, err := os.Stat(filepath.Join(paths.AudioDir, "a.flac")); err != nil {
t.Fatalf("expected copied flac: %v", err)
}
@@ -160,7 +160,7 @@ func TestPrepareStageS3AudioDownloadAndMaterialization(t *testing.T) {
env.Config.Pipeline.Spool = config.SpoolConfig{Root: filepath.Join(t.TempDir(), "spool")}
env.Config.Pipeline.Storage.S3 = &config.StorageS3Config{Bucket: "my-dnd-archive", RootPrefix: "dnd"}
m.RunID = "20260515T031522Z-a1b2c3d4"
m.LocalWorkDir = artifacts.SessionRunWorkDir(env.Config.Pipeline.Workspace.Root, "forsaken", m.SessionID, m.RunID)
m.LocalWorkDir = artifacts.SessionRunRootForCampaign(env.Config.Pipeline.Workspace.Root, "forsaken", m.SessionID, m.RunID)
m.LocalSpoolDir = artifacts.SessionSpoolAudioDir(env.Config.Pipeline.Spool.Root, "forsaken", m.SessionID, m.RunID)
fake := &storage.FakeBackend{}
@@ -256,7 +256,7 @@ func TestPrepareStageS3AudioFailures(t *testing.T) {
env.Config.Pipeline.Spool = config.SpoolConfig{Root: filepath.Join(t.TempDir(), "spool")}
env.Config.Pipeline.Storage.S3 = &config.StorageS3Config{Bucket: "my-dnd-archive", RootPrefix: "dnd"}
m.RunID = "20260515T031522Z-a1b2c3d4"
m.LocalWorkDir = artifacts.SessionRunWorkDir(env.Config.Pipeline.Workspace.Root, "forsaken", m.SessionID, m.RunID)
m.LocalWorkDir = artifacts.SessionRunRootForCampaign(env.Config.Pipeline.Workspace.Root, "forsaken", m.SessionID, m.RunID)
m.LocalSpoolDir = artifacts.SessionSpoolAudioDir(env.Config.Pipeline.Spool.Root, "forsaken", m.SessionID, m.RunID)
fake := &storage.FakeBackend{}

136
internal/stage/run_local.go Normal file
View File

@@ -0,0 +1,136 @@
package stage
import (
"fmt"
"os"
"path/filepath"
"strings"
"gitea.maximumdirect.net/eric/narratio/internal/artifacts"
"gitea.maximumdirect.net/eric/narratio/internal/manifest"
)
type runStageLayout struct {
Enabled bool
Root string
OutputsDir string
LogsDir string
ReportsDir string
ConfigDir string
ScratchDir string
}
func resolveRunStageLayout(
env *Env,
m *manifest.Manifest,
sessionPaths artifacts.SessionPaths,
sessionID, stageName string,
) (runStageLayout, error) {
if env == nil || env.Config == nil || env.Config.Pipeline == nil {
return runStageLayout{}, fmt.Errorf("stage environment pipeline config is required")
}
if strings.TrimSpace(sessionID) == "" {
return runStageLayout{}, fmt.Errorf("session id is required")
}
stageName = strings.TrimSpace(stageName)
if stageName == "" {
return runStageLayout{}, fmt.Errorf("stage name is required")
}
runID := ""
if m != nil {
runID = strings.TrimSpace(m.RunID)
}
campaign := strings.TrimSpace(env.Config.Session.Campaign)
if campaign == "" && m != nil {
campaign = strings.TrimSpace(m.Campaign)
}
// Compatibility fallback for direct stage tests and older call paths
// that execute a stage without a run id.
if runID == "" || campaign == "" {
return runStageLayout{}, nil
}
root := artifacts.SessionRunStageDirForCampaign(
env.Config.Pipeline.Workspace.Root,
campaign,
sessionID,
runID,
stageName,
)
layout := runStageLayout{
Enabled: true,
Root: root,
OutputsDir: filepath.Join(root, "outputs"),
LogsDir: filepath.Join(root, "logs"),
ReportsDir: filepath.Join(root, "reports"),
ConfigDir: filepath.Join(root, "config"),
ScratchDir: filepath.Join(root, "scratch"),
}
for _, dir := range []string{
layout.Root,
layout.OutputsDir,
layout.LogsDir,
layout.ReportsDir,
layout.ConfigDir,
layout.ScratchDir,
} {
if err := os.MkdirAll(dir, 0o755); err != nil {
return runStageLayout{}, fmt.Errorf("create run-stage directory %q: %w", dir, err)
}
}
return layout, nil
}
func runLocalPathForCanonical(layout runStageLayout, sessionPaths artifacts.SessionPaths, canonicalPath string) (string, error) {
if !layout.Enabled {
return filepath.Clean(canonicalPath), nil
}
cleanCanonical := filepath.Clean(strings.TrimSpace(canonicalPath))
if cleanCanonical == "" {
return "", fmt.Errorf("canonical path is required")
}
rel, err := filepath.Rel(filepath.Clean(sessionPaths.Root), cleanCanonical)
if err != nil {
return "", fmt.Errorf("derive session-relative path for %q: %w", cleanCanonical, err)
}
rel = filepath.Clean(rel)
if rel == "." || rel == ".." || strings.HasPrefix(rel, ".."+string(filepath.Separator)) {
return "", fmt.Errorf("canonical path %q is outside session root %q", cleanCanonical, sessionPaths.Root)
}
localPath := filepath.Join(layout.OutputsDir, rel)
if err := os.MkdirAll(filepath.Dir(localPath), 0o755); err != nil {
return "", fmt.Errorf("create run-local output parent for %q: %w", localPath, err)
}
return localPath, nil
}
func promoteRunLocalOutput(
store artifacts.Store,
srcPath, canonicalPath string,
ref artifacts.Ref,
) (artifacts.Ref, error) {
srcPath = filepath.Clean(strings.TrimSpace(srcPath))
canonicalPath = filepath.Clean(strings.TrimSpace(canonicalPath))
if srcPath == "" {
return artifacts.Ref{}, fmt.Errorf("source path is required")
}
if canonicalPath == "" {
return artifacts.Ref{}, fmt.Errorf("canonical destination path is required")
}
data, err := os.ReadFile(srcPath)
if err != nil {
return artifacts.Ref{}, fmt.Errorf("read run-local output %q: %w", srcPath, err)
}
if err := store.WriteFileAtomic(canonicalPath, data, 0o644); err != nil {
return artifacts.Ref{}, fmt.Errorf("promote output to %q: %w", canonicalPath, err)
}
checksum, err := store.Checksum(canonicalPath)
if err != nil {
return artifacts.Ref{}, fmt.Errorf("checksum promoted output %q: %w", canonicalPath, err)
}
ref.AbsolutePath = canonicalPath
ref.Checksum = checksum
return ref, nil
}

View File

@@ -0,0 +1,35 @@
package stage
import (
"os"
"path/filepath"
"testing"
"gitea.maximumdirect.net/eric/narratio/internal/artifacts"
)
func TestRunLocalPathForCanonicalCreatesParentDirectories(t *testing.T) {
root := t.TempDir()
sessionRoot := filepath.Join(root, "work", "dilfs", "2026-05-17")
layout := runStageLayout{
Enabled: true,
OutputsDir: filepath.Join(sessionRoot, "runs", "run-1", "merge", "outputs"),
}
if err := os.MkdirAll(layout.OutputsDir, 0o755); err != nil {
t.Fatalf("mkdir outputs dir: %v", err)
}
canonical := filepath.Join(sessionRoot, "transcripts", "merged.json")
got, err := runLocalPathForCanonical(layout, artifacts.SessionPaths{Root: sessionRoot}, canonical)
if err != nil {
t.Fatalf("runLocalPathForCanonical() error = %v", err)
}
want := filepath.Join(layout.OutputsDir, "transcripts", "merged.json")
if got != want {
t.Fatalf("runLocalPathForCanonical() = %q, want %q", got, want)
}
if _, err := os.Stat(filepath.Dir(got)); err != nil {
t.Fatalf("expected run-local parent directory to exist: %v", err)
}
}

View File

@@ -0,0 +1,23 @@
package stage
import (
"strings"
"gitea.maximumdirect.net/eric/narratio/internal/artifacts"
)
func sessionPathsForEnv(env *Env, sessionID string) artifacts.SessionPaths {
campaign := ""
if env != nil && env.Config != nil && env.Config.Session != nil {
campaign = strings.TrimSpace(env.Config.Session.Campaign)
}
return env.ArtifactStore.SessionPathsFor(campaign, sessionID)
}
func ensureLayoutForEnv(env *Env, sessionID string) (artifacts.SessionPaths, error) {
campaign := ""
if env != nil && env.Config != nil && env.Config.Session != nil {
campaign = strings.TrimSpace(env.Config.Session.Campaign)
}
return env.ArtifactStore.EnsureLayoutFor(campaign, sessionID)
}

View File

@@ -55,7 +55,11 @@ func (transcribeStage) Run(ctx context.Context, env *Env, m *manifest.Manifest)
return nil, fmt.Errorf("transcribe: session id is required")
}
paths := env.ArtifactStore.SessionPaths(sessionID)
paths := sessionPathsForEnv(env, sessionID)
runLayout, err := resolveRunStageLayout(env, m, paths, sessionID, "transcribe")
if err != nil {
return nil, fmt.Errorf("transcribe: resolve run-stage layout: %w", err)
}
audioFiles, err := discoverPreparedAudio(m, paths.AudioDir)
if err != nil {
return nil, fmt.Errorf("transcribe: resolve audio inputs: %w", err)
@@ -88,10 +92,15 @@ func (transcribeStage) Run(ctx context.Context, env *Env, m *manifest.Manifest)
return nil, fmt.Errorf("transcribe: duplicate speaker/audio basename %q from %q and %q", base, prev, audioPath)
}
seenSpeaker[base] = audioPath
canonicalOut := filepath.Join(paths.TranscriptsRawDir, base+".json")
runOut, err := runLocalPathForCanonical(runLayout, paths, canonicalOut)
if err != nil {
return nil, fmt.Errorf("transcribe: resolve run-local output path for %q: %w", base, err)
}
jobs = append(jobs, job{
speakerID: base,
audioPath: audioPath,
outPath: filepath.Join(paths.TranscriptsRawDir, base+".json"),
outPath: runOut,
})
}
@@ -197,12 +206,19 @@ dispatch:
sort.Strings(speakers)
outputs := make([]artifacts.Ref, 0, len(speakers))
runOutputPaths := make([]string, 0, len(speakers))
outputPaths := make([]string, 0, len(speakers))
orderedPerFile := make(map[string]any, len(speakers))
for _, speaker := range speakers {
ref := outputRef[speaker]
outputs = append(outputs, ref)
outputPaths = append(outputPaths, ref.AbsolutePath)
runOutputPaths = append(runOutputPaths, ref.AbsolutePath)
canonicalOut := filepath.Join(paths.TranscriptsRawDir, speaker+".json")
promoted, err := promoteRunLocalOutput(env.ArtifactStore, ref.AbsolutePath, canonicalOut, ref)
if err != nil {
return nil, fmt.Errorf("transcribe: promote %q output: %w", speaker, err)
}
outputs = append(outputs, promoted)
outputPaths = append(outputPaths, canonicalOut)
orderedPerFile[speaker] = perFile[speaker]
}
@@ -221,6 +237,7 @@ dispatch:
"retries": retries,
"retry_delay": env.Config.Pipeline.WhisperX.RetryDelay,
"timeout": env.Config.Pipeline.WhisperX.Timeout,
"run_output_paths": runOutputPaths,
"output_paths": outputPaths,
"per_file": orderedPerFile,
},

View File

@@ -53,8 +53,8 @@ func TestTranscribeStageTranscribesPreparedAudio(t *testing.T) {
}
sort.Strings(gotPaths)
wantPaths := []string{
filepath.Join(env.ArtifactStore.SessionPaths(m.SessionID).TranscriptsRawDir, "alice.json"),
filepath.Join(env.ArtifactStore.SessionPaths(m.SessionID).TranscriptsRawDir, "bob.json"),
filepath.Join(sessionPathsForEnv(env, m.SessionID).TranscriptsRawDir, "alice.json"),
filepath.Join(sessionPathsForEnv(env, m.SessionID).TranscriptsRawDir, "bob.json"),
}
sort.Strings(wantPaths)
if strings.Join(gotPaths, "|") != strings.Join(wantPaths, "|") {
@@ -191,6 +191,36 @@ func TestTranscribeStageInvalidJSONFails(t *testing.T) {
}
}
func TestTranscribeStageUsesRunLocalOutputAndPromotesCanonical(t *testing.T) {
env, m := setupTranscribeEnv(t, []string{"alice.flac"})
env.Config.Session.Campaign = "sample-campaign"
m.Campaign = "sample-campaign"
m.RunID = "20260518T010203Z-abcdef12"
fake := &whisperx.FakeClient{}
env.WhisperX = fake
if _, err := (prepareStage{}).Run(context.Background(), env, m); err != nil {
t.Fatalf("prepare.Run() error = %v", err)
}
result, err := (transcribeStage{}).Run(context.Background(), env, m)
if err != nil {
t.Fatalf("transcribe.Run() error = %v", err)
}
if len(fake.Requests) != 1 {
t.Fatalf("requests = %d, want 1", len(fake.Requests))
}
runOut := fake.Requests[0].OutputRawTranscriptPath
if !strings.Contains(runOut, filepath.Join("runs", m.RunID, "transcribe", "outputs")) {
t.Fatalf("run-local output path = %q, want runs/{run_id}/transcribe/outputs path", runOut)
}
if len(result.Outputs) != 1 {
t.Fatalf("outputs = %#v, want one output", result.Outputs)
}
if strings.Contains(result.Outputs[0].AbsolutePath, string(filepath.Separator)+"runs"+string(filepath.Separator)) {
t.Fatalf("promoted output path = %q, want canonical session path", result.Outputs[0].AbsolutePath)
}
}
func setupTranscribeEnv(t *testing.T, audioFiles []string) (*Env, *manifest.Manifest) {
t.Helper()
@@ -203,7 +233,7 @@ func setupTranscribeEnv(t *testing.T, audioFiles []string) (*Env, *manifest.Mani
sessionPath := filepath.Join(cfgDir, "session.yml")
pipelinePath := filepath.Join(cfgDir, "pipeline.yml")
writeFile(t, sessionPath, "session_id: 2026-05-03\n")
writeFile(t, sessionPath, "session_id: 2026-05-03\ncampaign: sample-campaign\n")
writeFile(t, pipelinePath, "workspace:\n root: "+workspace+"\n")
writeFile(t, filepath.Join(cfgDir, "speakers.yml"), "alice: alice.flac\n")
writeFile(t, filepath.Join(cfgDir, "autocorrect.yml"), "[]\n")
@@ -227,6 +257,7 @@ func setupTranscribeEnv(t *testing.T, audioFiles []string) (*Env, *manifest.Mani
},
Session: &config.SessionConfig{
SessionID: "2026-05-03",
Campaign: "sample-campaign",
Inputs: config.SessionInputsConfig{
AudioDir: "./audio",
SpeakersFile: "./speakers.yml",

View File

@@ -54,7 +54,11 @@ func (trimStage) Run(ctx context.Context, env *Env, m *manifest.Manifest) (*Stag
return nil, fmt.Errorf("trim: session id is required")
}
paths := env.ArtifactStore.SessionPaths(sessionID)
paths := sessionPathsForEnv(env, sessionID)
runLayout, err := resolveRunStageLayout(env, m, paths, sessionID, "trim")
if err != nil {
return nil, fmt.Errorf("trim: resolve run-stage layout: %w", err)
}
normalizedPath, normalizedSource, err := discoverNormalizedTranscript(m, paths)
if err != nil {
return nil, fmt.Errorf("trim: resolve normalized transcript: %w", err)
@@ -69,10 +73,14 @@ func (trimStage) Run(ctx context.Context, env *Env, m *manifest.Manifest) (*Stag
trimCfg := env.Config.Pipeline.Trim
enabled := trimCfg != nil && trimCfg.Enabled
trimmedPath, err := resolveTrimmedOutputPath(paths, trimCfg)
canonicalTrimmedPath, err := resolveTrimmedOutputPath(paths, trimCfg)
if err != nil {
return nil, fmt.Errorf("trim: resolve trimmed output path: %w", err)
}
trimmedPath, err := runLocalPathForCanonical(runLayout, paths, canonicalTrimmedPath)
if err != nil {
return nil, fmt.Errorf("trim: resolve run-local trimmed output path: %w", err)
}
logPaths := []string{}
generatedConfigs := []string{}
@@ -81,7 +89,8 @@ func (trimStage) Run(ctx context.Context, env *Env, m *manifest.Manifest) (*Stag
"trim_enabled": enabled,
"normalized_transcript_path": normalizedPath,
"normalized_transcript_source": normalizedSource,
"trimmed_output_path": trimmedPath,
"run_trimmed_output_path": trimmedPath,
"trimmed_output_path": canonicalTrimmedPath,
}
if !enabled {
@@ -91,14 +100,17 @@ func (trimStage) Run(ctx context.Context, env *Env, m *manifest.Manifest) (*Stag
if err := validateProcessedTranscriptOutput(trimmedPath); err != nil {
return nil, fmt.Errorf("trim: copied trimmed transcript %q invalid: %w", trimmedPath, err)
}
promotedTrimmed, err := promoteRunLocalOutput(env.ArtifactStore, trimmedPath, canonicalTrimmedPath, artifacts.Ref{
Kind: "transcript_trimmed",
Category: "transcripts",
SessionID: sessionID,
})
if err != nil {
return nil, fmt.Errorf("trim: promote trimmed transcript: %w", err)
}
metadata["trim_action"] = "copy_disabled"
return &StageResult{
Outputs: []artifacts.Ref{{
Kind: "transcript_trimmed",
Category: "transcripts",
SessionID: sessionID,
AbsolutePath: trimmedPath,
}},
Outputs: []artifacts.Ref{promotedTrimmed},
Metadata: metadata,
}, nil
}
@@ -114,13 +126,22 @@ func (trimStage) Run(ctx context.Context, env *Env, m *manifest.Manifest) (*Stag
}
boundsCfg := trimCfg.Bounds
boundsOutputPath, err := resolveScriptoriumOutputPath(paths, boundsCfg.OutputPath)
canonicalBoundsOutputPath, err := resolveScriptoriumOutputPath(paths, boundsCfg.OutputPath)
if err != nil {
return nil, fmt.Errorf("trim: resolve bounds output path: %w", err)
}
boundsOutputPath, err := runLocalPathForCanonical(runLayout, paths, canonicalBoundsOutputPath)
if err != nil {
return nil, fmt.Errorf("trim: resolve run-local bounds output path: %w", err)
}
boundsStdoutLogPath := filepath.Join(paths.LogsDir, "scriptorium.bounds.stdout.log")
boundsStderrLogPath := filepath.Join(paths.LogsDir, "scriptorium.bounds.stderr.log")
boundsGeneratedConfigPath := filepath.Join(paths.ConfigDir, "scriptorium.bounds.generated.yml")
if runLayout.Enabled {
boundsStdoutLogPath = filepath.Join(runLayout.LogsDir, "scriptorium.bounds.stdout.log")
boundsStderrLogPath = filepath.Join(runLayout.LogsDir, "scriptorium.bounds.stderr.log")
boundsGeneratedConfigPath = filepath.Join(runLayout.ConfigDir, "scriptorium.bounds.generated.yml")
}
boundsTimeout, err := resolveScriptoriumTimeout(env.Config.Pipeline.Scriptorium.Timeout, boundsCfg.Timeout)
if err != nil {
return nil, fmt.Errorf("trim: resolve bounds timeout: %w", err)
@@ -133,7 +154,8 @@ func (trimStage) Run(ctx context.Context, env *Env, m *manifest.Manifest) (*Stag
metadata["bounds_prompt_id"] = boundsCfg.PromptID
metadata["bounds_profile_id"] = boundsCfg.ProfileID
metadata["bounds_output_path"] = boundsOutputPath
metadata["run_bounds_output_path"] = boundsOutputPath
metadata["bounds_output_path"] = canonicalBoundsOutputPath
metadata["bounds_timeout"] = boundsTimeout.String()
metadata["bounds_input_name"] = boundsCfg.TranscriptInputName
metadata["bounds_input_path"] = normalizedPath
@@ -141,13 +163,22 @@ func (trimStage) Run(ctx context.Context, env *Env, m *manifest.Manifest) (*Stag
renderOutputPath := ""
if boundsCfg.RenderDebug {
renderOutputPath, err = resolveScriptoriumOutputPath(paths, boundsCfg.RenderOutputPath)
canonicalRenderOutputPath, err := resolveScriptoriumOutputPath(paths, boundsCfg.RenderOutputPath)
if err != nil {
return nil, fmt.Errorf("trim: resolve bounds render output path: %w", err)
}
renderOutputPath, err = runLocalPathForCanonical(runLayout, paths, canonicalRenderOutputPath)
if err != nil {
return nil, fmt.Errorf("trim: resolve run-local bounds render output path: %w", err)
}
renderStdoutLogPath := filepath.Join(paths.LogsDir, "scriptorium.bounds.render.stdout.log")
renderStderrLogPath := filepath.Join(paths.LogsDir, "scriptorium.bounds.render.stderr.log")
renderGeneratedConfigPath := filepath.Join(paths.ConfigDir, "scriptorium.bounds.render.generated.yml")
if runLayout.Enabled {
renderStdoutLogPath = filepath.Join(runLayout.LogsDir, "scriptorium.bounds.render.stdout.log")
renderStderrLogPath = filepath.Join(runLayout.LogsDir, "scriptorium.bounds.render.stderr.log")
renderGeneratedConfigPath = filepath.Join(runLayout.ConfigDir, "scriptorium.bounds.render.generated.yml")
}
renderReq := scriptorium.RenderArtifactRequest{
Binary: env.Config.Pipeline.Scriptorium.Binary,
@@ -254,7 +285,7 @@ func (trimStage) Run(ctx context.Context, env *Env, m *manifest.Manifest) (*Stag
metadata["end_segment_id"] = boundsPayload.EndSegmentID
metadata["warnings"] = boundsPayload.Warnings
metadata["keep_selector"] = keepSelector
metadata["bounds_output_path"] = finalBoundsOutputPath
metadata["run_bounds_output_path"] = finalBoundsOutputPath
metadata["bounds_stdout_log_path"] = boundsStdoutLogPath
metadata["bounds_stderr_log_path"] = boundsStderrLogPath
metadata["bounds_generated_config_path"] = boundsGeneratedConfigPath
@@ -279,6 +310,11 @@ func (trimStage) Run(ctx context.Context, env *Env, m *manifest.Manifest) (*Stag
trimStdoutLogPath := filepath.Join(paths.LogsDir, "seriatim.trim.stdout.log")
trimStderrLogPath := filepath.Join(paths.LogsDir, "seriatim.trim.stderr.log")
trimGeneratedConfigPath := filepath.Join(paths.ConfigDir, "seriatim.trim.generated.yml")
if runLayout.Enabled {
trimStdoutLogPath = filepath.Join(runLayout.LogsDir, "seriatim.trim.stdout.log")
trimStderrLogPath = filepath.Join(runLayout.LogsDir, "seriatim.trim.stderr.log")
trimGeneratedConfigPath = filepath.Join(runLayout.ConfigDir, "seriatim.trim.generated.yml")
}
trimTimeout, err := resolveTrimSeriatimTimeout(env.Config.Pipeline.Seriatim.Timeout)
if err != nil {
return nil, fmt.Errorf("trim: resolve seriatim timeout: %w", err)
@@ -315,23 +351,25 @@ func (trimStage) Run(ctx context.Context, env *Env, m *manifest.Manifest) (*Stag
return nil, fmt.Errorf("trim: trimmed transcript %q invalid: %w", trimmedPath, err)
}
outputs := []artifacts.Ref{
{
Kind: "transcript_trimmed",
Category: "transcripts",
SessionID: sessionID,
AbsolutePath: trimmedPath,
},
{
Kind: "session_bounds",
Category: "artifacts",
SessionID: sessionID,
AbsolutePath: finalBoundsOutputPath,
},
promotedTrimmed, err := promoteRunLocalOutput(env.ArtifactStore, trimmedPath, canonicalTrimmedPath, artifacts.Ref{
Kind: "transcript_trimmed",
Category: "transcripts",
SessionID: sessionID,
})
if err != nil {
return nil, fmt.Errorf("trim: promote trimmed transcript: %w", err)
}
promotedBounds, err := promoteRunLocalOutput(env.ArtifactStore, finalBoundsOutputPath, canonicalBoundsOutputPath, artifacts.Ref{
Kind: "session_bounds",
Category: "artifacts",
SessionID: sessionID,
})
if err != nil {
return nil, fmt.Errorf("trim: promote session bounds: %w", err)
}
return &StageResult{
Outputs: outputs,
Outputs: []artifacts.Ref{promotedTrimmed, promotedBounds},
Logs: logPaths,
GeneratedConfigs: generatedConfigs,
Metadata: metadata,

View File

@@ -18,7 +18,7 @@ import (
func TestTrimStageConsumesNormalizedAndProducesTrimmedTranscript(t *testing.T) {
env, m, scr, ser := setupTrimEnv(t)
paths := env.ArtifactStore.SessionPaths(m.SessionID)
paths := sessionPathsForEnv(env, m.SessionID)
normalized := filepath.Join(paths.TranscriptsDir, "normalized.json")
writeFile(t, normalized, `{"segments":[{"id":10},{"id":868}]}`)
m.MarkStageSucceeded("normalize", time.Now().UTC(), []manifest.ArtifactRecord{
@@ -70,7 +70,7 @@ func TestTrimStageConsumesNormalizedAndProducesTrimmedTranscript(t *testing.T) {
func TestTrimStageUsesConfiguredScriptoriumInputName(t *testing.T) {
env, m, scr, _ := setupTrimEnv(t)
paths := env.ArtifactStore.SessionPaths(m.SessionID)
paths := sessionPathsForEnv(env, m.SessionID)
normalized := filepath.Join(paths.TranscriptsDir, "normalized.json")
writeFile(t, normalized, `{"segments":[{"id":10},{"id":11}]}`)
scr.BoundsBody = `{"trim_action":"trim","start_segment_id":10,"end_segment_id":11}`
@@ -94,7 +94,7 @@ func TestTrimStageUsesConfiguredScriptoriumInputName(t *testing.T) {
func TestTrimStageRecordsLogAndGeneratedConfigRefs(t *testing.T) {
env, m, scr, _ := setupTrimEnv(t)
paths := env.ArtifactStore.SessionPaths(m.SessionID)
paths := sessionPathsForEnv(env, m.SessionID)
writeFile(t, filepath.Join(paths.TranscriptsDir, "normalized.json"), `{"segments":[{"id":1},{"id":2}]}`)
scr.BoundsBody = `{"trim_action":"trim","start_segment_id":1,"end_segment_id":2}`
@@ -114,7 +114,7 @@ func TestTrimStageRecordsLogAndGeneratedConfigRefs(t *testing.T) {
func TestTrimStageRenderDebugDiagnosticsAreNotStageOutputs(t *testing.T) {
env, m, scr, _ := setupTrimEnv(t)
paths := env.ArtifactStore.SessionPaths(m.SessionID)
paths := sessionPathsForEnv(env, m.SessionID)
writeFile(t, filepath.Join(paths.TranscriptsDir, "normalized.json"), `{"segments":[{"id":1},{"id":2}]}`)
scr.BoundsBody = `{"trim_action":"trim","start_segment_id":1,"end_segment_id":2}`
@@ -152,7 +152,7 @@ func TestTrimStageFailsWhenNormalizedTranscriptMissing(t *testing.T) {
func TestTrimStageDoesNotFallBackToProcessedTranscript(t *testing.T) {
env, m, scr, ser := setupTrimEnv(t)
paths := env.ArtifactStore.SessionPaths(m.SessionID)
paths := sessionPathsForEnv(env, m.SessionID)
writeFile(t, filepath.Join(paths.TranscriptsDir, "processed.json"), `{"segments":[{"id":1},{"id":2}]}`)
_, err := (trimStage{}).Run(context.Background(), env, m)
@@ -172,7 +172,7 @@ func TestTrimStageDoesNotFallBackToProcessedTranscript(t *testing.T) {
func TestTrimStageFailsWhenNormalizedTranscriptInvalidJSON(t *testing.T) {
env, m, _, _ := setupTrimEnv(t)
paths := env.ArtifactStore.SessionPaths(m.SessionID)
paths := sessionPathsForEnv(env, m.SessionID)
writeFile(t, filepath.Join(paths.TranscriptsDir, "normalized.json"), "not-json")
_, err := (trimStage{}).Run(context.Background(), env, m)
if err == nil {
@@ -185,7 +185,7 @@ func TestTrimStageFailsWhenNormalizedTranscriptInvalidJSON(t *testing.T) {
func TestTrimStageFailsWhenNormalizedTranscriptMissingSegmentsArray(t *testing.T) {
env, m, _, _ := setupTrimEnv(t)
paths := env.ArtifactStore.SessionPaths(m.SessionID)
paths := sessionPathsForEnv(env, m.SessionID)
writeFile(t, filepath.Join(paths.TranscriptsDir, "normalized.json"), `{"schema":"audita.processed.v1"}`)
_, err := (trimStage{}).Run(context.Background(), env, m)
if err == nil {
@@ -198,7 +198,7 @@ func TestTrimStageFailsWhenNormalizedTranscriptMissingSegmentsArray(t *testing.T
func TestTrimStageFailsWhenBoundsOutputInvalidJSON(t *testing.T) {
env, m, scr, _ := setupTrimEnv(t)
paths := env.ArtifactStore.SessionPaths(m.SessionID)
paths := sessionPathsForEnv(env, m.SessionID)
writeFile(t, filepath.Join(paths.TranscriptsDir, "normalized.json"), `{"segments":[{"id":1},{"id":2}]}`)
scr.BoundsBody = "not-json"
_, err := (trimStage{}).Run(context.Background(), env, m)
@@ -212,7 +212,7 @@ func TestTrimStageFailsWhenBoundsOutputInvalidJSON(t *testing.T) {
func TestTrimStageFailsWhenBoundsRangeIsDescending(t *testing.T) {
env, m, scr, _ := setupTrimEnv(t)
paths := env.ArtifactStore.SessionPaths(m.SessionID)
paths := sessionPathsForEnv(env, m.SessionID)
writeFile(t, filepath.Join(paths.TranscriptsDir, "normalized.json"), `{"segments":[{"id":1},{"id":2}]}`)
scr.BoundsBody = `{"trim_action":"trim","start_segment_id":2,"end_segment_id":1}`
_, err := (trimStage{}).Run(context.Background(), env, m)
@@ -226,7 +226,7 @@ func TestTrimStageFailsWhenBoundsRangeIsDescending(t *testing.T) {
func TestTrimStageFailsWhenBoundsIDsMissingFromTranscript(t *testing.T) {
env, m, scr, _ := setupTrimEnv(t)
paths := env.ArtifactStore.SessionPaths(m.SessionID)
paths := sessionPathsForEnv(env, m.SessionID)
writeFile(t, filepath.Join(paths.TranscriptsDir, "normalized.json"), `{"segments":[{"id":20},{"id":21}]}`)
scr.BoundsBody = `{"trim_action":"trim","start_segment_id":10,"end_segment_id":21}`
_, err := (trimStage{}).Run(context.Background(), env, m)
@@ -240,7 +240,7 @@ func TestTrimStageFailsWhenBoundsIDsMissingFromTranscript(t *testing.T) {
func TestTrimStageFailsWhenScriptoriumAdapterFails(t *testing.T) {
env, m, scr, _ := setupTrimEnv(t)
paths := env.ArtifactStore.SessionPaths(m.SessionID)
paths := sessionPathsForEnv(env, m.SessionID)
writeFile(t, filepath.Join(paths.TranscriptsDir, "normalized.json"), `{"segments":[{"id":1},{"id":2}]}`)
scr.RunErr = errors.New("bounds failed")
_, err := (trimStage{}).Run(context.Background(), env, m)
@@ -254,7 +254,7 @@ func TestTrimStageFailsWhenScriptoriumAdapterFails(t *testing.T) {
func TestTrimStageFailsWhenSeriatimTrimAdapterFails(t *testing.T) {
env, m, scr, ser := setupTrimEnv(t)
paths := env.ArtifactStore.SessionPaths(m.SessionID)
paths := sessionPathsForEnv(env, m.SessionID)
writeFile(t, filepath.Join(paths.TranscriptsDir, "normalized.json"), `{"segments":[{"id":1},{"id":2}]}`)
scr.BoundsBody = `{"trim_action":"trim","start_segment_id":1,"end_segment_id":2}`
ser.TrimErr = errors.New("trim failed")
@@ -269,7 +269,7 @@ func TestTrimStageFailsWhenSeriatimTrimAdapterFails(t *testing.T) {
func TestTrimStageDisabledCopiesNormalizedTranscript(t *testing.T) {
env, m, scr, ser := setupTrimEnv(t)
paths := env.ArtifactStore.SessionPaths(m.SessionID)
paths := sessionPathsForEnv(env, m.SessionID)
normalized := filepath.Join(paths.TranscriptsDir, "normalized.json")
normalizedBody := `{"segments":[{"id":1,"text":"alpha"},{"id":2,"text":"beta"}]}`
writeFile(t, normalized, normalizedBody)
@@ -298,6 +298,39 @@ func TestTrimStageDisabledCopiesNormalizedTranscript(t *testing.T) {
}
}
func TestTrimStageUsesRunLocalPathsAndPromotesCanonical(t *testing.T) {
env, m, scr, ser := setupTrimEnv(t)
env.Config.Session.Campaign = "sample-campaign"
m.Campaign = "sample-campaign"
m.RunID = "20260518T010203Z-abcdef12"
paths := sessionPathsForEnv(env, m.SessionID)
normalized := filepath.Join(paths.TranscriptsDir, "normalized.json")
writeFile(t, normalized, `{"segments":[{"id":10},{"id":11}]}`)
scr.BoundsBody = `{"trim_action":"trim","start_segment_id":10,"end_segment_id":11,"warnings":[]}`
result, err := (trimStage{}).Run(context.Background(), env, m)
if err != nil {
t.Fatalf("trim.Run() error = %v", err)
}
if len(scr.RunRequests) != 1 || len(ser.TrimRequests) != 1 {
t.Fatalf("scriptorium run=%d seriatim trim=%d, want 1/1", len(scr.RunRequests), len(ser.TrimRequests))
}
if !strings.Contains(scr.RunRequests[0].OutputPath, filepath.Join("runs", m.RunID, "trim", "outputs")) {
t.Fatalf("bounds run output path = %q, want run-local path", scr.RunRequests[0].OutputPath)
}
if !strings.Contains(ser.TrimRequests[0].OutputTrimmedPath, filepath.Join("runs", m.RunID, "trim", "outputs")) {
t.Fatalf("trim output path = %q, want run-local path", ser.TrimRequests[0].OutputTrimmedPath)
}
if len(result.Outputs) < 2 {
t.Fatalf("outputs = %#v, want promoted trimmed+bounds outputs", result.Outputs)
}
for _, out := range result.Outputs {
if strings.Contains(out.AbsolutePath, string(filepath.Separator)+"runs"+string(filepath.Separator)) {
t.Fatalf("promoted output path = %q, want canonical session path", out.AbsolutePath)
}
}
}
type boundsScriptoriumRunner struct {
RunRequests []scriptorium.RunArtifactRequest
RenderRequests []scriptorium.RenderArtifactRequest
@@ -430,7 +463,7 @@ func setupTrimEnv(t *testing.T) (*Env, *manifest.Manifest, *boundsScriptoriumRun
}
store := artifacts.NewLocalStore(workspace)
if _, err := store.EnsureLayout("2026-05-03"); err != nil {
if _, err := store.EnsureLayoutFor("sample-campaign", "2026-05-03"); err != nil {
t.Fatalf("EnsureLayout() error = %v", err)
}

View File

@@ -1,31 +0,0 @@
# Narratio UX Evaluation Report
## 1. Executive Summary
Narratio has a functional core pipeline with robust S3 integration for input and output, but it currently falls short of the intended "minimalist" operator UX. The primary gaps are the lack of session configuration discovery, the absence of session template support (and the `--session-id` flag), and the missing local cleanup logic. While the pipeline runs successfully, the operator must currently provide explicit session file paths for every run.
## 2. Feature Matrix
| Feature | Status | Evidence | Tests | Documentation | Notes |
| :--- | :--- | :--- | :--- | :--- | :--- |
| **Pipeline Config Discovery** | Implemented | `internal/app/pipeline_config_path_test.go` | Yes | Accurate | Checks `/usr/local/etc` and `/etc`. |
| **Session Config Discovery** | Missing | `internal/app/run.go:30` | N/A | Stale | `--session` is mandatory. |
| **Session Templates** | Missing | `internal/config/load.go` | N/A | Missing | No variable interpolation in `session.yml`. |
| **`--session-id` CLI Flag** | Missing | `cmd/narratio` | N/A | Missing | Not implemented in CLI. |
| **Minimal Seriatim Config** | Implemented | `internal/config/load.go` | Yes | Accurate | Defaults for timeout/schema provided. |
| **Minimal Audita Config** | Implemented | `internal/config/load.go` | Yes | Accurate | Defaults for base_url/model provided. |
| **S3 Audio Input** | Implemented | `internal/stage/prepare.go` | Yes | Accurate | Supports `.flac` downloads from S3. |
| **S3 Archive & Promotion** | Implemented | `internal/stage/archive.go` | Yes | Accurate | Correct paths and commit markers. |
| **Local Cleanup** | Missing | `architecture.md:136` | No | Stale | Config exists, logic is not implemented. |
## 3. Current Happy Path
The shortest command that works today is:
`narratio run --session <path_to_session.yml>`
*(Assuming `pipeline.yml` is present in `/etc/narratio/` or `/usr/local/etc/narratio/`)*.
## 4. Gaps to Intended UX
1. **Session Discovery & Templates (High):** The requirement to pass `--session` and the inability to use `--session-id` with a template is the largest friction point for operators.
2. **Local Cleanup (Medium):** Spool and work directories are not cleaned up after successful archival, leading to local disk growth.
3. **Local Pipeline Config (Low):** Narratio does not check `./pipeline.yml`, requiring users to use `--config` or move files to system directories.
## 5. Recommended Next Implementation Prompt
"Implement session configuration discovery and template support. Specifically: 1) Add a search order for `session.yml` (e.g., `./session.yml`, `/etc/narratio/session.yml`) if `--session` is omitted. 2) Implement the `--session-id` CLI flag. 3) Add variable interpolation to `session.yml` so that `{{session_id}}` can be replaced by the value from the flag or the discovered session config before YAML decoding."

View File

@@ -1,54 +0,0 @@
## 1. Executive Summary
Narratio is close on S3 input/archive mechanics but not yet close on the intended minimal operator UX.
Core S3 workflow is implemented (prepare S3 audio download, archive run upload, promotions, current pointers), but key UX items are missing: no `--session-id` flag, no session auto-discovery, and no session template variable injection. Cleanup/retention for spool/workdirs after archive is also still future work.
## 2. Feature Matrix
| Feature | Status | Evidence | Tests | Documentation status | Notes |
|---|---|---|---|---|---|
| Pipeline config auto-discovery when `--config` omitted | Implemented | `internal/app/pipeline_config_path.go`, `internal/config/defaults.go` | `internal/app/pipeline_config_path_test.go`, `internal/app/commands_test.go` | Accurate in `README.md`, `architecture.md` | Order: `/usr/local/etc/narratio/pipeline.yml`, then `/etc/narratio/pipeline.yml`; no `./pipeline.yml` default |
| Session config auto-discovery when `--session` omitted | Missing | `--session` required in `internal/app/run.go`, `plan.go`, `resume.go`, `run_stage.go` | Covered by missing-flag tests in `internal/app/commands_test.go` | Accurate (docs do not claim auto-discovery) | No precedence order exists for session file search |
| Session template variables in `session.yml` | Missing | Strict decode path in `internal/config/load.go` + strict YAML behavior | No template tests found | Not documented as implemented | No render-before-decode templating mechanism found |
| `--session-id` CLI injection | Missing | No `--session-id` flag in command parsers (`run/plan/resume/run-stage`) | No tests for `--session-id` | Not documented as implemented | Intended minimal UX command not currently supported |
| Campaign/run-aware work+spool paths | Implemented | `internal/artifacts/paths.go`, usage in prepare/archive | Path/helper tests in `internal/artifacts` + stage tests | Documented in README/architecture/roadmap | Layout includes `{campaign}/{session_id}/{run_id}` |
| Run ID generation format | Implemented | `internal/artifacts/run_id.go` | Run ID tests in `internal/artifacts` | Documented | UTC timestamp + random suffix format present |
| Storage backend abstraction | Implemented | `internal/adapters/storage/object_store.go` | Storage backend tests in `internal/adapters/storage` | Documented in README/architecture | Narrow interface (`List/Download/Upload/Exists`) |
| S3 backend + fake backend | Implemented | `internal/adapters/storage/s3_backend.go`, `fake.go` | Adapter tests pass without live S3 | Documented | No AWS creds in config schema/examples |
| Prepare S3 audio input (`inputs.audio_s3`) | Implemented | `internal/stage/prepare.go` | `internal/stage/prepare_test.go` | Documented in `docs/s3-audio-input.md`, README, architecture | Lists prefix, filters `.flac`, downloads/materializes, fails on none |
| Local audio workflow | Implemented | Prepare logic still supports `audio_dir`/`audio_files` | Prepare tests cover local behavior and conflict with `audio_s3` | Documented | Local+S3 conflict is enforced |
| Manifest provenance for S3 audio | Implemented | S3 source metadata assignment in prepare stage | Covered by S3 prepare tests | Documented | ETag recorded as metadata, not checksum |
| Archive run upload under `runs/{run_id}` | Implemented | `internal/stage/archive.go` | `internal/stage/archive_test.go` | Documented in `docs/archive-storage.md`, README, architecture | Successful/completed runs only |
| Archive promotion rules | Implemented | Archive stage promotion handling | Archive tests cover required/optional/mapping behavior | Documented | Default promoted outputs: `transcripts/trimmed.json`, `artifacts/session_recap.md` |
| `current/manifest.json` + `current/run_id.txt` last | Implemented | Archive stage upload order logic | Archive tests verify ordering and pointer content | Documented | `current/run_id.txt` is commit marker; written last |
| Avoid upload of failed/incomplete runs | Implemented | Archive prerequisite checks | Archive tests cover prerequisite failure path | Documented | Failed runs stay local |
| Spool/workdir cleanup after successful archive | Missing | `spool.delete_audio_after_archive` exists but no cleanup behavior in stages/app | No cleanup behavior tests found | Docs accurately call cleanup future work | Gap vs intended UX item 12 |
| Minimal Seriatim config | Partial | Validation requires `seriatim.binary`; defaults fill timeout/schema/gap | Config load/validate tests | Docs mostly accurate | “Binary-only” works after defaults, but still validated post-defaults |
| Minimal Audita config | Partial | Validation requires `audita.binary` and `audita.model`; defaults for timeout/base_url/etc in loader | Config tests in `internal/config` | Docs currently list `timeout`/`base_url` as required in README section | UX expectation “binary + llm_api_key_env only” does not hold because model is required |
## 3. Current Happy Path
Shortest realistic command today is:
`narratio run --session /path/to/session.yml`
That works only if pipeline config is discoverable at `/usr/local/etc/narratio/pipeline.yml` or `/etc/narratio/pipeline.yml`.
Otherwise minimum is:
`narratio run --config /path/to/pipeline.yml --session /path/to/session.yml`
`narratio run --session-id 2026-04-04` does not work today (flag not implemented).
## 4. Gaps to Intended UX
1. Missing `--session-id` flow with session template injection (largest UX gap).
2. No session config auto-discovery order when `--session` is omitted.
3. No session template rendering engine / unresolved-variable handling.
4. Cleanup policy not implemented (`spool.delete_audio_after_archive` is modeled only).
5. Audita minimal config UX still stricter than intended (model required).
6. Optional doc refinement: explicitly call out that `./pipeline.yml` is not in current default search order.
## 5. Recommended Next Implementation Prompt
Implement session template and `--session-id` UX only:
> Add session discovery and template rendering support so `narratio run --session-id <id>` works with no `--session` in normal setups.
> Requirements: define deterministic session discovery order; support rendering template variables in `session.yml` before strict YAML decode; inject CLI `--session-id` into template variables; fail clearly on unresolved variables; preserve strict field validation after render; keep existing `--session` explicit path behavior; add tests for discovery precedence, render success/failure, and CLI integration; update README/architecture/examples accordingly; do not change archive/prepare storage behavior.
Validation note: `go test ./...` passes for the inspected state.