36 Commits

Author SHA1 Message Date
dd03c09d75 Fixed a bug in the S3 credential loading for the restore command
All checks were successful
ci/woodpecker/tag/release Pipeline was successful
2026-05-19 22:51:26 -05:00
5bc8e8683f Documentation update for the restore subcommand 2026-05-19 22:32:55 -05:00
648001a8fe Add workflow integration tests for the restore command 2026-05-19 22:23:25 -05:00
6684774f52 Add restore report and operator summary 2026-05-19 22:15:40 -05:00
f3b63bd5e5 Implement restore execution for the restore subcommand 2026-05-19 22:06:48 -05:00
23d6470b0f Implement restore planning for the restore subcommand 2026-05-19 21:58:38 -05:00
128449040f Implement remote current-state discovery for the restore subcommand 2026-05-19 21:49:19 -05:00
02ab106ade Implement initial CLI command for narratio restore, and extract shared helper functions from the run stages 2026-05-19 21:39:13 -05:00
c128970f58 Updated documentation to remove the completed runtime artifacts roadmap and add a new restore subcommand roadmap 2026-05-19 21:21:06 -05:00
d001baa660 Use artifact source IDs for archive promotion 2026-05-19 20:05:24 -05:00
c5c35cd3b4 Moved example configuration from docs/examples/ to top-level examples/
All checks were successful
ci/woodpecker/tag/release Pipeline was successful
2026-05-19 19:46:40 -05:00
574b1cde6c Update documentation for the new analyze stage and artifact registry 2026-05-19 19:42:28 -05:00
ebb21b9201 Removed the legacy built-in session_recap from the analyze stage 2026-05-19 19:26:07 -05:00
958f446387 Add archive stage integration test for the new analyze stage features 2026-05-19 19:12:10 -05:00
86caf4b222 Update analyze-stage metadata and manifest output 2026-05-19 19:07:44 -05:00
e38ed8ba97 Refactor the analyze stage to actually produce the configured artifacts 2026-05-19 18:58:28 -05:00
3e79cf4724 Update artifact resolution so configured artifact IDs are resolved through the runtime catalog 2026-05-19 18:49:27 -05:00
859ae1ae10 Add abstractions for the internal artifact catalog 2026-05-19 18:43:55 -05:00
c63ecbab32 Add new configuration fields and CLI flags for the upcoming analyze stage enhancements 2026-05-19 18:36:30 -05:00
8480b74283 Updated the roadmap for configurable artifact generation 2026-05-19 11:21:28 -05:00
087869f7fa Added a roadmap for new work to support configurable artifacts defined at runtime 2026-05-19 09:26:46 -05:00
2b2a314d65 Move documentation for external integrations into the docs/integrations subfolder 2026-05-19 09:12:40 -05:00
08b0f4edc5 Removed legacy transcript artifact aliases 2026-05-19 09:00:41 -05:00
571a289296 Added new internal documentation 2026-05-19 08:47:45 -05:00
9f80635b42 Updated configuration docs to reflect the minimal pipeline config 2026-05-19 08:15:12 -05:00
11a3e174b6 Set default value for workspace.root and updated config documentation 2026-05-19 07:07:19 -05:00
9c5e5d6dc1 Simplified the reference tables in docs/config.md 2026-05-19 06:51:55 -05:00
c4e87f58c7 Complete documentation rebuild 2026-05-18 22:03:59 -05:00
37daab7857 Bugfix involving nested directory creation
All checks were successful
ci/woodpecker/tag/release Pipeline was successful
2026-05-18 11:52:40 +00:00
2356688cb9 Removed legacy interfaces and old documentation references to the previous on-disk layout 2026-05-18 03:02:22 +00:00
1054b64d9f Implement minimal downstream invalidation after forced upstream reruns 2026-05-18 01:50:51 +00:00
01fb02426c Update the analyze stage to utilize the new artifact package 2026-05-18 01:29:18 +00:00
7dc79e052f Aligned the archive stage with the new work directory layout 2026-05-18 01:13:51 +00:00
cb525c0f72 Implemented run-local stage execution + immediate promotion for core output-producing stages 2026-05-18 00:55:35 +00:00
622677d038 Added run manifest scaffolding and helpers 2026-05-17 21:15:51 +00:00
550288e008 Add campaign-aware workspace path foundation 2026-05-17 20:57:27 +00:00
116 changed files with 12198 additions and 4162 deletions

465
README.md
View File

@@ -1,461 +1,22 @@
# narratio
`narratio` is a Go orchestration application for processing D&D session audio into transcripts and generated artifacts.
Narratio is a Go orchestration application that turns D&D session audio into polished transcripts and generated session artifacts.
## Current Implementation
Implemented now:
- strict config loading/validation (`pipeline.yml` and `session.yml`)
- local workspace/session layout, locking, and manifest persistence
- resumable stage control (`run`, `plan`, `resume`, `run-stage`, `status`)
- real `prepare`, `transcribe`, `merge`, `polish`, `normalize`, `trim`, and `analyze` stages
- real WhisperX, Seriatim, and Audita adapters
- real Scriptorium subprocess adapter
- optional Scriptorium render diagnostics (`render_debug`)
Not implemented yet:
- `notify` stage behavior
- additional analyze artifacts beyond `session_recap`
- generic DAG orchestration
## Config Files
Narratio expects two YAML files:
- `pipeline.yml`: pipeline/workspace settings
- `session.yml`: per-session settings
Pipeline config lookup for CLI commands:
- if `--config <path>` is provided, Narratio uses that path
- if `--config` is omitted, Narratio searches in this order:
- `/usr/local/etc/narratio/pipeline.yml`
- `/etc/narratio/pipeline.yml`
Session config lookup for CLI commands:
- if `--session <path>` is provided, Narratio uses that path
- if `--session` is omitted, Narratio searches in this order:
- `./session.yml`
- `/usr/local/etc/narratio/session.yml`
- `/etc/narratio/session.yml`
Session template support:
- Narratio renders `session.yml` templates before strict YAML decode.
- `--session-id <value>` provides the `session_id` template variable.
- Supported placeholder forms:
- `{{session_id}}`
- `{{ session_id }}`
- unresolved template placeholders fail with a clear error.
- strict YAML validation still runs after rendering.
- concrete `session.yml` files without templates remain fully supported.
Optional secrets-from-files config:
- `pipeline.secrets.env_dir` may point to a directory of secret files
- each top-level file with an env-var-style name is loaded as an environment variable:
- file name = env var name
- file contents = env var value (trailing newline/CRLF trimmed)
- process environment wins: existing env vars are not overwritten
- if configured, Narratio fails fast when `env_dir` is missing/unreadable
- relative `env_dir` values resolve from Narratios current working directory
YAML decoding is strict (`KnownFields(true)`), so unknown fields fail fast.
Maintainer note: application defaults are centralized in [`internal/config/defaults.go`](internal/config/defaults.go).
## Storage And Archive Foundations
Narratio now includes configuration and path-model foundations for archive support, plus implemented prepare-stage S3 audio input.
Implemented foundations:
- `pipeline.storage.s3` config shape (`bucket`, `root_prefix`, `region`, `endpoint`, `force_path_style`, `access_key_id_env`, `secret_access_key_env`)
- `pipeline.spool` config shape (`root`, `delete_audio_after_archive`)
- `pipeline.archive` config shape (`enabled`, `upload_run`, `promote_artifacts`)
- promotion-rule validation (`from`/`to` required, relative-only paths, traversal rejected)
- `session.campaign` requirement for campaign-aware path construction
- optional `session.inputs.audio_s3.prefix` modeling and prepare-stage S3 audio download
- run ID generation and S3/local path helper foundations
- manifest run/path identity fields
Current defaults:
- `pipeline.storage.s3.root_prefix`: `dnd`
- `pipeline.storage.s3.access_key_id_env`: `OBJECT_STORAGE_KEY_ID`
- `pipeline.storage.s3.secret_access_key_env`: `OBJECT_STORAGE_KEY`
- `pipeline.workspace.cleanup_after_archive`: `false`
- `pipeline.spool.root`: `/var/spool/narratio`
- `pipeline.spool.delete_audio_after_archive`: `false`
- `pipeline.archive.enabled`: `true`
- `pipeline.archive.upload_run`: `true`
- default `pipeline.archive.promote_artifacts`:
- `transcripts/trimmed.json` -> `transcripts/trimmed.json` (`required: true`)
- `artifacts/session_recap.md` -> `artifacts/session_recap.md` (`required: true`)
Current boundaries:
- local development audio (`audio_dir` / `audio_files`) still works
- `audio_dir`/`audio_files` and `audio_s3` are mutually exclusive
- real S3-compatible backend now exists in the storage adapter package
- storage backend tests use fake storage and do not require live S3
- archive uploads successful run records under `runs/{run_id}/`
- archive does not upload local audio by default
- archive uploads promoted outputs to session-level keys using `archive.promote_artifacts`
- archive uploads `current/manifest.json`
- archive uploads `current/run_id.txt` last as the effective commit marker
- required missing promotions fail archive
- optional missing promotions are skipped and recorded
- cleanup remains conservative and opt-in:
- `pipeline.spool.delete_audio_after_archive: true` removes only the run-scoped spool audio directory after successful archive commit
- `pipeline.workspace.cleanup_after_archive: true` removes only the run-scoped local workdir after successful archive commit
- cleanup executes only after all selected stages for the command invocation succeed
- cleanup does not run for failed, incomplete, skipped, or unarchived runs
- local development `audio_dir`/`audio_files` source inputs are never deleted by spool cleanup
- S3 credentials are resolved from configured env-var names when both are present; if either is missing, Narratio falls back to the AWS SDK default credential chain
S3 input details and current boundaries are documented in [docs/s3-audio-input.md](docs/s3-audio-input.md).
## Remote Storage Backend
Narratio includes an object-store backend layer for future prepare/archive work:
- `List(ctx, prefix)`
- `Download(ctx, key, localPath)`
- `Upload(ctx, localPath, key, opts)`
- `Exists(ctx, key)`
Implemented backends:
- fake storage backend for deterministic tests
- S3-compatible backend built from `pipeline.storage.s3`
Key invariant:
- callers pass full bucket-relative object keys
- storage backends do not prepend `root_prefix` and do not infer session/campaign paths
Current boundary:
- `prepare` uses `List` + `Download` through the backend when `session.inputs.audio_s3` is configured
- `archive` uses `Upload` through the backend for successful run-record uploads
- `archive` also uses `Upload` for promotion writes and current pointers
- no failed or incomplete runs are uploaded
- local audio is not re-uploaded by default
Archive run-upload details and boundaries are documented in [docs/archive-storage.md](docs/archive-storage.md).
## Canonical Stage Order
1. `prepare`
2. `transcribe`
3. `merge`
4. `polish`
5. `normalize`
6. `trim`
7. `analyze`
8. `archive`
9. `notify`
## Transcript Tiers
- `transcripts/merged.json`: canonical deterministic merged transcript from Seriatim merge
- `transcripts/processed.json`: full raw Audita-polished transcript output
- `transcripts/normalized.json`: Seriatim-normalized transcript from the normalize stage
- `transcripts/trimmed.json`: gameplay-only normalized polished transcript from trim stage
## Seriatim Configuration
`pipeline.seriatim` configures the Seriatim subprocess adapter used by `merge`, `normalize`, and `trim`.
Minimal behavior:
- `pipeline.seriatim` may be omitted entirely.
- when omitted, Narratio defaults to:
- `binary: seriatim`
- `timeout: 10m`
- `output_schema: seriatim-intermediate`
- `coalesce_gap: 3.0`
- `report: true`
Optional overrides in `pipeline.seriatim` continue to work, including explicit binary paths and advanced `env` tuning values.
## Audita Configuration
`pipeline.audita` configures the real Audita subprocess adapter used by `polish`.
Minimal behavior:
- `pipeline.audita` may be omitted entirely.
- when omitted, Narratio defaults to:
- `binary: audita`
- `timeout: 3h`
- `report: true`
Optional:
- `llm_api_key_env` (when set, Narratio requires that env var and passes it to Audita as `AUDITA_LLM_API_KEY`)
- `modules` override list (when empty/omitted, Narratio does not pass `--modules`)
- `base_url` (when omitted, Narratio does not pass `--base-url`; Audita runtime defaults/config may apply)
- `model` (when omitted, Narratio does not pass `--model`; Audita runtime defaults/config may apply)
- `transcript_description`
- `config_path`
- `output_schema` (`bare-segments` or `audita-v1`)
- `work_dir_retention` (`always`, `auto`, or `never`)
- `total_llm_concurrency` (> 0 when provided)
- `proposal_llm_concurrency` (> 0 when provided)
- `validation_model`
- `validation_llm_concurrency` (> 0 when provided)
- `report` (defaults to `true`)
Narratio passes only configured optional Audita flags. Omitted optional values are left to Audita runtime defaults/config.
## Normalize Configuration
`pipeline.normalize` is optional. When omitted, Narratio defaults to:
- `output_path: transcripts/normalized.json`
- `output_schema: seriatim-intermediate`
- `report: true`
Allowed `normalize.output_schema` values:
- `seriatim-minimal`
- `seriatim-intermediate`
- `seriatim-full`
`normalize.output_path` is treated as session-workdir-relative when not absolute.
Normalize stage behavior summary:
- normalize runs after `polish` and before `trim`
- normalize resolves `transcripts/processed.json`
- normalize runs Seriatim `normalize` to produce `transcripts/normalized.json`
- normalize diagnostics are written to:
- `artifacts/seriatim.normalize.report.json` (when enabled)
- `logs/seriatim.normalize.stdout.log`
- `logs/seriatim.normalize.stderr.log`
- `config/seriatim.normalize.generated.yml`
## Trim Configuration
`pipeline.trim` is optional. If omitted, no trim config is loaded. If `trim.enabled` is omitted, it defaults to `false`.
When `trim.enabled: true`:
- `trim.output_path` is required
- `trim.bounds.prompt_id` is required
- `trim.bounds.transcript_input_name` is required
- `trim.bounds.output_path` is required
- `trim.bounds.timeout` must be a valid Go duration when provided
- `trim.bounds.render_debug: true` requires `trim.bounds.render_output_path`
- `trim.bounds.profile_id` may be empty to use the prompt default profile
Trim paths are treated as session-workdir-relative when not absolute.
Example trim config:
```yaml
trim:
enabled: true
output_path: "transcripts/trimmed.json"
bounds:
prompt_id: "dnd_session.bounds"
profile_id: ""
transcript_input_name: "transcript"
output_path: "artifacts/session_bounds.json"
timeout: "10m"
render_debug: false
render_output_path: "artifacts/session_bounds.render.json"
seriatim:
report: false
```
Trim behavior summary:
- trim discovers and validates `transcripts/normalized.json`
- trim uses Scriptorium bounds (`dnd_session.bounds` by example config) to produce `artifacts/session_bounds.json`
- bounds IDs are validated against the same normalized transcript ID space that Seriatim trim will consume
- trim converts bounds to Seriatim keep selector (for example `10-868`) and runs Seriatim trim
- if trim is disabled, Narratio copies normalized transcript to trimmed transcript and records `trim_action=copy_disabled`
Trim outputs and diagnostics:
- `artifacts/session_bounds.json`
- `transcripts/trimmed.json`
- `logs/scriptorium.bounds.stdout.log`
- `logs/scriptorium.bounds.stderr.log`
- `config/scriptorium.bounds.generated.yml`
- `logs/seriatim.trim.stdout.log`
- `logs/seriatim.trim.stderr.log`
- `config/seriatim.trim.generated.yml`
- optional bounds render-debug outputs:
- `artifacts/session_bounds.render.json`
- `logs/scriptorium.bounds.render.stdout.log`
- `logs/scriptorium.bounds.render.stderr.log`
- `config/scriptorium.bounds.render.generated.yml`
Render-debug files are diagnostics and are not treated as canonical stage output artifact refs.
## Scriptorium Configuration
`pipeline.scriptorium` is optional. When present, Narratio validates and uses it for analyze-stage artifact generation.
Key points:
- `scriptorium.binary` defaults to `scriptorium` when section is present
- `scriptorium.config_path` is optional
- `scriptorium.timeout` defaults to `10m` when omitted
- `scriptorium.render_debug` enables render diagnostics globally
- artifacts are configured under `scriptorium.artifacts` (map shape supports multiple artifacts)
- enabled artifacts require `prompt_id` and `output_path`
- artifact `render_debug` may override global render setting
- `vars` currently support boolean and string values
Example `session_recap` artifact definition:
```yaml
scriptorium:
binary: "scriptorium"
config_path: "/etc/scriptorium/config.yml"
timeout: "10m"
render_debug: false
artifacts:
session_recap:
enabled: true
prompt_id: "dnd.session_recap"
profile_id: "local-quality" # optional
output_path: "artifacts/session_recap.md"
timeout: "10m"
# render_debug: true # optional per-artifact override
inputs:
transcript:
source: "trimmed_transcript"
required: true
previous_recap:
source: "previous_session_artifact"
artifact: "session_recap"
path: "" # optional; set when available
required: false
vars:
session_id: true
session_date: true
campaign_name: true
previous_session_id: true
output_kind: "session_recap"
```
Prompt IDs and profile IDs are configuration values. They are not hardcoded in analyze-stage logic.
Do not put secrets in `pipeline.yml`. If API-key behavior is configured, use env var names only.
If `pipeline.secrets.env_dir` is configured, keep only references and secret files there; secret values are still not written to manifests, generated configs, or Narratio-managed logs.
## Scriptorium Runtime Behavior
Narratio integrates with Scriptorium through the public CLI subprocess contract:
- generation: `scriptorium run`
- diagnostics/testing: `scriptorium render --format json` when `render_debug` is enabled
For the initial implementation, only `session_recap` generation is supported.
Analyze-stage session recap behavior:
- available transcript input sources for configured artifacts: `processed_transcript`, `normalized_transcript`, `trimmed_transcript`
- session recap should use gameplay-only transcript input (`source: trimmed_transcript`)
- Narratio resolves `trimmed_transcript` from trim manifest output (`transcript_trimmed`) or fallback `transcripts/trimmed.json`
- Narratio resolves `normalized_transcript` from normalize manifest output (`transcript_normalized`) or fallback `transcripts/normalized.json`
- missing trimmed transcript fails clearly and advises running trim stage first
- `normalized_transcript` is the preferred full-transcript source for future table/meta-analysis artifacts
- `processed_transcript` remains supported for advanced/debug use cases
- optionally includes `previous_recap` when configured and resolvable
- omits optional previous recap when unavailable
- fails if required inputs are missing
- validates output file exists and is non-empty
Expected session output paths:
- `artifacts/session_recap.md`
- `logs/scriptorium.session_recap.stdout.log`
- `logs/scriptorium.session_recap.stderr.log`
- `config/scriptorium.session_recap.generated.yml`
- `artifacts/session_recap.render.json` when render diagnostics are enabled
## Examples
Starter files:
- `examples/pipeline.minimal.yml`
- `examples/pipeline.audita-overrides.yml`
- `examples/session.minimal.yml`
- `examples/session.template.yml`
- `examples/speakers.yml`
## Commands
Run tests:
It coordinates transcription, merge/polish/normalize/trim processing, artifact generation, archive publishing, and resumable run state in one operator workflow.
```bash
go test ./...
narratio run --session-id 2026-04-04
```
Plan a run:
This command requires discoverable `pipeline.yml` and `session.yml` files (or explicit `--config` and `--session` flags).
```bash
go run ./cmd/narratio plan --session examples/session.minimal.yml
```
## Documentation
Use `--config <path>` to override default pipeline lookup when needed.
Run with a discoverable session template:
```bash
go run ./cmd/narratio run --session-id 2026-04-04
```
Run full pipeline:
```bash
go run ./cmd/narratio run --config examples/pipeline.minimal.yml --session examples/session.minimal.yml
```
Run analyze only:
```bash
go run ./cmd/narratio run-stage --config examples/pipeline.minimal.yml --session examples/session.minimal.yml analyze
```
Resume with a template session ID:
```bash
go run ./cmd/narratio resume --config examples/pipeline.minimal.yml --session examples/session.template.yml --session-id 2026-04-04
```
## Operational Note
Checksum-based stale detection is not implemented yet.
If prepared inputs or prompt/runtime config change, rerun the appropriate upstream stages before relying on downstream artifacts.
Examples:
- glossary/autocorrect/speaker-context changes: rerun at least `merge`, `polish`, `normalize`, `trim`, and `analyze`
- trim bounds prompt/profile/config changes: rerun at least `normalize`, `trim`, and `analyze`
- session recap prompt/profile/input-source changes: rerun `analyze`
## Roadmap
Near-term roadmap:
- extend analyze to additional configured artifacts
- support workflows where later artifacts consume earlier generated artifacts
- keep orchestration explicit without a generic DAG engine
- implement archive and notify backends
- [Configuration](docs/config.md)
- [CLI Reference](docs/cli.md)
- [Operations and Recovery](docs/operations.md)
- [Troubleshooting](docs/troubleshooting.md)
- [Development Guide](docs/development.md)
- [Architecture Principles](docs/architecture.md)
- [Internal Component Contracts](docs/internal/README.md)
- [Config Examples](examples/)

View File

@@ -1,502 +0,0 @@
# Narratio Architecture
## 1. Purpose
`narratio` is a Go orchestrator for D&D session processing. It runs a stage-based local pipeline from audio input through transcript processing and artifact generation, with manifest-based skip/force/resume behavior.
Narratio integrates with Scriptorium through the **public CLI** (`scriptorium run` and `scriptorium render`) via synchronous subprocess execution.
## 2. Current Status
Implemented:
- strict `pipeline.yml` + `session.yml` loading with strict YAML field checking (`KnownFields(true)`)
- local workspace/session layout, lock file handling, artifact path helpers, checksums, and atomic writes
- manifest store and stage status transitions for resumable runs
- real `prepare`, `transcribe`, `merge`, and `polish` stages
- real WhisperX HTTP adapter
- real Seriatim subprocess adapter
- real Audita subprocess adapter
- real Scriptorium subprocess adapter
- real `normalize` stage producing `transcripts/normalized.json`
- real `trim` stage producing `transcripts/trimmed.json`
- real `analyze` stage for initial `session_recap` generation
- optional Scriptorium render diagnostics (`render_debug`) before production run
- storage/archive configuration and validation foundations for:
- `pipeline.storage.s3`
- `pipeline.spool`
- `pipeline.archive` promotion rules
- `session.inputs.audio_s3`
- run identity and path-model foundations:
- run ID generation (`YYYYMMDDTHHMMSSZ-xxxxxxxx`)
- S3 session/run/current key builders
- campaign/session/run local work/spool path helpers
- manifest run/path identity fields (`campaign`, `run_id`, local and S3 prefixes)
- remote storage backend layer:
- narrow object-store interface (`List`, `Download`, `Upload`, `Exists`)
- fake storage backend for deterministic tests (no network dependency)
- S3-compatible backend using AWS SDK v2
- config-based object-store construction helper
Still placeholder/future:
- `notify` stage behavior
- additional Scriptorium artifact types beyond `session_recap`
- artifact-to-artifact workflows beyond the initial single-artifact implementation
- generic stale detection based on input/config checksums
## 3. Pipeline and Stage Boundaries
Canonical stage order:
1. `prepare`
2. `transcribe`
3. `merge`
4. `polish`
5. `normalize`
6. `trim`
7. `analyze`
8. `archive`
9. `notify`
Boundary rules:
- orchestration logic lives in `internal/app`
- stage business logic lives in `internal/stage`
- external-tool CLI construction lives in adapter packages
- Scriptorium CLI details stay in `internal/adapters/scriptorium`
## 4. Scriptorium Integration Model
Integration mode:
- public CLI subprocesses only (no Scriptorium internal Go packages, no HTTP API)
- production generation uses `scriptorium run`
- diagnostics/testing render uses `scriptorium render --format json`
Run invocation shape used by adapter:
```bash
scriptorium run --prompt <prompt_id> --input name=path --out <output_path>
```
Optional flags passed when configured:
- `--config <path>`
- `--profile <profile_id>`
- repeated `--var name=value`
- repeated `--input name=path`
- `--timeout <duration>`
- `--api-key-env <ENV_NAME>` when configured
Render invocation shape used by adapter:
```bash
scriptorium render --prompt <prompt_id> --input name=path --format json --out <render_output_path>
```
Adapter behavior:
- always passes `--out`
- captures stdout/stderr separately
- writes generated invocation metadata YAML (redacted, no secrets)
- treats exit code `0` as success
- treats exit code `1` as failure
- treats exit code `2` as failure with `validation_failed=true` and preserves output metadata when available
- validates successful output files exist and are non-empty
- does not treat non-empty stderr as failure by itself
## 5. Configuration Contract
CLI pipeline config path resolution:
- when `--config <path>` is provided, that path is used
- when `--config` is omitted, Narratio searches defaults in order:
- `/usr/local/etc/narratio/pipeline.yml`
- `/etc/narratio/pipeline.yml`
- default values are centralized in `internal/config/defaults.go`
CLI session config path resolution:
- when `--session <path>` is provided, that path is used
- when `--session` is omitted, Narratio searches defaults in order:
- `./session.yml`
- `/usr/local/etc/narratio/session.yml`
- `/etc/narratio/session.yml`
Session template rendering:
- session templates are rendered before strict YAML decode
- `--session-id <value>` provides the `session_id` template variable
- supported placeholders:
- `{{session_id}}`
- `{{ session_id }}`
- unresolved placeholders fail clearly
- strict `KnownFields(true)` YAML validation still applies after rendering
- if rendered `session.session_id` conflicts with `--session-id`, load fails clearly
Optional pipeline secrets directory:
- `pipeline.secrets.env_dir` enables loading environment variables from local files before command execution
- file name = env var name; file contents = env var value (trailing newline/CRLF trimmed)
- only env-var-style file names are considered; other entries are ignored
- existing process environment values are preserved (not overwritten)
- if configured, unreadable/missing `env_dir` fails command execution early
- relative `env_dir` values are resolved from current working directory
Storage and archive foundations:
- `pipeline.storage.s3` is available for modeling S3 coordinates:
- `bucket`
- `root_prefix` (default `dnd`)
- `region`
- `endpoint`
- `force_path_style` (default `false`)
- `access_key_id_env` (default `OBJECT_STORAGE_KEY_ID`)
- `secret_access_key_env` (default `OBJECT_STORAGE_KEY`)
- `pipeline.spool.root` defaults to `/var/spool/narratio`
- `pipeline.workspace.cleanup_after_archive` defaults to `false`
- `pipeline.spool.delete_audio_after_archive` defaults to `false`
- `pipeline.archive` is optional and defaults to:
- `enabled: true`
- `upload_run: true`
- default `promote_artifacts`:
- `transcripts/trimmed.json`
- `artifacts/session_recap.md`
- archive promotion rules enforce safe relative paths:
- `from` and `to` are required
- absolute paths are rejected
- traversal segments such as `..` are rejected
Session input foundations:
- `session.campaign` is required
- local audio remains supported through `session.inputs.audio_dir` or `session.inputs.audio_files`
- optional S3 audio input shape is `session.inputs.audio_s3.prefix`
- `audio_dir`/`audio_files` and `audio_s3` are mutually exclusive
- when `audio_s3` is configured, `prepare` lists and downloads `.flac` objects through the object-store backend
Cross-config validation scope:
- `pipeline.storage.s3.bucket` is required only when an S3-dependent feature is explicitly configured (for current foundations, that includes `session.inputs.audio_s3`, and archive upload intent when using `storage.backend: s3`)
- no AWS credential values are stored in Narratio config; only env-var names are configured
- when both configured credential env vars resolve to non-empty values, the S3 backend uses them as static credentials
- when either configured credential value is missing, the S3 backend falls back to the AWS SDK default credential chain
Remote object-store backend scope:
- remote storage APIs are isolated to `internal/adapters/storage`
- AWS SDK types remain contained within the S3 backend implementation package
- S3 key/session path semantics remain outside the backend, with this invariant:
- callers pass full bucket-relative object keys
- backend methods do not prepend `root_prefix` or infer campaign/session/run paths
- `prepare` now uses object-store `List` and `Download` for S3 audio input
- `archive` now uses object-store `Upload` for successful run-record upload under the run prefix
- `archive` now uses object-store `Upload` for promoted outputs and current pointers
Prepare S3 audio behavior (implemented):
- compute session prefix as `{root_prefix}/campaigns/{campaign}/sessions/{session_id}/`
- resolve `session.inputs.audio_s3.prefix` under that session prefix
- list objects under the computed audio prefix and filter `.flac` keys
- fail clearly when no `.flac` objects are found
- download selected objects to spool audio path:
- `{spool.root}/{campaign}/{session_id}/{run_id}/audio/`
- materialize audio files into workdir audio path:
- `{workspace.root}/work/{campaign}/{session_id}/{run_id}/audio/`
- record S3 provenance in manifest input records (bucket/key/metadata/local paths/checksum)
- no AWS SDK types are used in stage code; storage implementation details stay in storage adapter packages
Archive publishing behavior (implemented):
- `archive` verifies prerequisite stage success before upload:
- `prepare`, `transcribe`, `merge`, `polish`, `normalize`, `trim`, `analyze`
- only successful/completed runs are uploaded
- uploaded run record destination is:
- `{root_prefix}/campaigns/{campaign}/sessions/{session_id}/runs/{run_id}/`
- uploaded existing local paths include:
- `inputs/`, `transcripts/`, `artifacts/`, optional `reports/`, `config/`, `logs/`, and `manifest.json`
- local `audio/` is intentionally excluded from upload by default
- file upload order is deterministic (sorted relative paths)
- `archive.enabled: false` and `archive.upload_run: false` skip upload cleanly
- stage metadata records non-secret upload context:
- run upload details, promoted output details, current manifest key, current pointer key
- no secrets, transcript contents, prompt contents, or environment dumps
- promotion rules:
- `from` resolves from local workdir
- `to` resolves under session-level S3 root
- missing required source fails archive
- missing optional source is skipped and recorded
- default promoted outputs:
- `transcripts/trimmed.json`
- `artifacts/session_recap.md`
- current pointers:
- `current/manifest.json` uploaded after run upload and promotions
- `current/run_id.txt` uploaded last with `{run_id}\n`
- `current/run_id.txt` is the effective commit marker
- if promotion or current-manifest upload fails, archive returns failure and does not write `current/run_id.txt`
- failed/incomplete runs remain local and are not uploaded
- post-archive local cleanup (implemented, opt-in):
- cleanup runs only after archive succeeded and wrote `current/run_id.txt`
- cleanup is executed after all selected stages in the command invocation succeed (for example, a later `notify` failure leaves local files intact)
- `pipeline.spool.delete_audio_after_archive: true` removes only `{spool.root}/{campaign}/{session_id}/{run_id}/audio/`
- `pipeline.workspace.cleanup_after_archive: true` removes only `{workspace.root}/work/{campaign}/{session_id}/{run_id}/`
- cleanup does not run when archive is skipped/disabled/fails or when run upload is disabled
- local development `audio_dir`/`audio_files` inputs are never removed by spool cleanup
`pipeline.scriptorium` is optional. Existing pipelines without Scriptorium continue to work.
`pipeline.trim` is optional. Existing pipelines without trim config continue to work.
`pipeline.normalize` is optional. Existing pipelines without normalize config continue to work.
`pipeline.audita` drives the real Audita subprocess adapter for the `polish` stage.
Audita defaulted fields:
- `binary` defaults to `audita`
- `timeout` defaults to `3h`
- `report` defaults to `true`
Audita optional fields:
- `llm_api_key_env` (enforced only when configured)
- `modules` override list (when omitted/empty, Narratio does not pass `--modules`)
- `base_url` (when omitted, Narratio does not pass `--base-url`)
- `model` (when omitted, Narratio does not pass `--model`)
- `transcript_description`
- `config_path`
- `output_schema` (`bare-segments` or `audita-v1`)
- `work_dir_retention` (`always`, `auto`, `never`)
- `total_llm_concurrency` (> 0 when provided)
- `proposal_llm_concurrency` (> 0 when provided)
- `validation_model`
- `validation_llm_concurrency` (> 0 when provided)
- `report` override
Narratio passes only configured optional Audita flags; omitted optional values defer to Audita runtime defaults/config.
Seriatim defaults:
- `pipeline.seriatim` may be omitted
- `binary` defaults to `seriatim`
- `timeout` defaults to `10m`
- `output_schema` defaults to `seriatim-intermediate`
- `coalesce_gap` defaults to `3.0`
- `report` defaults to `true`
When `pipeline.normalize` is omitted, defaults are applied:
- `output_path: transcripts/normalized.json`
- `output_schema: seriatim-intermediate`
- `report: true`
When `pipeline.normalize` is present:
- `output_path` must be non-empty
- `output_schema` must be one of `seriatim-minimal`, `seriatim-intermediate`, or `seriatim-full`
- relative `output_path` values are session-workdir-relative paths
- Seriatim binary settings still come from `pipeline.seriatim`
When `pipeline.trim` is present:
- `enabled` is optional and defaults to `false` when omitted
- relative `output_path`, `bounds.output_path`, and `bounds.render_output_path` values are session-workdir-relative paths
- do not store secrets in trim config values
When `pipeline.trim.enabled: true`:
- `output_path` is required and non-empty
- `bounds.prompt_id` is required and non-empty
- `bounds.transcript_input_name` is required and non-empty
- `bounds.output_path` is required and non-empty
- `bounds.timeout` must parse as a Go duration when provided
- `bounds.render_debug: true` requires non-empty `bounds.render_output_path`
- `bounds.profile_id` may be empty to use the prompt default profile
- prompt IDs are config values, not hardcoded stage logic
When `pipeline.scriptorium` is present:
- `binary` defaults to `scriptorium` when omitted
- `config_path` is optional; when provided it must be non-empty
- `timeout` is optional; when provided it must parse as a Go duration
- default `timeout` is `10m`
- unknown YAML fields fail strict decode
Artifacts are configured as a map under `pipeline.scriptorium.artifacts` so multiple artifacts are possible in the config shape.
For each artifact definition:
- `enabled: true` requires non-empty `prompt_id`
- `enabled: true` requires non-empty `output_path`
- `timeout` must parse as Go duration when present
- optional per-artifact `render_debug` may override global `scriptorium.render_debug`
- `inputs` are named and each input requires non-empty `source`
- inputs may be optional (`required: false`)
- `vars` values currently support `string` and `bool`
Prompt IDs and profile IDs are configuration values, not hardcoded stage logic.
Trim config shape:
```yaml
trim:
enabled: true
output_path: "transcripts/trimmed.json"
bounds:
prompt_id: "dnd_session.bounds"
profile_id: ""
transcript_input_name: "transcript"
output_path: "artifacts/session_bounds.json"
timeout: "10m"
render_debug: false
render_output_path: "artifacts/session_bounds.render.json"
seriatim:
report: false
```
## 6. Transcript Tiers
Narratio currently produces and uses four transcript tiers:
- `transcripts/merged.json`: canonical deterministic merged transcript from Seriatim merge
- `transcripts/processed.json`: full raw Audita-polished transcript output (includes pre/post-game content)
- `transcripts/normalized.json`: normalized transcript generated by Seriatim normalize
- `transcripts/trimmed.json`: gameplay-only normalized polished transcript from trim stage
Trim reads `transcripts/normalized.json`, validates bounds IDs against that same transcript ID space, and writes `transcripts/trimmed.json`.
## 7. Normalize Stage (Current Implementation)
Normalize stage behavior:
- stage order position: after `polish` and before `trim`
- discovers processed transcript from manifest polish outputs (`transcript_processed`) when present, else `work/<session_id>/transcripts/processed.json`
- validates processed transcript JSON shape (`segments` array required)
- runs Seriatim `normalize` to produce normalized transcript
- validates normalized transcript JSON shape (`segments` array required)
- validates normalize report JSON when enabled
Expected normalize outputs and diagnostics:
- `transcripts/normalized.json`
- `artifacts/seriatim.normalize.report.json` (when normalize report is enabled)
- `logs/seriatim.normalize.stdout.log`
- `logs/seriatim.normalize.stderr.log`
- `config/seriatim.normalize.generated.yml`
## 8. Trim Stage (Current Implementation)
Trim stage behavior:
- stage order position: after `normalize` and before `analyze`
- discovers normalized transcript from manifest normalize outputs (`transcript_normalized`) when present, else `work/<session_id>/transcripts/normalized.json`
- validates normalized transcript JSON shape (`segments` array required)
- when `trim.enabled: false` (or trim config omitted), deterministically copies normalized transcript to `transcripts/trimmed.json` and records `trim_action=copy_disabled`
- when `trim.enabled: true`:
- runs Scriptorium bounds prompt using configured `trim.bounds.prompt_id`
- writes bounds output to configured path (typically `artifacts/session_bounds.json`)
- parses and validates bounds output against the same normalized transcript being trimmed
- converts bounds range to Seriatim keep selector (for example `10-868`)
- runs Seriatim `trim` to produce `transcripts/trimmed.json`
- supports no-trim bounds actions (`none`/`copy`) by copying normalized transcript unchanged
- validates trimmed transcript JSON shape (`segments` array required)
Expected trim outputs and diagnostics:
- `artifacts/session_bounds.json`
- `transcripts/trimmed.json`
- `logs/scriptorium.bounds.stdout.log`
- `logs/scriptorium.bounds.stderr.log`
- `config/scriptorium.bounds.generated.yml`
- `logs/seriatim.trim.stdout.log`
- `logs/seriatim.trim.stderr.log`
- `config/seriatim.trim.generated.yml`
- optional bounds render-debug outputs when enabled:
- `artifacts/session_bounds.render.json`
- `logs/scriptorium.bounds.render.stdout.log`
- `logs/scriptorium.bounds.render.stderr.log`
- `config/scriptorium.bounds.render.generated.yml`
Render-debug files are diagnostics. They are recorded in stage metadata/log/config refs and are not treated as canonical stage output artifact refs.
## 9. Analyze Stage (Current Implementation)
The current real analyze implementation supports only `scriptorium.artifacts.session_recap`.
Behavior:
- if `pipeline.scriptorium` is missing, analyze returns a skipped result with metadata
- if no Scriptorium artifacts are enabled, analyze returns a skipped result with metadata
- if enabled artifacts exist but `session_recap` is not enabled, analyze fails clearly
- available transcript input sources for configured artifacts: `processed_transcript`, `normalized_transcript`, `trimmed_transcript`
- `session_recap` should use `trimmed_transcript` input (`transcripts/trimmed.json`) for in-universe recap generation
- `trimmed_transcript` input is resolved from manifest (`trim` output kind `transcript_trimmed`) when available, otherwise fallback path `work/<session_id>/transcripts/trimmed.json`
- `normalized_transcript` input is resolved from manifest (`normalize` output kind `transcript_normalized`) when available, otherwise fallback path `work/<session_id>/transcripts/normalized.json`
- `processed_transcript` input is resolved from manifest (`polish` output kind `transcript_processed`) when available, otherwise fallback path `work/<session_id>/transcripts/processed.json`
- `normalized_transcript` is the preferred full-transcript source for future table/meta-analysis artifacts
- `processed_transcript` remains available for advanced/debug use cases
- transcript inputs are validated as JSON with top-level `segments` array
- configured inputs are resolved by source
- optional `previous_recap` is omitted when unavailable
- required `previous_recap` fails before invocation when unavailable
- vars are built from config + session metadata
- `render_debug` controls pre-run `scriptorium render` diagnostics
- render failure stops stage before production run
- render output is validated as JSON
- production call uses Scriptorium adapter `RunArtifact`
- successful run output must exist and be non-empty
- missing `trimmed_transcript` input for configured `trimmed_transcript` source fails clearly with guidance to run trim stage first
- manifest records output refs, logs, generated config paths, and non-secret provenance metadata
## 10. Session Recap Paths
Current expected paths for `session_recap`:
- artifact output: `artifacts/session_recap.md`
- run stdout log: `logs/scriptorium.session_recap.stdout.log`
- run stderr log: `logs/scriptorium.session_recap.stderr.log`
- run generated invocation/config: `config/scriptorium.session_recap.generated.yml`
- render output (when enabled): `artifacts/session_recap.render.json`
- render stdout log: `logs/scriptorium.session_recap.render.stdout.log`
- render stderr log: `logs/scriptorium.session_recap.render.stderr.log`
- render generated invocation/config: `config/scriptorium.session_recap.render.generated.yml`
## 11. Security and Privacy
- do not store secrets in pipeline YAML, generated invocation YAML, logs, or manifest metadata
- if API-key integration is configured, pass env var names only (never raw key values)
- with `pipeline.secrets.env_dir`, secret file values are loaded into process env only and are not persisted in manifest metadata or generated configs
- avoid logging transcript content or rendered prompt content by default
- treat generated artifacts and logs as potentially sensitive session material
## 12. Operational Caveat (Pre-Stale-Detection)
Checksum-based stale detection is not implemented yet.
If prepared inputs or prompt/runtime configuration change (for example glossary files, prompt IDs, profile IDs, or relevant pipeline settings), rerun the appropriate prior stages to refresh downstream artifacts.
Examples:
- glossary or autocorrect changes usually require rerunning at least `merge`, `polish`, `normalize`, `trim`, and `analyze`
- trim prompt/profile changes require rerunning at least `normalize`, `trim`, and `analyze`
- session recap prompt/profile/input-source changes require rerunning `analyze`
## 13. Roadmap
Planned next steps:
- extend analyze beyond `session_recap` to additional configured artifacts
- support artifact inputs that consume prior generated artifacts
- keep this composable without adding a generic DAG engine in the near term
- implement real `archive` backend behavior
- implement real `notify` backend behavior
- add checksum-based stale detection and stale transitions
Architectural invariants remain:
- strict config decoding/validation
- manifest-driven run control
- clear stage/adapter separation
- configuration-driven prompt/profile/input/vars/output mapping
- Scriptorium integration through public CLI subprocess contract

202
docs/architecture.md Normal file
View File

@@ -0,0 +1,202 @@
# Narratio Architecture
## Purpose
`narratio` is a Go orchestration application for processing D&D session audio into polished transcripts and generated session artifacts.
This document defines the development principles for the project. It is inward-facing: its audience is developers and LLM coding agents. It should guide future changes, not serve as a complete implementation reference.
Implemented component details belong under `docs/internal/`.
## Project Shape
Narratio is a modular, stage-driven orchestrator.
It coordinates specialized downstream systems rather than reimplementing their domains:
- WhisperX handles transcription.
- Seriatim handles deterministic transcript merge/normalization/trim behavior.
- Audita handles transcript correction and polishing.
- Scriptorium handles prompt execution and generated artifacts.
Narratio owns orchestration, configuration loading, session/run state, local and remote path modeling, manifest persistence, stage sequencing, resume behavior, and archive semantics.
Narratio should remain explicit and comprehensible. It is not intended to become a generic workflow engine.
## Core Principles
### Modular and composable
Code should be organized around clear responsibilities. Stages, adapters, config loading, manifest persistence, path construction, and storage behavior should remain separable and independently testable.
### Hexagonal boundaries
External systems should be isolated behind narrow adapters. Stage logic should depend on Narratio-level interfaces and data structures, not on external SDK types, subprocess argument construction, or transport-specific details.
### Standard library preference
Prefer the Go standard library. Add dependencies only when they provide substantial value, are necessary for an external integration, or are a widely used de facto standard.
Accepted examples include a YAML library for configuration and the AWS SDK for S3-compatible storage.
### Explicit orchestration
The pipeline should remain stage-driven and explicit. New behavior should be added through clear stage, adapter, config, or manifest contracts rather than implicit side effects or generic workflow abstraction.
## Stage Design
Each stage should have a clear scope of responsibility.
A stage should define:
- its purpose;
- required input state;
- produced output state;
- config fields it consumes;
- external adapters it uses;
- manifest refs it reads or writes;
- skip, force, and resume behavior;
- failure behavior;
- tests that protect its contract.
Stages should avoid reaching across boundaries. If shared behavior is needed, prefer a helper or service with a narrow interface over duplicating ad hoc logic between stages.
## Transactionality and Resume
A stage should behave transactionally.
A stage is complete only when its outputs have been written, validated, and recorded in the manifest. If a stage fails, Narratio should preserve enough local state for inspection, recovery, and resume.
A failed or incomplete run must not be treated as successful. Later stages should depend on manifest-recorded success, not merely on incidental files existing on disk.
## Manifest Model
The manifest is the durable local ledger for a run.
It should record:
- run identity;
- stage status;
- input and output refs;
- logs and generated config refs;
- checksums or provenance where useful;
- non-secret adapter and archive metadata.
Resume behavior should be manifest-driven. Filesystem state may be inspected and validated, but it should not replace manifest stage state as the source of run progress.
## Adapter Boundaries
Adapters own external integration details.
Expected boundaries:
- WhisperX HTTP details stay in the WhisperX adapter.
- Seriatim CLI construction stays in the Seriatim adapter.
- Audita CLI construction stays in the Audita adapter.
- Scriptorium CLI construction stays in the Scriptorium adapter.
- Object-storage details stay behind the storage adapter interface.
- AWS SDK types stay inside the S3 storage implementation.
Stage code should express intent in Narratio terms and call adapters through narrow contracts.
## Configuration Philosophy
Configuration should be strict, explicit, and operator-friendly.
Principles:
- YAML decoding should reject unknown fields.
- Defaults should be centralized and testable.
- Empty configured values should not silently override meaningful defaults.
- Session templating should remain narrow and deterministic.
- Template support should serve operator convenience, not become a general configuration language.
Narratio should not become a secondary configuration system for downstream tools. Seriatim, Audita, and Scriptorium should own their runtime defaults wherever practical. Narratio should pass required stage-contract paths and explicit operator overrides.
## Path and Storage Discipline
Local and remote paths are part of Narratios application contract.
Code should use centralized path helpers for workspace, spool, session, run, artifact, log, config, and archive paths. Stages should avoid reconstructing canonical paths through scattered string concatenation.
Storage backends should receive explicit bucket-relative keys. Storage implementations should not infer campaign, session, run, or root-prefix semantics.
## Archive Invariants
Archive behavior must preserve a clear commit boundary.
A remote run is current only after the archive stage has successfully uploaded the run record, required promoted outputs, `current/manifest.json`, and finally `current/run_id.txt`.
`current/run_id.txt` is the final remote commit marker and must be written last.
Failed, incomplete, skipped, or uncommitted archive attempts must not be presented as current remote state. Local cleanup is permitted only after successful archive commit and only when explicitly configured.
## Security and Privacy
Narratio handles private campaign material.
Rules:
- Do not store raw secrets in pipeline or session YAML.
- Use environment variable names or secret-file references for secret handling.
- Do not write raw secret values to manifests, logs, generated configs, or archive metadata.
- Treat transcripts, generated artifacts, prompts, reports, and logs as potentially sensitive.
- Avoid logging transcript or prompt content unless there is a deliberate diagnostic reason.
## Diagnostics
Diagnostics should be durable and discoverable, but distinct from canonical outputs.
Logs, reports, generated invocation/config files, and render-debug files support debugging. Transcript tiers and configured artifacts are pipeline products.
Manifest refs should preserve that distinction.
## Determinism
Where practical, Narratio should prefer deterministic behavior:
- stable local path layout;
- stable remote key layout;
- sorted upload order;
- predictable generated config files;
- repeatable command construction;
- tests that do not depend on live external services.
Run IDs and timestamps may be intentionally variable, but surrounding behavior should remain testable.
## Testing Expectations
Core behavior should be testable without live external services.
Tests should cover:
- config loading, defaults, and validation;
- CLI parsing and command construction;
- path helpers;
- manifest transitions;
- stage success, failure, skip, and resume behavior;
- adapter command construction;
- fake storage behavior;
- archive commit ordering;
- example config validity where practical.
Live S3, WhisperX, LLM, or subprocess integration tests should be explicit integration tests, not required for ordinary unit test runs.
## Documentation Expectations
Documentation must follow `docs/documentation/policy.md`.
Current behavior belongs in user-facing docs and `docs/internal/`. Future, planned, aspirational, experimental, or unimplemented work belongs only under `docs/roadmap/`.
`docs/architecture.md` should remain concise and principle-focused. It should not duplicate the full config reference, CLI reference, operations guide, or internal stage documentation.
## Non-Goals
Narratio is not:
- a generic DAG or workflow engine;
- a replacement configuration layer for Seriatim, Audita, or Scriptorium;
- a storage backend abstraction beyond the needs of this pipeline;
- a place to embed raw secrets;
- a place for stage logic to depend directly on AWS SDK types or downstream tool internals;
- a prompt-authoring system.

278
docs/cli.md Normal file
View File

@@ -0,0 +1,278 @@
# CLI
## Shortest Useful Command
```bash
narratio run --session-id 2026-04-04
```
This command uses default discovery for `pipeline.yml` and `session.yml`; both files must be discoverable unless you pass explicit `--config` and `--session` paths.
## Command Overview
Implemented commands:
- `run`: execute pipeline stages and persist manifest state.
- `plan`: validate config, prepare workspace layout, and print stage run/skip decisions.
- `resume`: continue from first non-succeeded stage unless forced.
- `status`: read and print stage statuses from an existing manifest.
- `run-stage`: execute exactly one stage.
- `restore`: restore durable local session state from the committed remote archive state.
Unknown commands print usage and exit non-zero.
For config semantics, see [docs/config.md](./config.md). For operator lifecycle and recovery, see [docs/operations.md](./operations.md).
## Complete Flag Reference
### `run`
- `--config <path>`: optional explicit `pipeline.yml` path.
- `--session <path>`: optional explicit `session.yml` path.
- `--session-id <value>`: session template variable value.
- `--force`: force stage execution.
- `--artifacts <names>`: analyze artifact keys to execute (repeatable or comma-separated).
### `plan`
- `--config <path>`
- `--session <path>`
- `--session-id <value>`
- `--force`
### `resume`
- `--config <path>`
- `--session <path>`
- `--session-id <value>`
- `--force`
- `--artifacts <names>`: analyze artifact keys to execute (repeatable or comma-separated).
### `run-stage`
- `--config <path>`
- `--session <path>`
- `--session-id <value>`
- `--force`
- `--artifacts <names>`: analyze artifact keys to execute (repeatable or comma-separated).
- positional `<stage>`: required stage name.
Valid stage names:
- `prepare`
- `transcribe`
- `merge`
- `polish`
- `normalize`
- `trim`
- `analyze`
- `archive`
- `notify`
### `restore`
- `--config <path>`
- `--session <path>`
- `--session-id <value>`
- `--dry-run`: plan restore actions without writing local files.
- `--force`: overwrite local conflicting files with remote archive files.
- `--include-audio`: include durable archived `audio/**` files in restore scope.
### `status`
- `--manifest <path>`: required manifest path.
## Command Reference
### `run`
Purpose:
- Execute configured stages in canonical order.
Syntax:
```bash
narratio run [--config <pipeline.yml>] [--session <session.yml>] [--session-id <id>] [--force] [--artifacts <name[,name...]>]
```
Success output:
- `narratio run: session <session_id>; executed=<n> skipped=<n>; manifest=<path>`
Common failure cases:
- missing default config/session paths when flags omitted.
- invalid template/rendered session mismatch.
- unknown/invalid `--artifacts` value.
- `--artifacts` with unknown configured artifact key.
### `plan`
Purpose:
- Validate config, load secrets (if configured), prepare workdir, and print stage run/skip decisions.
Syntax:
```bash
narratio plan [--config <pipeline.yml>] [--session <session.yml>] [--session-id <id>] [--force]
```
Success output includes:
- `narratio plan: workdir prepared at <path>`
- one line per stage (`<stage>: run|skip`)
- `totals: run=<n> skip=<n>`
Common failure cases:
- same config/session discovery and validation failures as `run`.
- secrets directory read failures when `pipeline.secrets.env_dir` is configured.
### `resume`
Purpose:
- Continue from session-manifest stage status.
Syntax:
```bash
narratio resume [--config <pipeline.yml>] [--session <session.yml>] [--session-id <id>] [--force] [--artifacts <name[,name...]>]
```
Success output:
- `narratio resume: session <session_id> has no remaining stages`
- or `narratio resume: session <session_id>; executed=<n> skipped=<n>; manifest=<path>`
Common failure cases:
- same discovery/template/validation failures as `run`.
- manifest load errors when existing manifest is unreadable.
- invalid or unknown artifact selections.
### `status`
Purpose:
- Inspect one manifest file without executing stages.
Syntax:
```bash
narratio status --manifest <manifest.json>
```
Success output includes:
- `session_id: <id>`
- `updated_at: <timestamp>`
- `stages:` entries (`- <stage>: <status>`)
Common failure cases:
- missing `--manifest`.
- unreadable or invalid manifest path.
### `run-stage`
Purpose:
- Execute exactly one stage.
Syntax:
```bash
narratio run-stage [--config <pipeline.yml>] [--session <session.yml>] [--session-id <id>] [--force] [--artifacts <name[,name...]>] <stage>
```
Success output:
- `narratio run-stage: stage=<name> executed=<n> skipped=<n> force=<true|false>; manifest=<path>`
`--artifacts` behavior:
- accepted only when `<stage>` is `analyze`.
- names are normalized (trimmed, deduplicated, sorted).
- unknown configured artifact keys fail.
Common failure cases:
- missing stage positional arg.
- unknown stage name.
- using `--artifacts` with any non-`analyze` stage.
### `restore`
Purpose:
- Restore durable session state (`manifest.json`, `transcripts/**`, `artifacts/**`, and optional `audio/**`) from the committed remote archive current state.
Syntax:
```bash
narratio restore [--config <pipeline.yml>] [--session <session.yml>] [--session-id <id>] [--dry-run] [--force] [--include-audio]
```
Success output (dry-run):
- `Restore plan for <campaign>/<session_id>`
- `Remote run: <run_id>`
- `Would download: <n>`
- `Would skip unchanged: <n>`
- `Conflicts: <n>`
Success output (non-dry-run):
- `Restored session archive for <campaign>/<session_id>`
- `Remote run: <run_id>`
- `Downloaded: <n>`
- `Skipped unchanged: <n>`
- `Conflicts: <n>`
Common failure cases:
- storage backend is not configured.
- remote `current/run_id.txt` missing/empty.
- remote `current/manifest.json` missing or invalid.
- remote manifest session/campaign mismatch.
- local conflicts without `--force`.
- session lock conflict.
## Common Workflows
Default-discovery run:
```bash
narratio run --session-id 2026-04-04
```
Run only selected analyze artifacts:
```bash
narratio run --session-id 2026-04-04 --artifacts session_recap,player_handout
```
Resume with selected analyze artifacts:
```bash
narratio resume --session-id 2026-04-04 --artifacts player_handout
```
Run only analyze stage with selected artifacts:
```bash
narratio run-stage --session-id 2026-04-04 --artifacts player_handout analyze
```
Preview restore actions without writes:
```bash
narratio restore --session-id 2026-04-04 --dry-run
```
Restore and then force analyze:
```bash
narratio restore --session-id 2026-04-04
narratio run-stage --session-id 2026-04-04 --force analyze
```
## Diagnostic / Recovery Commands
Inspect stage status:
```bash
narratio status --manifest <manifest.json>
```
Get manifest path from previous output:
- `run`, `resume`, and `run-stage` print `manifest=<path>` on success.
## `--artifacts` and `--force`
- `--artifacts` filters which configured artifacts are executable when analyze runs.
- `--artifacts` does not imply `--force`.
- if analyze is already `succeeded` and `--force` is not set, runner-level skip still applies.

330
docs/config.md Normal file
View File

@@ -0,0 +1,330 @@
# Configuration
## 1. Overview
Narratio loads two YAML files:
- `pipeline.yml`: pipeline-level runtime configuration.
- `session.yml`: per-session metadata and input selection.
These commands load and validate both files before running:
- `narratio run`
- `narratio plan`
- `narratio resume`
- `narratio run-stage`
- `narratio restore`
Behavior:
- strict YAML decode is enabled (`KnownFields(true)`): unknown fields fail.
- session templates render before session YAML decode.
- defaults are applied for optional pipeline fields.
- validation enforces required fields, value formats, and cross-field constraints.
## 2. Config file discovery
Pipeline config lookup for `run`, `plan`, `resume`, `run-stage`, and `restore`:
- if `--config <path>` is provided, that path is used.
- if omitted, Narratio searches in order:
1. `/usr/local/etc/narratio/pipeline.yml`
2. `/etc/narratio/pipeline.yml`
- first existing file wins.
## 3. Session file discovery and templating
Session config lookup for `run`, `plan`, `resume`, `run-stage`, and `restore`:
- if `--session <path>` is provided, that path is used.
- if omitted, Narratio searches in order:
1. `./session.yml`
2. `/usr/local/etc/narratio/session.yml`
3. `/etc/narratio/session.yml`
- first existing file wins.
Template behavior:
- supported placeholders:
- `{{session_id}}`
- `{{ session_id }}`
- `--session-id <value>` supplies the placeholder value.
- unresolved placeholders fail load.
- if rendered `session_id` mismatches `--session-id`, load fails.
## 4. Minimal pipeline config
```yaml
whisperx:
transcribe_url: "https://transcription.example.com/transcribe"
```
Why this is sufficient:
- `whisperx.transcribe_url` is required.
- `workspace.root` defaults to `/var/lib/narratio`.
- optional sections (`seriatim`, `audita`, `archive`, `scriptorium`, `trim`, `normalize`, etc.) receive defaults or stay inactive.
## 5. Minimal session template
```yaml
session_id: "{{ session_id }}"
campaign: sample-campaign
inputs:
audio_dir: ./audio
speakers_file: ./examples/speakers.yml
autocorrect_file: ./examples/autocorrect.yml
glossary_file: ./examples/glossary.yml
```
Usage:
```bash
narratio run --config /path/to/pipeline.yml --session ./session.yml --session-id 2026-05-03
```
## 6. Production-oriented config
```yaml
workspace:
root: /var/lib/narratio/workspace
cleanup_after_archive: true
storage:
backend: s3
s3:
bucket: my-dnd-archive
root_prefix: dnd
region: us-east-1
access_key_id_env: OBJECT_STORAGE_KEY_ID
secret_access_key_env: OBJECT_STORAGE_KEY
spool:
root: /var/spool/narratio
delete_audio_after_archive: true
archive:
enabled: true
upload_run: true
promote_artifacts:
- source: narratio.transcript.trimmed
dest: transcripts/trimmed.json
required: true
- source: narratio.artifact.session_recap
dest: artifacts/session_recap.md
required: true
whisperx:
transcribe_url: "https://transcription.example.com/transcribe"
scriptorium:
artifacts:
session_recap:
enabled: true
prompt_id: dnd.session_recap
output_path: artifacts/session_recap.md
inputs:
transcript:
source: narratio.transcript.trimmed
required: true
```
Operational notes:
- archive promotion is explicit and source-based via `archive.promote_artifacts`.
- `source` is required; `dest` is optional and derived when omitted.
- Narratio does not auto-promote all generated analyze artifacts.
- `restore` reads the same config/session inputs and restore scope is bounded by committed archive current state.
## 7. Full pipeline reference
| Path | Type | Required | Default |
| --- | --- | --- | --- |
| `pipeline.workspace.root` | string | No | `/var/lib/narratio` |
| `pipeline.workspace.cleanup_after_archive` | bool | No | `false` |
| `pipeline.secrets.env_dir` | string | Conditional | none |
| `pipeline.storage.backend` | string | No | empty |
| `pipeline.storage.bucket` | string | No | empty |
| `pipeline.storage.prefix` | string | No | empty |
| `pipeline.storage.s3.bucket` | string | Conditional | empty |
| `pipeline.storage.s3.root_prefix` | string | No | `dnd` |
| `pipeline.storage.s3.region` | string | No | empty |
| `pipeline.storage.s3.endpoint` | string | No | empty |
| `pipeline.storage.s3.force_path_style` | bool | No | `false` |
| `pipeline.storage.s3.access_key_id_env` | string | No | `OBJECT_STORAGE_KEY_ID` |
| `pipeline.storage.s3.secret_access_key_env` | string | No | `OBJECT_STORAGE_KEY` |
| `pipeline.spool.root` | string | No | `/var/spool/narratio` |
| `pipeline.spool.delete_audio_after_archive` | bool | No | `false` |
| `pipeline.archive.enabled` | bool | No | `true` |
| `pipeline.archive.upload_run` | bool | No | `true` |
| `pipeline.archive.promote_artifacts[]` | list | No | trimmed transcript rule |
| `pipeline.archive.promote_artifacts[].source` | string | Yes (per rule) | none |
| `pipeline.archive.promote_artifacts[].dest` | string | No | derived from source |
| `pipeline.archive.promote_artifacts[].required` | bool | No | `true` |
| `pipeline.whisperx.transcribe_url` | string | Yes | none |
| `pipeline.whisperx.language` | string | No | `en` |
| `pipeline.whisperx.timeout` | duration string | No | `30m` |
| `pipeline.whisperx.retries` | int | No | `3` |
| `pipeline.whisperx.retry_delay` | duration string | No | `2s` |
| `pipeline.whisperx.concurrency` | int | No | `2` |
| `pipeline.seriatim.binary` | string | No | `seriatim` |
| `pipeline.seriatim.timeout` | duration string | No | `10m` |
| `pipeline.seriatim.output_schema` | string | No | `seriatim-intermediate` |
| `pipeline.seriatim.coalesce_gap` | float | No | `3.0` |
| `pipeline.seriatim.report` | bool | No | `true` |
| `pipeline.seriatim.env.overlap_word_run_gap` | float | No | unset |
| `pipeline.seriatim.env.overlap_word_run_reorder_window` | float | No | unset |
| `pipeline.seriatim.env.backchannel_max_duration` | float | No | unset |
| `pipeline.seriatim.env.filler_max_duration` | float | No | unset |
| `pipeline.audita.binary` | string | No | `audita` |
| `pipeline.audita.timeout` | duration string | No | `3h` |
| `pipeline.audita.llm_api_key_env` | string | No | empty |
| `pipeline.audita.modules[]` | list[string] | No | empty |
| `pipeline.audita.base_url` | string | No | empty |
| `pipeline.audita.model` | string | No | empty |
| `pipeline.audita.total_llm_concurrency` | int | No | unset |
| `pipeline.audita.proposal_llm_concurrency` | int | No | unset |
| `pipeline.audita.validation_model` | string | No | empty |
| `pipeline.audita.validation_llm_concurrency` | int | No | unset |
| `pipeline.audita.transcript_description` | string | No | empty |
| `pipeline.audita.config_path` | string | No | empty |
| `pipeline.audita.output_schema` | string | No | empty |
| `pipeline.audita.work_dir_retention` | string | No | empty |
| `pipeline.audita.report` | bool | No | `true` |
| `pipeline.normalize.output_path` | string | No | `transcripts/normalized.json` |
| `pipeline.normalize.output_schema` | string | No | `seriatim-intermediate` |
| `pipeline.normalize.report` | bool | No | `true` |
| `pipeline.trim.enabled` | bool | No | `false` |
| `pipeline.trim.output_path` | string | Conditional | none |
| `pipeline.trim.bounds.prompt_id` | string | Conditional | none |
| `pipeline.trim.bounds.profile_id` | string | No | empty |
| `pipeline.trim.bounds.transcript_input_name` | string | Conditional | none |
| `pipeline.trim.bounds.output_path` | string | Conditional | none |
| `pipeline.trim.bounds.timeout` | duration string | No | `10m` |
| `pipeline.trim.bounds.render_debug` | bool | No | `false` |
| `pipeline.trim.bounds.render_output_path` | string | Conditional | none |
| `pipeline.trim.seriatim.report` | bool | No | `false` |
| `pipeline.scriptorium.binary` | string | No | `scriptorium` |
| `pipeline.scriptorium.config_path` | string | No | empty |
| `pipeline.scriptorium.timeout` | duration string | No | `10m` |
| `pipeline.scriptorium.render_debug` | bool | No | `false` |
| `pipeline.scriptorium.artifacts` | map | No | empty |
| `pipeline.scriptorium.artifacts.<name>.enabled` | bool | No | `false` |
| `pipeline.scriptorium.artifacts.<name>.depends_on[]` | list[string] | No | empty |
| `pipeline.scriptorium.artifacts.<name>.render_debug` | bool | No | unset |
| `pipeline.scriptorium.artifacts.<name>.prompt_id` | string | Conditional | none |
| `pipeline.scriptorium.artifacts.<name>.profile_id` | string | No | empty |
| `pipeline.scriptorium.artifacts.<name>.output_path` | string | Conditional | none |
| `pipeline.scriptorium.artifacts.<name>.timeout` | duration string | No | empty |
| `pipeline.scriptorium.artifacts.<name>.inputs.<key>.source` | string | Conditional | none |
| `pipeline.scriptorium.artifacts.<name>.inputs.<key>.artifact` | string | No | empty |
| `pipeline.scriptorium.artifacts.<name>.inputs.<key>.path` | string | No | empty |
| `pipeline.scriptorium.artifacts.<name>.inputs.<key>.required` | bool | No | `false` |
| `pipeline.scriptorium.artifacts.<name>.vars.<key>` | map value | No | empty |
| `pipeline.analyzer.binary_path` | string | No | empty |
| `pipeline.analyzer.timeout` | duration string | No | empty |
| `pipeline.analyzer.artifacts.output_dir` | string | No | empty |
| `pipeline.analyzer.artifacts.types[]` | list[string] | No | empty |
| `pipeline.notification.backend` | string | No | empty |
| `pipeline.notification.recipient` | string | No | empty |
| `pipeline.notification.timeout` | duration string | No | empty |
Scriptorium artifact-key and dependency rules:
- artifact keys must match `^[a-z][a-z0-9_]*$`.
- enabled artifacts require `prompt_id` and `output_path`.
- `output_path` must be relative, traversal-safe, and under `artifacts/`.
- configured artifact input sources use `narratio.artifact.<name>`.
- if input source references `narratio.artifact.<name>`, artifact `<name>` must exist and must be listed in `depends_on`.
- every `depends_on` entry must be a configured artifact key.
- self-dependency is rejected.
- enabled dependency cycles are rejected.
- any artifact referenced by `depends_on` or `narratio.artifact.<name>` source must define `output_path` (even if not enabled).
Allowed `pipeline.scriptorium.artifacts.<name>.inputs.<key>.source` values:
- `previous_session_artifact`
- `narratio.transcript.merged`
- `narratio.transcript.polished`
- `narratio.transcript.full`
- `narratio.transcript.trimmed`
- `narratio.bounds.session`
- `narratio.artifact.<configured_artifact_key>`
`pipeline.archive.promote_artifacts[].source` values:
- `narratio.transcript.merged`
- `narratio.transcript.polished`
- `narratio.transcript.full`
- `narratio.transcript.trimmed`
- `narratio.bounds.session`
- `narratio.artifact.<configured_artifact_key>`
Archive promotion destination rules:
- `dest` must be a clean relative path (not absolute, no traversal).
- duplicate `dest` values are rejected.
- if `dest` is omitted:
- built-in sources derive their canonical destination path;
- configured sources derive from `pipeline.scriptorium.artifacts.<name>.output_path`;
- derivation failure is a config validation error.
Restore-related implications:
- restore remote identity requires archive S3 identity to resolve (`pipeline.storage.s3.bucket` and session prefix derivation inputs).
- restore scope considers only committed current state and durable paths (`manifest.json`, `transcripts/**`, `artifacts/**`, optional `audio/**`).
## 8. Full session reference
| Path | Type | Required | Default |
| --- | --- | --- | --- |
| `session.session_id` | string | Yes | none |
| `session.campaign` | string | Yes | none |
| `session.date` | string | No | empty |
| `session.title` | string | No | empty |
| `session.inputs.audio_dir` | string | Conditional | empty |
| `session.inputs.audio_files[]` | list[string] | Conditional | empty |
| `session.inputs.audio_s3.prefix` | string | Conditional | none |
| `session.inputs.speakers_file` | string | Yes | none |
| `session.inputs.autocorrect_file` | string | Yes | none |
| `session.inputs.glossary_file` | string | Yes | none |
Audio-source rule:
- configure exactly one mode:
- `audio_dir`, or
- `audio_files` (at least one), or
- `audio_s3.prefix`
- `audio_s3` cannot be combined with local audio fields.
## 9. Secrets
Narratio supports filesystem-based secret injection via `pipeline.secrets.env_dir`.
Behavior:
- `env_dir` may be absolute or relative.
- relative `env_dir` resolves from current working directory.
- files with valid env-var names (`[A-Za-z_][A-Za-z0-9_]*`) are loaded.
- values are loaded from file contents with trailing newline trimming.
- existing process env vars are preserved.
- invalid names and subdirectories are skipped.
- missing/unreadable `env_dir` fails command execution.
Guidance:
- do not put secret values directly in YAML.
- configure env var names in config and provide values via env/secrets files.
## 10. Examples
Maintained examples:
- `examples/pipeline.minimal.yml`
- `examples/pipeline.production.yml`
- `examples/pipeline.full.annotated.yml`
- `examples/session.template.yml`
- `examples/session.local-audio.yml`
- `examples/session.s3-audio.yml`
These examples are validated by `internal/config` tests.

92
docs/development.md Normal file
View File

@@ -0,0 +1,92 @@
# Development Guide
## Purpose
Canonical contributor workflow and engineering conventions for implemented Narratio behavior.
## Repository layout
- `cmd/narratio/`: CLI entrypoint.
- `internal/app/`: command handlers, plan/run/resume orchestration, cleanup gates, secrets loading.
- `internal/config/`: strict YAML loading, defaults, and validation.
- `internal/stage/`: stage implementations and stage registry/order.
- `internal/adapters/`: external boundary adapters (WhisperX, Seriatim, Audita, Scriptorium, storage, notify).
- `internal/manifest/`: session/run manifest types and persistence.
- `internal/artifacts/`: canonical local/remote path helpers and local artifact store.
- `docs/`: canonical documentation set.
- `examples/`: maintained config examples used by tests.
## Build and test commands
- Run focused CLI behavior checks:
```bash
go test ./internal/app -run TestExecute -v
```
- Run config example load/validate checks:
```bash
go test ./internal/config -run TestExamplesLoadAndValidate -v
```
- Run full test suite:
```bash
go test ./...
```
## Coding conventions
- Keep orchestration explicit and stage-driven; do not introduce generic workflow/DAG abstractions.
- Keep external-system details inside adapter packages; stages should consume Narratio-level contracts only.
- Use centralized path helpers from `internal/artifacts` rather than ad hoc path concatenation.
- Preserve manifest-driven state transitions (`running`, `succeeded`, `failed`, `skipped`, `stale`) as the source of run progress.
- Keep user/operator docs implementation-accurate; planned work belongs only under `docs/roadmap/`.
For design principles and invariants, see [docs/architecture.md](./architecture.md). For stage/adapter contracts, see [docs/internal/README.md](./internal/README.md).
## Dependency policy
- Prefer Go standard library where practical.
- Add third-party dependencies only when they provide clear value for required behavior.
- Keep dependency additions narrow to the boundary package that needs them.
## Change playbooks
### Add config fields
1. Add fields to config structs in `internal/config`.
2. Set defaults in `internal/config/defaults.go` when appropriate.
3. Add validation rules in `internal/config/validate.go`.
4. Add or update load/validate tests in `internal/config/*_test.go`.
5. Update canonical config docs and examples:
- [docs/config.md](./config.md)
- relevant files under `examples/`
### Add CLI flags or commands
1. Update command parsing and behavior in `internal/app`.
2. Add or update command tests (`TestExecute` and command-specific tests).
3. Update [docs/cli.md](./cli.md) and, if operator workflow changes, [docs/operations.md](./operations.md).
### Add or modify stages/adapters
1. Implement stage behavior in `internal/stage` with clear input/output boundaries.
2. Keep external transport/subprocess details in `internal/adapters`.
3. Preserve manifest and promotion semantics expected by runner and archive logic.
4. Add/update stage and adapter tests.
5. Update internal component contracts in `docs/internal/`.
### Update examples
1. Keep canonical examples only in `examples/`.
2. Ensure examples load and validate through runtime config paths.
3. Update `internal/config/load_validate_test.go` as needed.
4. Update links in `docs/config.md` if example filenames change.
### Update docs and roadmap
1. Keep implemented behavior in canonical docs (`README`, `docs/*.md`, `docs/internal/`).
2. Keep planned/unimplemented behavior only in `docs/roadmap/`.
3. After completing roadmap items, remove or mark them complete in `docs/roadmap/documentation.md`.
4. Run a link/path sweep before finalizing changes.

View File

@@ -1,507 +0,0 @@
# Workspace Architecture Implementation Plan (Audit)
## 1. Executive Summary
**Complexity assessment:** **heavy**.
This is not a single path-helper refactor. The current codebase has a hybrid session/run model that works for current behavior, but diverges from `docs/development/workspace.md` in foundational places (workspace root shape, manifest responsibilities, stage output placement, and archive symmetry).
Highest-risk areas:
1. Splitting the current single manifest model into durable **session manifest** vs per-invocation **run manifest** without regressing skip/force/resume UX.
2. Migrating path helpers and artifact-store interfaces from session-only roots (`work/{session}`) to campaign-aware roots (`work/{campaign}/{session}`) while preserving existing runs.
3. Introducing run-local stage outputs + immediate promotion while keeping stage tests and archive behavior stable.
4. Avoiding stale downstream skips after forced upstream reruns.
Surprising findings:
1. Code already has campaign/run-aware helpers (`SessionRunWorkDir`, `SessionSpoolAudioDir`) but core session helpers and manifest pathing remain campaign-unaware.
2. Archive recently gained run/session fallback behavior for manifest/promotion sources, which confirms an existing hybrid-layout pressure point.
3. Analyze input resolution is functional but ad hoc and stage-local; there is no centralized artifact registry/resolver.
---
## 2. Current-State Map
## 2.1 Workspace Path Construction
Primary path model:
- [`internal/artifacts/paths.go`](../../internal/artifacts/paths.go)
- `SessionWorkDir(rootDir, sessionID)` => `{root}/work/{session_id}`
- `buildSessionPaths(workspaceRoot, sessionID)` roots all canonical paths under `{root}/work/{session_id}`
- `SessionRunWorkDir(rootDir, campaign, sessionID, runID)` exists, but is not the default session root helper.
Artifact store abstraction:
- [`internal/artifacts/store.go`](../../internal/artifacts/store.go)
- `SessionPaths(sessionID string)` / `EnsureLayout(sessionID string)` are session-id-only (no campaign argument).
- [`internal/artifacts/local.go`](../../internal/artifacts/local.go)
- `EnsureLayout` creates session-level folders under `SessionWorkDir`.
Path normalization helper:
- [`internal/artifacts/resolve.go`](../../internal/artifacts/resolve.go)
- `ResolveSessionLocalPathForRead` accepts absolute/workspace/session-relative values and probes filesystem.
Where campaign-aware/run-aware support exists:
- Local run path helpers: `SessionRunWorkDir`, `SessionSpoolAudioDir`.
- S3 key helpers: [`internal/artifacts/s3_keys.go`](../../internal/artifacts/s3_keys.go) (`campaigns/sessions/runs/current`).
Where session-root assumptions remain `{workspace}/work/{session}`:
- Manifest path computation: [`internal/app/runner.go`](../../internal/app/runner.go) `manifestPathFor`.
- Artifact store layout and most stage `paths := env.ArtifactStore.SessionPaths(sessionID)` calls.
- Many tests hardcode `workspace/work/<session>/...` (examples below).
Manual/ad hoc path construction (not through a single resolver API):
- Common stage patterns: `filepath.Join(paths.TranscriptsDir, "...")`, `filepath.Join(paths.ArtifactsDir, "...")`, etc.
- `prepare` run/work selection: [`internal/stage/prepare.go`](../../internal/stage/prepare.go) `pathsWorkDirForManifest`.
- Analyze input fallback path resolution: `resolveInputPathForRead` in [`internal/stage/analyze.go`](../../internal/stage/analyze.go).
## 2.2 Manifest and Run Identity
Current manifest model:
- [`internal/manifest/manifest.go`](../../internal/manifest/manifest.go) `Manifest` includes both session and run fields:
- `SessionID`, `Campaign`
- `RunID`, `LocalWorkDir`, `LocalSpoolDir`
- `S3Bucket`, `S3SessionPrefix`, `S3RunPrefix`
- `Stages`, `Inputs`, stage outputs/logs/generated configs/metadata.
Current persistence:
- [`internal/manifest/store.go`](../../internal/manifest/store.go) `LocalStore` reads/writes one JSON manifest path.
- Runner always loads/saves one manifest path via `manifestPathFor(cfg)` (session-root path under current layout).
Identity initialization:
- [`internal/app/runner.go`](../../internal/app/runner.go) `ensureManifestIdentity` populates run fields if absent.
- `RunID` is generated once for an empty manifest and reused thereafter (hybrid semantics).
Interpretation today:
- Best described as a **hybrid session manifest** with run identity fields, not as distinct session + run manifests.
What this means for redesign:
- Session-vs-run split is not just file relocation; it requires new responsibilities and write flows.
- A backward-compatible evolution path is possible by:
- preserving current fields in session manifest for migration/read-compat,
- adding explicit run-manifest type + path,
- gradually moving invocation-specific details to run manifests.
## 2.3 Stage Output Paths (Current)
All implemented stages currently write canonical artifacts directly into session-level `paths.*` roots (under current session root), with logs/configs typically also session-level.
1. `prepare` ([`internal/stage/prepare.go`](../../internal/stage/prepare.go))
- Inputs copied to `inputs/` (`session.yml`, `pipeline.resolved.yml`, `speakers.yml`, `autocorrect.yml`, `glossary.yml`)
- Audio copied to `audio/`
- Manifest input provenance recorded in `m.Inputs`
- S3 audio uses run-scoped spool/work helpers when `RunID` is present.
2. `transcribe` ([`internal/stage/transcribe.go`](../../internal/stage/transcribe.go))
- Outputs: `transcripts/raw/{speaker}.json`
- Stage metadata only (no stage logs/config files produced here).
3. `merge` ([`internal/stage/merge.go`](../../internal/stage/merge.go))
- Pre-normalize intermediates: `transcripts/raw/normalized/{basename}.normalized.json`
- Merge output: `transcripts/merged.json`
- Report: `artifacts/seriatim.report.json` (if enabled)
- Logs/configs:
- per-input normalize logs/configs in session-level `logs/` + `config/`
- merge logs/config in session-level `logs/` + `config/`
4. `polish` ([`internal/stage/polish.go`](../../internal/stage/polish.go))
- Output: `transcripts/processed.json`
- Report: `artifacts/audita.report.json` (if enabled)
- Work dir: `artifacts/audita-work`
- Logs/config: `logs/audita.*`, `config/audita.generated.yml`
5. `normalize` ([`internal/stage/normalize.go`](../../internal/stage/normalize.go))
- Output: `transcripts/normalized.json` (configurable)
- Report: `artifacts/seriatim.normalize.report.json` (if enabled)
- Logs/config: `logs/seriatim.normalize.*`, `config/seriatim.normalize.generated.yml`
6. `trim` ([`internal/stage/trim.go`](../../internal/stage/trim.go))
- Output: `transcripts/trimmed.json` (configurable)
- Bounds output path from config (default examples use `artifacts/session_bounds.json`)
- Logs/configs in session-level `logs/` and `config/` for scriptorium + seriatim invocations
7. `analyze` ([`internal/stage/analyze.go`](../../internal/stage/analyze.go))
- Current implemented artifact: `artifacts/session_recap.md`
- Logs/config in session-level `logs/` + `config/`
- Optional render diagnostics under session-level artifacts/logs/config.
8. `archive` ([`internal/stage/archive.go`](../../internal/stage/archive.go))
- Reads from a "workDir" derived by `archiveWorkDir`:
- prefers `m.LocalWorkDir` if it exists,
- else tries run-scoped campaign/session/run path,
- else falls back to legacy session path.
- Uploads run record and promotions.
- Current code now resolves manifest and promotion sources via run/session fallback.
9. `notify`
- Placeholder only in [`internal/stage/placeholders.go`](../../internal/stage/placeholders.go); no durable outputs.
Durable-vs-diagnostic split today:
- Durable artifacts and diagnostics are mixed at session level.
- No run-local stage directories exist yet.
## 2.4 Idempotency / Force / Resume / Sparse Runs
Run control implementation:
- [`internal/app/run_control.go`](../../internal/app/run_control.go)
- skip rule: `!force && stageSucceeded(manifest, stage)`
- stale detection TODO only; no invalidation logic.
Command behavior:
- `run`: full plan through `executeStages`.
- `run-stage`: single selected stage through `executeStages`.
- `resume`: starts at first non-succeeded stage unless `--force`.
Current assumptions:
- Command invocation mutates a single durable workspace + single manifest for the session path model.
- There is no per-invocation run manifest write path.
Smallest safe UX-preserving invariant to keep during migration:
- Session manifest remains source-of-truth for skip decisions across invocations.
## 2.5 Archive and S3 Alignment
S3 semantics are relatively mature:
- [`internal/artifacts/s3_keys.go`](../../internal/artifacts/s3_keys.go)
- session prefix + run prefix + `current/manifest.json` + `current/run_id.txt`.
Archive stage behavior:
- uploads run records under run prefix,
- uploads promoted session outputs,
- uploads current manifest then current run pointer,
- current pointer is commit marker,
- required promotions fail; optional promotions skipped.
Local-vs-remote mismatch still present:
- Local canonical session root currently defaults to `work/{session}` (artifact store),
- while archive/run helpers expect campaign-aware run locations (`work/{campaign}/{session}/{run}`),
- causing fallback logic and hybrid handling in `archive`.
## 2.6 Artifact Source Resolution in Analyze
Current implementation is stage-local and ad hoc:
- Transcript discoverers:
- `discoverProcessedTranscript`
- `discoverNormalizedTranscript`
- `discoverTrimmedTranscript`
- Input source switch in `resolveScriptoriumInput` supports:
- `processed_transcript`
- `normalized_transcript`
- `trimmed_transcript`
- `previous_session_artifact`
No first-class artifact registry exists yet. Aliases and canonical IDs are not modeled.
## 2.7 Stale/Invalidation
- `manifest.StageStatus` already defines `stale` (`internal/manifest/status.go`), but no stage uses it.
- Skip logic ignores stale state and only checks `succeeded`.
- Forced upstream rerun does not invalidate downstream stage success markers.
## 2.8 Test Coverage Relevant to Redesign
High-value existing coverage:
- Workspace helpers:
- [`internal/artifacts/paths_model_test.go`](../../internal/artifacts/paths_model_test.go)
- [`internal/artifacts/resolve_test.go`](../../internal/artifacts/resolve_test.go)
- Runner semantics:
- [`internal/app/runner_test.go`](../../internal/app/runner_test.go)
- [`internal/app/resume_run_stage_test.go`](../../internal/app/resume_run_stage_test.go)
- Stage path/output behavior:
- `internal/stage/*_test.go` for prepare/transcribe/merge/polish/normalize/trim/analyze/archive
- Archive behavior:
- [`internal/stage/archive_test.go`](../../internal/stage/archive_test.go)
- [`internal/app/post_archive_cleanup_test.go`](../../internal/app/post_archive_cleanup_test.go)
Tests likely to fail during workspace redesign:
- Any tests hardcoding `work/{session}` manifest and transcript paths (many in `internal/app/*test.go`, `internal/stage/*test.go`).
- Archive tests assuming current hybrid fallback behavior.
---
## 3. Gap Analysis Against `docs/development/workspace.md`
| Intended concept | Current status | Notes |
|---|---|---|
| Session root at `work/{campaign}/{session}` | **Partial / mostly absent** | `SessionWorkDir` and artifact store still use `work/{session}`. Campaign-aware run helper exists separately. |
| Distinct session manifest vs run manifest | **Absent** | One hybrid manifest model/file is used. |
| Run-local stage dirs under `runs/{run_id}/{stage}` | **Absent** | Stages write canonical outputs/logs/config directly at session level. |
| Immediate promotion run-local -> session canonical | **Absent** | No run-local staging area to promote from today. |
| Session manifest skip source of truth | **Present** | Skip/resume use single manifest stage statuses. |
| Sparse runs represented under `runs/{run_id}` | **Absent** | No run-manifest/per-run stage records yet. |
| Artifact resolver with canonical IDs + aliases | **Absent** | Analyze resolves via stage-local source-name switch and fallback helpers. |
| Downstream invalidation for forced upstream reruns | **Absent** | TODO only; no stale propagation or status clearing. |
| Local semantics mirror archive semantics | **Partial** | Archive/S3 side models runs/current, local workspace core still session-layout-centric. |
| Safe path helpers centralization | **Partial** | Good helper base exists, but many stage-level manual joins still encode conventions. |
---
## 4. Recommended Implementation Sequence
## Step 1: Introduce campaign-aware session path model without behavior break
Purpose:
- Add first-class helpers for `work/{campaign}/{session}` and make them available everywhere.
Expected changes:
- `internal/artifacts`: add/extend path helpers and `SessionPaths` constructor variants that accept campaign.
- `internal/app`: pass campaign into path-model entrypoints where available.
Behavior change:
- None initially (can keep legacy fallback reads).
Tests:
- Add campaign-aware path-model tests.
- Keep legacy-path compatibility tests.
Risks:
- Wide compile-time touch due `Store` interface signatures.
Rollback:
- Keep legacy helper wrappers until full migration lands.
## Step 2: Split manifest responsibilities (session manifest + run manifest scaffolding)
Purpose:
- Preserve current UX while introducing explicit run execution records.
Expected changes:
- `internal/manifest`: add run manifest type/store helpers.
- `internal/app/runner.go`: create/load session manifest and initialize per-invocation run manifest path.
Behavior change:
- Session manifest remains skip truth source.
- Run manifest begins recording invocation metadata/stage actions.
Tests:
- New tests for both manifest files existing and being updated correctly.
Risks:
- Incorrect ordering of saves can regress crash consistency.
Rollback:
- Keep session-manifest-only decision logic until run manifest proves stable.
## Step 3: Move stage execution products to run-local directories with promotion
Purpose:
- Align with workspace architecture (`runs/{run_id}/{stage}/...`) while preserving canonical outputs.
Expected changes:
- `internal/stage`: each implemented stage writes outputs/logs/config/reports to run-local paths.
- Introduce shared promotion helpers (atomic copy/rename + output validation + manifest provenance).
Behavior change:
- Canonical outputs remain session-level; run-local diagnostics now preserved per run.
Tests:
- Stage tests updated to assert run-local outputs + promoted canonical outputs.
- New tests for immediate promotion and producer run metadata.
Risks:
- Highest regression risk (all implemented stages touched).
Rollback:
- Stage-by-stage migration flag or phased rollout by stage order.
## Step 4: Archive alignment pass
Purpose:
- Remove hybrid fallback complexity once local layout is canonical.
Expected changes:
- `internal/stage/archive.go`: resolve sources from canonical session outputs and run manifests deterministically.
- Keep `current/manifest.json` + `current/run_id.txt` publication semantics.
Behavior change:
- Simpler source selection; fewer cross-layout heuristics.
Tests:
- Archive tests for run uploads, promotions, current pointers, and fallback removal/compat gates.
Risks:
- Breaking current mixed-layout compatibility too early.
Rollback:
- Retain fallback compatibility for one migration window.
## Step 5: Artifact registry/resolver (analyze first consumer)
Purpose:
- Replace ad hoc analyze source resolution with canonical artifact IDs and aliases.
Expected changes:
- New resolver package (for example `internal/artifacts/registry` or `internal/stage/artifactresolve`) with IDs:
- `narratio.transcript.merged`
- `narratio.transcript.polished`
- `narratio.transcript.full`
- `narratio.transcript.trimmed`
- `narratio.bounds.session`
- `narratio.artifact.session_recap`
- Backward-compatible alias map for current source names.
Behavior change:
- Analyze input resolution becomes centralized and consistent.
Tests:
- Resolver unit tests for canonical IDs + aliases + missing-input error clarity.
- Analyze tests updated to assert resolver usage.
Risks:
- Input resolution edge cases for `previous_session_artifact` and required/optional handling.
Rollback:
- Keep old resolver path behind a temporary compatibility function.
## Step 6: Minimal downstream invalidation after forced upstream reruns
Purpose:
- Prevent stale downstream skips before full checksum stale detection exists.
Expected changes:
- `internal/app/run_control.go` + manifest transition helpers.
- On forced rerun success of stage `X`, clear/mark downstream success states.
Behavior change:
- Subsequent runs no longer skip stale downstream stages.
Tests:
- New run-control tests for force-induced downstream invalidation.
- Resume tests with forced sparse runs.
Risks:
- Over-invalidating too broadly and degrading UX.
Rollback:
- Start with deterministic downstream stage list based on pipeline order only.
## Step 7: Legacy layout migration strategy
Purpose:
- Handle existing `work/{session}` data safely.
Expected changes:
- Startup detection/migration path in app layer.
- Clear failure messages for ambiguous legacy states.
Behavior change:
- Explicit migration semantics instead of implicit fallback drift.
Tests:
- Migration detection tests for legacy-only, new-only, and ambiguous layouts.
Risks:
- Silent duplication if both layouts are partially populated.
Rollback:
- Prefer fail-fast ambiguity policy over auto-merge.
---
## 5. Minimal Viable v1.0 Scope
Must-have for v1.0:
1. Campaign-aware canonical session roots.
2. Session manifest remains skip/resume authority.
3. Introduce run manifests + run-local stage records.
4. Immediate promotion from run-local outputs to canonical session outputs.
5. Minimal downstream invalidation on forced upstream rerun.
6. Archive/local semantic alignment with `current/*` behavior preserved.
7. Backward-compatible analyze aliases (`processed_transcript`, `normalized_transcript`, `trimmed_transcript`).
Nice-to-have / defer if risky:
1. Full checksum-based stale detection graph.
2. Broad manifest schema-version migration framework.
3. Full artifact-registry rollout beyond analyzes initial needs.
4. Aggressive cleanup of all legacy fallback branches in one release.
---
## 6. Open Questions (with Recommendations)
1. **Should campaign be mandatory for all local path derivation immediately?**
- Recommendation: yes for new layout writes; keep controlled read compatibility for legacy session-only roots during migration window.
2. **Session manifest location transition policy:** auto-migrate vs explicit migrate command?
- Recommendation: if user base is small, prefer explicit fail-fast with actionable migration instructions to avoid silent split-brain state.
3. **Run manifest granularity:** per-stage detailed records vs summary + references?
- Recommendation: start with summary + stage status/paths; avoid duplicating full session artifact state to keep write path simple.
4. **Invalidation marking:** use `stale` status now or clear `succeeded` markers?
- Recommendation: if invasive to propagate new status semantics quickly, clear/overwrite downstream success states first; add formal `stale` usage in follow-up.
5. **Promotion timing:** promote every stage immediately vs delayed at end of run?
- Recommendation: immediate per-stage promotion after validation (matches current idempotent skip expectations and simplifies resume behavior).
6. **Docs consistency order after implementation starts:**
- Recommendation: update `README.md` and `architecture.md` in lockstep with each migration step. Current conflict is explicit:
- code/README/architecture still primarily describe session roots at `work/{session}`
- `docs/development/workspace.md` defines `work/{campaign}/{session}` plus run manifests.

View File

@@ -1,753 +0,0 @@
# Narratio Workspace, Run History, and Artifact Resolution Architecture
## 1. Purpose
This document defines the intended v1.0 architecture for Narratio's local workspace layout, run history model, durable session outputs, manifest responsibilities, and artifact resolution contract.
Narratio is an idempotent session orchestrator. The command:
```bash
narratio run --session-id 2026-05-07
```
means "bring the identified session to its desired completed state." It does **not** mean "always create an entirely new independent output tree and ignore prior session state."
This distinction drives the architecture:
* A **session** is the durable domain object and idempotency boundary.
* A **run** is an execution attempt that may update the session's durable state.
* Durable outputs live at the session level.
* Run-specific outputs, logs, generated configs, scratch files, and diagnostics live under `runs/{run_id}/`.
* Successful stage outputs are promoted from run-local locations into canonical session-level locations.
* The session manifest records current durable state.
* Run manifests record execution history and debugging/provenance details.
This model intentionally mirrors the S3 archive model: session-level current artifacts are distinct from run-record history.
## 2. Core Concepts
### 2.1 Session
A session is the stable unit of work identified by `campaign_id` and `session_id`.
Examples:
```text
campaign_id = dilfs
session_id = 2026-05-07
```
The session directory represents the current durable local state for that session. Re-running Narratio for the same session should consult this state, skip already-completed stages by default, and produce no changes unless work is incomplete, stale, forced, or explicitly selected.
### 2.2 Run
A run is a particular execution attempt identified by a generated `run_id`, for example:
```text
20260517T174748Z-abcd1234
```
A run may execute all stages or only a sparse subset of stages. Sparse runs are expected and desirable when the user invokes `--force`, `run-stage`, or a stage-limited command.
Run directories are provenance/debug records. They should reflect what actually happened during that invocation, not a synthetic complete pipeline layout.
### 2.3 Durable Output
A durable output is a canonical session-level artifact intended for later stages, user consumption, archive promotion, or future idempotency decisions.
Examples:
```text
transcripts/merged.json
transcripts/processed.json
transcripts/normalized.json
transcripts/trimmed.json
artifacts/session_recap.md
```
Durable outputs live directly under the session directory, not under a particular run directory.
### 2.4 Run-Local Output
A run-local output is the file initially produced by a stage during a specific run. After validation, durable outputs are promoted from run-local paths to session-level canonical paths.
Run-local outputs, logs, generated configs, reports, and scratch files should remain under:
```text
runs/{run_id}/{stage}/...
```
## 3. Local Workspace Layout
The canonical local workspace layout is:
```text
{workspace.root}/work/{campaign_id}/{session_id}/
manifest.json
current/
manifest.json
run_id.txt
inputs/
transcripts/
artifacts/
reports/
logs/
config/
runs/
{run_id}/
manifest.json
prepare/
transcribe/
merge/
polish/
normalize/
trim/
analyze/
archive/
notify/
```
Not every directory must exist at all times. Directories should be created idempotently when needed.
### 3.1 Session Root
The session root is:
```text
{workspace.root}/work/{campaign_id}/{session_id}/
```
The session root is the stable local home for the session. It is the default base for resolving canonical artifact paths.
The only files that should live directly in the session root are core session-state files, primarily:
```text
manifest.json
```
Lock files may also be session-root scoped if the implementation uses file locks there, but transient locks should not be treated as durable artifacts.
### 3.2 Session-Level Canonical Directories
The following directories contain current durable session state:
```text
inputs/
transcripts/
artifacts/
reports/
logs/
config/
current/
```
Recommended meanings:
| Directory | Purpose |
| -------------- | ----------------------------------------------------------------------------- |
| `inputs/` | Materialized or copied input files used by the current durable session state. |
| `transcripts/` | Canonical transcript tiers. |
| `artifacts/` | User-facing and machine-readable generated artifacts. |
| `reports/` | Canonical stage reports worth preserving at the session level. |
| `logs/` | Optional session-level logs or promoted/latest logs. |
| `config/` | Optional session-level generated config snapshots or promoted/latest configs. |
| `current/` | Current published session pointers, mirroring the archive backend. |
Canonical durable outputs should use stable paths under these directories.
### 3.3 Run History Directory
Run history lives under:
```text
{workspace.root}/work/{campaign_id}/{session_id}/runs/{run_id}/
```
Each run directory records what happened during that invocation. A run may contain all stage directories or only a sparse subset.
Example full run:
```text
runs/20260517T174748Z-abcd1234/
manifest.json
prepare/
transcribe/
merge/
polish/
normalize/
trim/
analyze/
archive/
notify/
```
Example sparse forced analyze run:
```text
runs/20260518T030000Z-efgh5678/
manifest.json
analyze/
```
Example sparse polish-through-analyze rerun:
```text
runs/20260518T041500Z-a1b2c3d4/
manifest.json
polish/
normalize/
trim/
analyze/
```
Run directories should not create stage folders for stages that were not selected, executed, skipped, or otherwise considered during that run unless there is a clear diagnostic reason to do so.
### 3.4 Stage Run-Local Directories
Each stage receives a run-local directory:
```text
runs/{run_id}/{stage}/
```
Within that stage directory, the stage may use subdirectories such as:
```text
outputs/
logs/
reports/
config/
scratch/
```
For example:
```text
runs/{run_id}/polish/
outputs/transcripts/processed.json
reports/audita.polish.report.json
logs/stdout.log
logs/stderr.log
config/audita.polish.generated.yml
scratch/
```
The exact internal layout of a stage directory may vary by stage, but it should be deterministic, documented, and generated through centralized path helpers rather than ad hoc path joins.
## 4. Promotion Model
Narratio uses stage-level promotion with immediate promotion after successful validation.
The stage lifecycle is:
1. Resolve required inputs from the current session state and/or run-local context.
2. Create the run-local stage directory.
3. Execute the stage, writing outputs under `runs/{run_id}/{stage}/...`.
4. Validate run-local outputs.
5. Promote durable outputs into session-level canonical paths.
6. Update the session manifest.
7. Update the run manifest.
Promotion means an atomic or effectively atomic copy/rename from a run-local path to a session-level canonical path.
Example:
```text
runs/{run_id}/polish/outputs/transcripts/processed.json
```
is promoted to:
```text
transcripts/processed.json
```
Promotion should be safe and deterministic:
* Validate before promotion.
* Write promoted files atomically where possible.
* Never leave partially written durable outputs.
* Record the producing `run_id` in the session manifest.
* Preserve run-local files for debugging unless retention policy deletes them.
## 5. Promotion Policy: Option A
Narratio uses immediate stage-level promotion.
If a selected stage succeeds, its durable outputs are promoted immediately, even if a later selected stage fails.
Example:
```bash
narratio run --session-id 2026-05-07 --force --stages polish,normalize,trim,analyze
```
If `polish` succeeds and `normalize` fails:
* `transcripts/processed.json` may be updated from the new run.
* `normalize`, `trim`, and `analyze` should not be marked succeeded for the new input state.
* Downstream outputs may now be stale relative to the newly promoted polished transcript.
This policy is simpler, transparent, and consistent with stage-level resumability. It does require explicit stale/invalidation handling.
## 6. Stale and Invalidation Semantics
Full checksum-based stale detection may be implemented later. Before that exists, Narratio should still use a simple deterministic invalidation rule for forced or explicit upstream reruns.
When a stage is successfully re-executed and promoted, downstream stages should be marked stale unless they are also re-executed successfully in the same command invocation.
Example stage order:
```text
prepare -> transcribe -> merge -> polish -> normalize -> trim -> analyze -> archive -> notify
```
If `polish` is forced and promoted, then the following downstream stages should be invalidated unless rerun successfully:
```text
normalize
trim
analyze
archive
notify
```
A stale stage is not equivalent to a failed stage. It means its current durable outputs may no longer correspond to current upstream inputs or configuration.
Minimum manifest state model:
```text
pending
running
succeeded
failed
skipped
stale
```
If adding a new `stale` state is too invasive for v1.0, the implementation should at least record stale metadata or clear downstream success markers in a way that prevents accidental idempotent skips based on obsolete outputs.
## 7. Manifest Responsibilities
Narratio should distinguish between session manifests and run manifests.
The same underlying Go types may be reused where practical, but the concepts should remain separate.
### 7.1 Session Manifest
Path:
```text
{workspace.root}/work/{campaign_id}/{session_id}/manifest.json
```
The session manifest answers:
```text
What is the current durable state of this session?
```
It should record:
* campaign ID
* session ID
* current or latest run ID
* current stage states
* canonical durable output refs
* artifact IDs and paths
* producing run ID for each current stage output
* relevant input/config checksums when available
* stale/invalidated stage information
* archive/current publication metadata
A session's durable state may be a composite of multiple runs.
For example:
```text
transcripts/merged.json produced by run A
transcripts/processed.json produced by run B
transcripts/normalized.json produced by run B
transcripts/trimmed.json produced by run B
artifacts/session_recap.md produced by run C
```
This is valid and expected.
### 7.2 Run Manifest
Path:
```text
{workspace.root}/work/{campaign_id}/{session_id}/runs/{run_id}/manifest.json
```
The run manifest answers:
```text
What happened during this specific execution attempt?
```
It should record:
* run ID
* campaign ID
* session ID
* command mode and selected stages
* force flags or stage selection flags
* stages considered during this run
* stages executed during this run
* stages skipped during this run and reasons
* run-local output paths
* promoted output paths
* logs
* reports
* generated configs
* timings
* errors
* non-secret subprocess invocation metadata
Run manifests are primarily for debugging, auditability, and archive history.
## 8. Idempotency and Resume Behavior
The idempotency boundary is the session, not the run.
By default:
```bash
narratio run --session-id 2026-05-07
```
should consult the session manifest and skip stages that are already succeeded and not stale.
If all stages are already complete, the command should execute zero stages and report that the session is already complete.
Forced execution creates a new run record but updates session-level durable state only for stages that actually succeed and promote outputs.
Examples:
```bash
narratio run --session-id 2026-05-07 --force
```
Creates a new run and attempts to re-execute the selected/default stage set.
```bash
narratio run-stage --session-id 2026-05-07 analyze --force
```
Creates a sparse run that executes only `analyze`, then promotes updated analysis artifacts if successful.
```bash
narratio resume --session-id 2026-05-07
```
Uses the session manifest to determine what remains incomplete or stale. Resume does not need to resume the same `run_id` unless the implementation explicitly supports resuming an interrupted active run.
## 9. Artifact Resolution Contract
Narratio should provide a first-class artifact registry and resolver.
The resolver maps symbolic artifact source names to canonical session-level paths and manifest output kinds.
Stages and adapters should not hardcode path fragments when resolving cross-stage inputs. They should ask the artifact resolver for the current durable artifact by ID.
### 9.1 Canonical Artifact IDs
Preferred artifact IDs should be namespaced:
```text
narratio.transcript.merged
narratio.transcript.polished
narratio.transcript.full
narratio.transcript.trimmed
narratio.bounds.session
narratio.artifact.session_recap
```
Recommended initial registry:
| Artifact ID | Canonical Path | Producer Stage | Output Kind | Meaning |
| --------------------------------- | ------------------------------- | -------------- | ------------------------ | ------------------------------------- |
| `narratio.transcript.merged` | `transcripts/merged.json` | `merge` | `transcript_merged` | Deterministic Seriatim merge. |
| `narratio.transcript.polished` | `transcripts/processed.json` | `polish` | `transcript_processed` | Full Audita-polished transcript. |
| `narratio.transcript.full` | `transcripts/normalized.json` | `normalize` | `transcript_normalized` | Preferred full normalized transcript. |
| `narratio.transcript.trimmed` | `transcripts/trimmed.json` | `trim` | `transcript_trimmed` | Gameplay-only transcript. |
| `narratio.bounds.session` | `artifacts/session_bounds.json` | `trim` | `session_bounds` | Trim bounds selected for the session. |
| `narratio.artifact.session_recap` | `artifacts/session_recap.md` | `analyze` | `artifact_session_recap` | Generated session recap. |
### 9.2 Backward-Compatible Aliases
Existing source names should remain supported:
| Legacy Source | Preferred Artifact ID |
| ----------------------- | ------------------------------ |
| `processed_transcript` | `narratio.transcript.polished` |
| `normalized_transcript` | `narratio.transcript.full` |
| `trimmed_transcript` | `narratio.transcript.trimmed` |
These aliases may be supported silently for v1.0. Documentation should prefer namespaced IDs.
### 9.3 Resolver Behavior
Artifact resolution should follow this order:
1. Normalize aliases to canonical artifact IDs.
2. Look for a current output reference in the session manifest.
3. Fall back to the canonical session-level path.
4. If the artifact is required, fail clearly if missing.
5. If the artifact is optional and missing, omit it from the downstream invocation.
6. Validate the artifact using the expected content validator.
7. Return a resolved artifact record containing ID, path, producer stage, output kind, and provenance.
Example conceptual result:
```json
{
"id": "narratio.transcript.trimmed",
"path": "/var/lib/narratio/work/dilfs/2026-05-07/transcripts/trimmed.json",
"producer_stage": "trim",
"producer_run_id": "20260517T174748Z-abcd1234",
"output_kind": "transcript_trimmed",
"content_type": "application/json"
}
```
### 9.4 Artifact Validation
Transcript artifacts must be valid JSON with a top-level `segments` array.
Markdown/text artifacts must exist and be non-empty when required.
Bounds artifacts must match the expected bounds schema and refer to segment IDs in the same transcript ID space used by the trim stage.
Validation should happen before a resolved artifact is passed to another stage or external subprocess.
## 10. Analyze Stage Implications
The analyze stage should consume artifacts through the artifact resolver.
Preferred Scriptorium config shape:
```yaml
scriptorium:
artifacts:
session_recap:
enabled: true
prompt_id: "dnd.session_recap"
output_path: "artifacts/session_recap.md"
inputs:
transcript:
source: "narratio.transcript.trimmed"
required: true
```
Additional artifacts can choose different transcript tiers:
```yaml
scriptorium:
artifacts:
table_summary:
enabled: true
prompt_id: "dnd.table_summary"
output_path: "artifacts/table_summary.md"
inputs:
transcript:
source: "narratio.transcript.full"
required: true
```
For v1.0, Narratio does not need a generic DAG engine. It may execute configured analyze artifacts in deterministic order and allow later artifacts to consume earlier artifacts only when that relationship is explicit and unambiguous.
Rules:
* Artifact inputs resolve from current session-level durable state.
* Outputs are first written run-locally.
* Successful analyze outputs are promoted to session-level `artifacts/` paths.
* Manifest output refs record the producing run ID.
* Optional inputs are omitted when unavailable.
* Required missing inputs fail before invoking Scriptorium.
## 11. Archive Alignment
Local workspace semantics should mirror archive semantics.
Local session-level durable paths:
```text
work/{campaign}/{session}/transcripts/trimmed.json
work/{campaign}/{session}/artifacts/session_recap.md
work/{campaign}/{session}/current/manifest.json
work/{campaign}/{session}/current/run_id.txt
work/{campaign}/{session}/runs/{run_id}/...
```
should map naturally to remote archive paths:
```text
{root_prefix}/campaigns/{campaign}/sessions/{session}/transcripts/trimmed.json
{root_prefix}/campaigns/{campaign}/sessions/{session}/artifacts/session_recap.md
{root_prefix}/campaigns/{campaign}/sessions/{session}/current/manifest.json
{root_prefix}/campaigns/{campaign}/sessions/{session}/current/run_id.txt
{root_prefix}/campaigns/{campaign}/sessions/{session}/runs/{run_id}/...
```
The archive stage should publish run records and promoted current artifacts consistently with the local model.
`current/run_id.txt` remains the effective commit marker for the archived current session state.
## 12. Path Helper Requirements
All code should use centralized path helpers for workspace paths.
Stage code should not manually assemble durable cross-stage paths using raw string joins except through the path model.
Recommended helper surface:
```text
SessionRoot(campaignID, sessionID)
SessionManifestPath(campaignID, sessionID)
SessionCurrentDir(campaignID, sessionID)
SessionTranscriptsDir(campaignID, sessionID)
SessionArtifactsDir(campaignID, sessionID)
SessionReportsDir(campaignID, sessionID)
SessionLogsDir(campaignID, sessionID)
SessionConfigDir(campaignID, sessionID)
RunsDir(campaignID, sessionID)
RunRoot(campaignID, sessionID, runID)
RunManifestPath(campaignID, sessionID, runID)
RunStageDir(campaignID, sessionID, runID, stage)
RunStageOutputsDir(campaignID, sessionID, runID, stage)
RunStageLogsDir(campaignID, sessionID, runID, stage)
RunStageReportsDir(campaignID, sessionID, runID, stage)
RunStageConfigDir(campaignID, sessionID, runID, stage)
CanonicalArtifactPath(campaignID, sessionID, artifactID)
```
Path helpers should enforce safe relative paths for configured output paths:
* reject absolute paths unless explicitly allowed for a particular config field
* reject `..` traversal
* normalize separators
* preserve deterministic output paths
## 13. Directory Creation Policy
Directory creation should be centralized and idempotent.
Recommended policy:
* `prepare` ensures the baseline session directory structure exists.
* Every stage also calls shared layout helpers to ensure its required run-local directories exist before writing.
* `run-stage` should not depend on a prior `prepare` invocation merely to create folders.
* Missing directories should be created with appropriate permissions.
* Directory creation should not imply stage success.
This provides consistent layout while keeping direct stage execution robust.
## 14. Cleanup and Retention
Cleanup must preserve the distinction between durable session state and run history.
Workspace cleanup after successful archive may remove selected local directories only according to explicit configuration.
Potential retention policies:
```text
keep_all_runs
keep_failed_runs
keep_last_n_runs
delete_run_after_success
```
For v1.0, conservative retention is preferred:
* Do not delete durable session-level outputs unless explicitly requested.
* Do not delete failed run directories by default.
* If cleanup is enabled, remove only documented run-scoped or spool-scoped paths.
* Local development audio inputs must never be deleted by workspace cleanup.
## 15. Migration From Existing Layout
Existing installations may currently use a simpler path such as:
```text
{workspace.root}/work/{session_id}/manifest.json
```
The v1.0 layout introduces campaign-aware session roots:
```text
{workspace.root}/work/{campaign_id}/{session_id}/manifest.json
```
Migration options:
1. Best-effort automatic discovery of legacy session manifests.
2. A one-time migration command.
3. Clear release notes requiring users to move or regenerate workspace state.
For v1.0, it is acceptable to require explicit migration if the user base is small and the archive contains the authoritative durable outputs. However, the application should fail clearly when it detects an ambiguous legacy layout rather than silently creating duplicate state.
## 16. Documentation Updates Required
The following documentation should be updated to reflect this architecture:
* `README.md`
* `docs/architecture.md`
* a dedicated workspace/run-history document, such as this file
* S3/archive documentation
* analyze/artifact configuration documentation
* example pipeline files
Documentation should consistently use the following terms:
| Term | Meaning |
| ---------------- | ---------------------------------------------------------------------- |
| Session | Durable domain object and idempotency boundary. |
| Run | Execution attempt that may update session state. |
| Durable output | Canonical current session-level output. |
| Run-local output | Output produced inside a specific run directory before promotion. |
| Promotion | Validated copy/rename from run-local output to durable session output. |
| Session manifest | Current durable state of the session. |
| Run manifest | Execution record for a particular run. |
| Artifact ID | Symbolic source name resolved by the artifact registry. |
## 17. Architectural Invariants
The following invariants should hold after implementation:
1. `session_id` remains the idempotency boundary for normal operator commands.
2. `run_id` identifies an execution attempt, not the primary durable workspace.
3. Session-level canonical artifacts are the default inputs for downstream stages.
4. Run-local outputs are promoted only after validation.
5. A session's current durable state may be composed of outputs from multiple runs.
6. Sparse run directories are valid and expected.
7. The session manifest records current stage/artifact state and producer run IDs.
8. The run manifest records what happened during one invocation.
9. Artifact consumers resolve symbolic artifact IDs through a registry/resolver.
10. Local workspace semantics mirror S3 archive semantics.
11. Directory creation is centralized and idempotent.
12. Stage code uses path helpers rather than ad hoc path construction.
13. Forced upstream reruns invalidate downstream stage success unless downstream stages are rerun successfully.
14. Cleanup never removes durable session outputs or local development inputs unless explicitly configured to do so.
## 18. Implementation Guidance
A practical implementation sequence is:
1. Add this architecture document.
2. Add or revise path model helpers for session roots, run roots, stage directories, and canonical artifact paths.
3. Introduce session manifest versus run manifest concepts.
4. Route stage outputs through run-local directories.
5. Add promotion helpers with validation and atomic writes.
6. Update existing stages to promote durable outputs to session-level canonical paths.
7. Add artifact registry and resolver.
8. Update analyze to use artifact IDs and aliases.
9. Add simple downstream stale invalidation for forced upstream reruns.
10. Align archive/local path behavior and documentation.
11. Update examples and README.
12. Add tests for idempotency, sparse forced runs, promotion, manifest provenance, and artifact resolution.
This sequence intentionally avoids introducing a generic DAG engine. The v1.0 goal is a clear, deterministic, stage-oriented orchestrator with stable session-level outputs and inspectable run history.

View File

@@ -0,0 +1,356 @@
# Go Project Documentation Policy
## Purpose
Project documentation must help four audiences:
1. users who need to run the application;
2. administrators/operators who need to configure and operate it;
3. developers who need to understand and change it safely;
4. LLM coding agents that need clear scope, boundaries, and invariants.
Docs should be accurate, concise, task-oriented, and organized by audience. Prefer links to canonical docs over repetition.
## Core Rules
### 1. Keep docs concise
Each document should cover a defined scope and only the essentials for that scope.
Avoid:
- long background explanations;
- repeated reference material;
- implementation detail in user-facing docs;
- aspirational language outside roadmap docs;
- verbose examples where one minimal example is clearer.
### 2. Document only implemented behavior outside roadmap files
Unimplemented, planned, aspirational, experimental, or future work may be described only under:
- `docs/roadmap/`
No other documentation file, including `README.md`, should describe code, features, modules, stages, commands, config fields, or behaviors that do not currently exist.
If a feature is partial, non-roadmap docs may describe only the implemented portion and its current boundary.
### 3. Use canonical homes
Each type of information should have one canonical location.
Canonical homes:
- project purpose and quickstart: `README.md`
- development principles: `docs/architecture.md`
- configuration reference: `docs/config.md`
- CLI reference: `docs/cli.md`
- operations and recovery: `docs/operations.md`
- troubleshooting: `docs/troubleshooting.md`
- implemented internals: `docs/internal/`
- future work: `docs/roadmap/`
- contributor workflow: `docs/development.md`
- copyable examples: `examples/`
Other files should summarize briefly and link to the canonical source.
### 4. Keep examples real
Examples should be valid, maintained, and free of secrets.
Where practical:
- example configs should load successfully;
- example commands should match real CLI syntax;
- important examples should be covered by tests.
## Documentation Profiles
All projects require:
- `README.md`
- `docs/architecture.md`
Additional docs depend on the project.
### Small library
Recommended:
- `docs/development.md`, if contributor conventions are non-obvious
### Simple CLI
Required:
- `docs/cli.md`
Recommended:
- `docs/development.md`
### Config-driven CLI
Required:
- `docs/cli.md`
- `docs/config.md`
Recommended:
- `examples/`
- `docs/development.md`
### Stateful or operator-facing application
Required:
- `docs/cli.md`, if CLI-based
- `docs/config.md`, if config-driven
- `docs/operations.md`
Recommended:
- `docs/troubleshooting.md`
- `examples/`
- `docs/development.md`
### Modular, staged, service-oriented, or orchestration application
Required:
- `docs/cli.md`, if CLI-based
- `docs/config.md`, if config-driven
- `docs/operations.md`
- `docs/internal/`
- `docs/development.md`
Recommended:
- `docs/troubleshooting.md`
- validated examples under `examples/`
## Required Documents
### README.md
**Audience:** users, administrators, operators
The README is the outward-facing project orientation page.
It should include, in order:
1. concise description;
2. elevator pitch;
3. shortest useful command or usage example;
4. links to targeted docs.
The README should be short. It is not a manual.
The “shortest useful command” means the simplest command that performs the projects core use case. (It does not mean `app --help`.)
### docs/architecture.md
**Audience:** developers, LLM coding agents
`docs/architecture.md` is required for every project.
It is an inward-facing development policy document. It should describe how the project is intended to be built and changed.
It should include:
- project shape;
- core design principles;
- package and boundary philosophy;
- state/persistence philosophy, if applicable;
- external integration philosophy, if applicable;
- error-handling and logging principles;
- testing expectations;
- documentation expectations;
- architectural invariants;
- explicit non-goals, if useful.
For small projects, this file may be brief. It may simply state that the project is intentionally narrow, monolithic, and dependency-light.
### docs/config.md
**Audience:** administrators, operators, advanced users
Required for applications with configuration files.
It should include, in order:
1. config file locations and discovery precedence;
2. minimal working config;
3. production-oriented config;
4. full configuration reference;
5. secrets handling, if applicable;
6. links to maintained examples.
The full configuration reference should be canonical.
### docs/cli.md
**Audience:** users, administrators, operators
Required for CLI applications.
It should include, in order:
1. shortest useful command;
2. command overview;
3. complete flag reference;
4. common workflows;
5. diagnostic or recovery commands, if applicable.
Explain when commands are useful, not just their syntax.
### docs/operations.md
**Audience:** administrators, operators
Required for applications that maintain state, support resume behavior, run multiple stages, write durable artifacts, use remote storage, or require recovery procedures.
It should cover:
- normal workflow;
- filesystem layout;
- remote storage layout, if applicable;
- logs and manifests;
- resume/retry behavior;
- cleanup behavior;
- archive/backup behavior;
- safe recovery procedures;
- operational caveats.
### docs/troubleshooting.md
**Audience:** administrators, operators
Recommended once recurring failure modes exist.
Each entry should include:
- symptom;
- likely cause;
- diagnostic command or inspection step;
- safe fix;
- relevant links.
### docs/development.md
**Audience:** developers, LLM coding agents
Required for projects maintained by humans and LLM coding agents.
It should include:
- repository layout;
- build/test commands;
- coding conventions;
- dependency policy;
- how to add config fields;
- how to add CLI flags;
- how to add stages/modules/adapters, if applicable;
- how to update examples;
- documentation update expectations.
### docs/internal/
**Audience:** developers, LLM coding agents
Required for modular, staged, service-oriented, or orchestration projects.
This directory describes implemented internal components. It is not the roadmap.
Use one file per major component where useful.
Each component doc should include:
1. purpose;
2. inputs and outputs;
3. boundaries;
4. config fields used;
5. external adapters used;
6. state or manifest behavior, if applicable;
7. skip/resume behavior, if applicable;
8. failure behavior;
9. tests to inspect before changing;
10. architectural invariants.
### docs/roadmap/
**Audience:** maintainers, developers, LLM coding agents
This is the only place for planned, future, aspirational, experimental, or unimplemented work.
Roadmap docs should clearly distinguish:
- proposed work;
- accepted plans;
- deferred ideas;
- rejected ideas;
- implementation prompts or task breakdowns, if useful.
Roadmap docs should not be confused with current behavior.
### docs/integrations/
**Audience:** developers, LLM coding agents
Required for projects that depend on external CLIs, APIs, services, protocols, or file formats where the integration contract is important to maintain.
This directory contains concise, versioned reference notes for external integration contracts. It should document only the parts of the external system that this project actually uses.
Use one file per integration where useful.
## Examples Directory
Projects with non-trivial configuration or workflows should include `examples/`.
Useful examples include:
- minimal working config;
- production-oriented config;
- full annotated config;
- local development config;
- remote/object-storage config;
- minimal session/input file.
Examples should be valid, maintained, tested when practical, and linked from relevant docs.
## Security and Privacy
Docs and examples must not include:
- real API keys;
- tokens;
- passwords;
- private keys;
- private environment dumps;
- sensitive user data;
- raw private transcripts;
- private infrastructure details unless intentionally public.
Document secret-handling mechanisms, not actual secret values.
## Maintenance Rules
When docs change, verify the affected behavior.
Where practical:
- load example config files in tests;
- test CLI examples or command parser behavior;
- validate documented flags against real flags;
- remove stale references;
- update links after renames;
- keep roadmap content out of non-roadmap docs.
If documentation and code disagree, fix the documentation and/or open a roadmap item; do not leave aspirational behavior in current-behavior docs.
Documentation is complete only when it matches the current code.
## Documentation Change Checklist
Before merging documentation changes, verify:
- README is concise and orientation-focused.
- `docs/architecture.md` describes development principles.
- Future work appears only under `docs/roadmap/`.
- User-facing docs avoid unnecessary internals.
- Developer-facing docs preserve boundaries and invariants.
- Config examples match the schema.
- CLI examples match real commands and flags.
- Defaults appear in the canonical config reference.
- No secrets or private data are included.
- Links are accurate.

View File

@@ -0,0 +1,15 @@
# Integration Documentation Index
## Audience
Developers and LLM coding agents changing Narratio's external integration contracts.
## Scope
Implemented-only reference notes for the external systems Narratio currently integrates with.
## Integration Docs
- `audita.md`: Audita adapter invocation and validation contract.
- `seriatim.md`: Seriatim normalize/merge/trim adapter contract.
- `scriptorium.md`: Scriptorium run/render adapter contract.
## Canonical Owner
`docs/integrations/` is the canonical home for external integration reference notes per `docs/documentation/policy.md`.

View File

@@ -1,96 +1,66 @@
# Audita Subprocess Operations
# Integration: audita
This document describes how parent processes should invoke `audita process` safely in production orchestration.
## Purpose
Define Narratio's adapter contract for transcript polishing via Audita CLI subprocess execution.
## Recommended command form
## Inputs and Outputs
Inputs (`audita.PolishRequest`):
- merged transcript path
- glossary path
- output processed transcript path
- optional report path (required when report enabled)
- work dir
- generated config path
- stdout/stderr log paths
- optional module/model/base URL and concurrency knobs
Use explicit file outputs for orchestrated runs:
Outputs (`audita.PolishResult`):
- processed transcript path
- optional report path
- generated config path
- stdout/stderr log paths
- exit code, duration, invoked binary
- adapter metadata
```sh
audita process <transcript.json> \
--transcript-description "Brief context that may help resolve ambiguous terms." \
--glossary <glossary.yaml> \
--output <output-transcript.json> \
--report-json <report.json>
```
## Boundaries
Owns:
- Deterministic CLI argument construction for `audita process`
- Environment bridging for API credentials
- Invocation config emission
- Output validation for processed transcript and report
Additional flags that may be situationally appropriate:
- `--config <path>` to select an explicit versioned config file.
- `--output-schema <bare-segments|audita-v1>` to select transcript output shape.
- `--work-dir <dir>` to control diagnostics location.
- `--work-dir-retention <always|auto|never>` to control retained run directories.
- `--total-llm-concurrency`, `--proposal-llm-concurrency`, and `--validation-llm-concurrency` when orchestration needs to set explicit LLM throughput controls.
- `--modules ...` only when intentionally overriding the default sequence.
Does not own:
- Upstream/downstream stage orchestration
- Credential sourcing policy beyond required env-var presence check
For config-driven orchestration, validate config files in CI/preflight:
## Config Fields Used
Via `pipeline.audita.*` mapped in app/stage wiring:
- `binary`, `timeout`, `llm_api_key_env`, `modules`, `base_url`, `model`
- `transcript_description`, `config_path`, `output_schema`, `work_dir_retention`
- `total_llm_concurrency`, `proposal_llm_concurrency`, `validation_model`, `validation_llm_concurrency`, `report`
```sh
audita config validate --config <path>
```
## External Adapters Used
- Shared subprocess helper (`internal/adapters/subprocess`) to run CLI and capture logs.
## Stdout behavior
## State and Manifest Behavior
- No direct manifest writes.
- Stage-level metadata records adapter provenance and credential-present signal.
- Generated invocation YAML is written when `GeneratedConfigPath` is provided.
- With `--output`: stdout is expected to be empty on success.
- Without `--output`: stdout contains transcript JSON only on success.
- Report JSON is never written to stdout.
## Skip and Resume Behavior
- Adapter has no skip/resume logic. Stage/runner controls this.
## Stderr behavior
## Failure Behavior
- Constructor validation fails on invalid binary/timeout/schema/concurrency/URL values.
- Run fails on missing required paths, missing required credential env var, subprocess errors, invalid processed JSON shape, or invalid report JSON.
- Failures preserve stdout/stderr paths in returned result metadata.
- Success path should be quiet or minimal human-readable logs.
- Failure path writes concise human-readable errors.
- When a diagnostics run directory exists, failure stderr includes its path.
- Prompt/response diagnostic payloads are not streamed to stderr.
## Tests to Inspect Before Changing
- `internal/adapters/audita/subprocess_test.go`
- `internal/adapters/audita/fake_test.go`
- `internal/stage/polish_test.go`
## Output file behavior
- `--output` writes transcript JSON in the selected output schema to the provided path.
- Output write failures return nonzero and surface actionable errors.
- The command does not silently ignore output write errors.
## Report JSON behavior
- `--report-json` writes a machine-readable process report to the requested path.
- Run-directory `report.json` is written independently under diagnostics.
- Best-effort failure reports are emitted when possible without masking the primary failure.
- Report write failures return nonzero with clear stderr messaging.
- Report diagnostics metadata references run-directory artifacts including utilization diagnostics and correction ledger paths when available.
## Diagnostics directory behavior
- Each run creates (when possible) a per-run diagnostics directory.
- Typical artifacts include transcript, normalization, chunking, invocation, effective config, LLM diagnostics, `utilization-diagnostics.json`, `correction-ledger.json`, `report.json`, and `error.log` on failure.
- Failed runs retain diagnostics.
- Under `auto` retention, successful runs with skipped/rejected corrections are retained; clean successful runs may be removed.
## Exit codes
- `0`: success.
- Nonzero: failure (input/schema/config/module/LLM/runtime/output/report/diagnostics errors).
Treat any nonzero as a failed subprocess invocation.
## Timeout and cancellation
- Runtime operations propagate context cancellation and request timeouts through LLM/scheduler paths.
- On cancellation or timeout, the process exits nonzero and should not hang.
- If diagnostics were initialized before failure, failure artifacts remain available for debugging.
## Secret redaction expectations
API keys and configured secret values are redacted from:
- reports (`--report-json` and run-dir `report.json`);
- diagnostics artifacts (including effective config and LLM interaction artifacts);
- surfaced adapter/runtime errors;
- test fixtures and regression outputs.
Parent-process logs should still avoid printing raw environment variables.
## Parent-process pipe guidance
To avoid deadlocks in orchestrators:
- always read both stdout and stderr concurrently when invoking as a subprocess;
- prefer file outputs (`--output`, `--report-json`) for machine workflows;
- treat stderr as human-readable diagnostics, not structured data;
- parse structured results from output/report files.
For Go callers, prefer `exec.CommandContext` with explicit timeout/cancellation and buffered/streamed readers for both pipes.
## Architectural Invariants
- Processed output must be valid JSON with top-level `segments` array.
- When report is enabled, report output must be valid JSON.
- If `llm_api_key_env` is configured, credential must be present in environment.

View File

@@ -1,339 +1,64 @@
# Narratio -> Scriptorium CLI Integration
## 1. Purpose
This document defines how Narratio should invoke Scriptorium through the **public CLI**.
This is a **subprocess integration contract**, not an internal Go API contract.
## 2. Assumptions
- `scriptorium` is installed and available on `PATH`.
- Scriptorium is configured with `config.yml`.
- `config.yml` provides `prompt_dir`, `profile_dir`, and `schema_dir` as needed.
- Prompt and profile libraries are already deployed for the environment.
- Narratio provides prepared artifact files (for example polished transcript, glossary, previous recap, campaign notes).
- Initial integration is synchronous subprocess execution.
- Narratio remains the orchestrator.
In normal operation, Narratio does not need to pass `--prompt-dir` and `--profile-dir` if they are supplied by Scriptorium config.
Narratio may pass `--config <PATH>` when it must use a non-default Scriptorium config file.
## 3. Core Commands Narratio May Call
Primary commands for subprocess integration:
- `scriptorium run`
- `scriptorium render`
For production generation, use `scriptorium run`.
`scriptorium render` is for debugging, dry-runs, test assertions, and validating command construction without LLM execution.
Note: `scriptorium serve` and HTTP API exist, but they are not the initial integration path.
## 4. Command Selection Guidance
- Use `run` to generate an output artifact.
- Use `render` to inspect the prepared prompt and effective settings without calling the LLM.
- Use `render --format json` when Narratio/tests need structured prepare output.
## 5. Recommended `run` Invocation Shape
Production shape:
```bash
scriptorium run \
--prompt <prompt_id> \
--input transcript=<processed-transcript-path> \
--out <output-artifact-path>
```
Common optional additions:
- `--config <path>`: use a specific Scriptorium config file.
- `--profile <profile_id>`: override prompt default profile.
- `--var name=value` (repeatable): small metadata values.
- `--input name=path` (repeatable): additional named artifacts.
- `--timeout <duration>`: per-run timeout override.
- Runtime model override flags (`--llm-base-url`, `--model`, etc.) only for exceptional/operator-directed cases.
## 6. Recommended `render` Invocation Shape
Human-readable debug shape:
```bash
scriptorium render \
--prompt <prompt_id> \
--input transcript=<processed-transcript-path> \
--format text
```
Structured debug/test shape:
```bash
scriptorium render \
--prompt <prompt_id> \
--input transcript=<processed-transcript-path> \
--format json \
--out <render-debug-path>
```
`render` does **not** call the LLM, does **not** validate model output, and does **not** perform repair.
## 7. Inputs
- Pass inputs as repeated `--input name=path` flags.
- `name` must match the Prompt Definition input name.
- Prefer absolute paths, or paths relative to a working directory controlled by Narratio.
- Pass Audita output as the primary transcript input.
- Additional inputs may include glossary, previous recap, campaign notes, event logs, final state maps, or other prompt-specific artifacts.
- Scriptorium reads input files directly; Narratio does not need to inline file content for CLI use.
## 8. Variables
Use repeated `--var name=value` for small metadata values.
Typical examples:
- `session_date`
- `session_id`
- `campaign_name`
- `previous_session_id`
- `output_kind`
Large content belongs in input files, not `--var` values.
## 9. Prompt IDs and Output Artifact Types
Narratio should treat prompt IDs as configuration, not hardcoded business logic.
Narratio config may map stage/output names to prompt IDs, for example:
- session recap prompt
- structured event extraction prompt
- glossary suggestion prompt
- player-facing summary prompt
Prompt IDs used by Narratio should come from the deployed Scriptorium prompt library.
## 10. Profiles
- Prompts may declare `default_profile`.
- Narratio may omit `--profile` to use prompt default profile.
- Narratio may pass `--profile` to force profile selection.
- This enables environment/profile selection like `local-fast`, `local-quality`, `frontier`, `batch`, or test profiles.
- Profile names should generally be Narratio configuration values.
## 11. Runtime Overrides
Supported runtime override flags:
- `--llm-base-url`
- `--model`
- `--api-key-env`
- `--temperature`
- `--max-tokens`
- `--top-p`
- `--timeout`
Guidance:
- Keep normal model/runtime settings in Execution Profiles.
- Use runtime overrides only for explicit per-run exceptions, tests, or operator overrides.
- Never pass raw API keys on the command line.
- `--api-key-env` names an environment variable; Narratio must ensure that variable is set in subprocess environment.
## 12. Config Behavior
- Default config path: `/etc/scriptorium/config.yml`.
- `--config <PATH>` overrides default path.
- Missing default config is allowed by Scriptorium.
- If `--config` is provided explicitly, the file must exist and be valid.
- CLI flags override `config.yml`.
- `config.yml` overrides built-in application defaults.
Narratio can either:
- rely on system default config path, or
- carry an explicit config path and pass `--config`.
## 13. Environment Handling
Subprocess environment recommendations:
- Pass through required API-key environment variables referenced by `api_key_env`.
- Do not pass raw API keys as CLI arguments.
- Avoid logging full environment dumps.
- Capture stdout and stderr separately.
- Use a controlled working directory.
- Prefer absolute artifact paths.
## 14. Output Handling
For `scriptorium run`:
- Use `--out` when Narratio needs durable artifact files.
- Without `--out`, artifact content is written to stdout.
- Preferred orchestration pattern: always use `--out`, then treat the file as stage output artifact.
- Capture stderr for diagnostics.
For `scriptorium render`:
- Use `--out` to store render diagnostics.
- Use `--format json` when tests need to inspect selected profile, effective runtime settings, input hashes, prompt hash, and rendered messages.
## 15. Exit Status and Errors
Current CLI behavior (verified from implementation/tests):
- `0`: success.
- `1`: runtime/parse/config/load/render/generation/IO error.
- `2`: run completed but output validation failed (`ValidationFailed`).
Additional details:
- On `run`, output artifact write happens before exit code selection. If validation fails, artifact may still be written and exit code is `2`.
- `stderr` carries both errors and normal run summary output; non-empty stderr alone does not imply failure.
- `render` returns `0` on success and `1` on failures.
Narratio should treat non-zero exit codes as failed stage execution, but may record generated artifact paths if a run exited `2` and output file exists.
## 16. Recommended Narratio Integration Pattern
1. Build CLI args from Narratio stage configuration.
2. Use subprocess context cancellation/timeout.
3. Pass absolute input paths.
4. Pass `--out` to a session-scoped artifact path.
5. Add `--var` metadata values.
6. Optionally add `--config`.
7. Optionally add `--profile`.
8. Ensure required API-key env vars are present.
9. Run subprocess synchronously.
10. Capture stdout/stderr separately.
11. On success, store output artifact path and invocation metadata in stage artifacts.
12. On failure, store exit code and stderr diagnostics in stage status.
## 17. Suggested Narratio Configuration Shape
Illustrative `pipeline.yml` shape:
```yaml
scriptorium:
binary: scriptorium
config_path: /etc/scriptorium/config.yml
timeout: 10m
render_debug: false
artifacts:
session_recap:
enabled: true
prompt_id: dnd.session_recap
profile_id: local-quality # optional
output_path: artifacts/session_recap.md
timeout: 10m
render_debug: false # optional artifact override
inputs:
transcript:
source: trimmed_transcript
required: true
previous_recap:
source: previous_session_artifact
artifact: session_recap
path: "" # optional
required: false
vars:
session_id: true
session_date: true
campaign_name: true
previous_session_id: true
output_kind: session_recap
```
The key idea: map Narratio artifact names to prompt ID, optional profile, expected inputs, vars, and output destination.
## 18. Testing Strategy for Narratio Integration
- Use `scriptorium render --format json` to verify command construction without LLM calls.
- Use dedicated test prompt/profile libraries for integration tests.
- Use small fixture transcripts.
- Verify missing-input failure behavior.
- Verify prompt `default_profile` behavior.
- Verify explicit `--profile` override behavior.
- Verify `--config` behavior (default and explicit).
- Verify output file creation when `--out` is used.
- Verify stderr capture on failures.
- Avoid real API keys in tests.
## 19. Security and Privacy Notes
- Never pass raw API keys on command line.
- Do not log full rendered prompts by default; transcripts may contain sensitive content.
- Avoid logging prompt content unless explicit debug mode is enabled.
- Treat generated artifacts as potentially sensitive.
- Use session-scoped, access-controlled output paths.
- `api_key_env` names should come from environment management, not embedded secrets.
## 20. Initial D&D Artifact Generation Examples
These are examples only. Use prompt IDs from the deployed prompt library.
Session recap:
```bash
scriptorium run \
--prompt dnd.session_recap \
--input transcript=/work/session-42/transcript.polished.md \
--input glossary=/work/session-42/glossary.yml \
--out /work/session-42/artifacts/session_recap.md
```
Structured events:
```bash
scriptorium run \
--prompt dnd.structured_events \
--input transcript=/work/session-42/transcript.polished.md \
--out /work/session-42/artifacts/structured_events.json
```
Glossary suggestions:
```bash
scriptorium run \
--prompt dnd.glossary_suggestions \
--input transcript=/work/session-42/transcript.polished.md \
--input previous_recap=/work/session-41/artifacts/session_recap.md \
--out /work/session-42/artifacts/glossary_suggestions.md
```
Player-facing summary:
```bash
scriptorium run \
--prompt dnd.player_summary \
--input transcript=/work/session-42/transcript.polished.md \
--input structured_events=/work/session-42/artifacts/structured_events.json \
--out /work/session-42/artifacts/player_summary.md
```
## 21. Non-Goals
Initial Narratio integration should not:
- call Scriptorium internal Go packages
- use HTTP API as the primary path
- expect Scriptorium to read S3 refs directly
- make Scriptorium responsible for Narratio stage state
- make Scriptorium responsible for notification
- require Scriptorium to understand D&D workflow semantics beyond prompt definitions
## 22. Future Extension Notes
Possible later extensions:
- HTTP API integration
- S3 artifact references if Scriptorium adds S3 reader support
- richer render diagnostics and policy controls
- token budgeting/prompt-size checks
- batch execution if Scriptorium later adds batch support
# Integration: scriptorium
## Purpose
Define Narratio's adapter contract for Scriptorium artifact generation and render-debug subprocess invocations.
## Inputs and Outputs
Inputs:
- `RunArtifactRequest`: binary, config path, prompt/profile IDs, input map, vars map, timeout, output path, logs/config paths, optional API env and working dir
- `RenderArtifactRequest`: same core fields for render mode
Outputs (`ArtifactResult`):
- output path
- stdout/stderr log paths
- generated config path
- exit code and duration
- command mode (`run` or `render`)
- prompt/profile provenance
- validation failure signal
- adapter metadata
## Boundaries
Owns:
- Deterministic CLI arg construction for `scriptorium run` and `scriptorium render`
- Common request validation
- Invocation config emission
- Output existence/non-empty checks
- Validation-failure mapping for run exit code 2
Does not own:
- Artifact selection policy (`analyze` stage)
- Bounds semantic validation (`trim` stage)
## Config Fields Used
Via `pipeline.scriptorium.*` and stage-level artifact config:
- `binary`, `config_path`, `timeout`, `render_debug`
- artifact-level `prompt_id`, `profile_id`, `timeout`, `inputs`, `vars`, `output_path`
## External Adapters Used
- Shared subprocess helper (`internal/adapters/subprocess`).
## State and Manifest Behavior
- No direct manifest writes.
- Stage metadata records adapter outputs and command mode.
- Generated invocation YAML is written when requested.
## Skip and Resume Behavior
- Adapter has no skip/resume logic. Stage/runner controls execution.
## Failure Behavior
- Request validation fails for missing binary/prompt/output, invalid timeout, invalid input/var names, or missing required API env var.
- Subprocess errors bubble with command context.
- `run` exit code 2 is treated as `ValidationFailed=true` and surfaced as error by calling stage.
- Successful subprocess still fails if output file is missing/empty.
## Tests to Inspect Before Changing
- `internal/adapters/scriptorium/subprocess_test.go`
- `internal/adapters/scriptorium/fake_test.go`
- `internal/stage/analyze_test.go`
- `internal/stage/trim_test.go`
## Architectural Invariants
- Both modes require explicit timeout > 0.
- Input/var maps are sorted into deterministic CLI argument order.
- Run-mode validation failures are represented explicitly, not silently skipped.

View File

@@ -1,403 +1,60 @@
# seriatim
`seriatim` merges per-speaker WhisperX-style JSON transcripts into a single JSON transcript that preserves speaker identity and chronological order.
The current implementation supports the `merge` command. It reads one or more input JSON files, optionally maps each input file to a canonical speaker using `speakers.yml`, sorts all segments by timestamp, detects and resolves overlaps when word-level timing is available, assigns consecutive numeric `id` values, and writes a merged JSON artifact.
## Usage
Run from source:
```sh
go run ./cmd/seriatim merge \
--input-file samples/raw/2026-04-19-Eric_Rakestraw.json \
--input-file samples/raw/2026-04-19-Mike_Brown.json \
--output-file merged.json
```
Optional report output:
```sh
go run ./cmd/seriatim merge \
--input-file eric.json \
--input-file mike.json \
--output-file merged.json \
--report-file report.json
```
## CLI
```text
seriatim merge [flags]
```
Global flags:
| Flag | Description |
| --- | --- |
| `--help` | Show command help. |
| `--version` | Show application version. Local builds default to `dev`; release builds inject the release version. |
`merge` flags:
| Flag | Required | Default | Description |
| --- | --- | --- | --- |
| `--input-file` | Yes | none | Input transcript JSON file. Repeat once per speaker/input file. |
| `--output-file` | Yes | none | Merged transcript JSON output path. |
| `--report-file` | No | none | Optional report JSON output path. |
| `--speakers` | No | none | Speaker map YAML file. When omitted, input file basenames are used as speaker labels. |
| `--autocorrect` | No | none | Autocorrect rules YAML file. When omitted, the default `autocorrect` module leaves text unchanged. |
| `--input-reader` | No | `json-files` | Input reader module. |
| `--output-modules` | No | `json` | Comma-separated output modules. |
| `--output-schema` | No | `seriatim-intermediate` | JSON output contract. Allowed values are `seriatim-minimal`, `seriatim-intermediate`, and `seriatim-full`. If omitted, the runtime default is used; consumers that depend on a specific shape should set this explicitly. |
| `--preprocessing-modules` | No | `validate-raw,normalize-speakers,trim-text` | Comma-separated preprocessing modules, evaluated in order. |
| `--postprocessing-modules` | No | `detect-overlaps,resolve-overlaps,backchannel,filler,resolve-danglers,coalesce,detect-overlaps,autocorrect,assign-ids,validate-output` | Comma-separated postprocessing modules, evaluated in order. |
| `--coalesce-gap` | No | `3.0` | Maximum same-speaker gap in seconds for `coalesce`; also used as the `resolve-overlaps` context window. Must be a non-negative float. |
Environment variables:
| Environment Variable | Default | Description |
| --- | --- | --- |
| `SERIATIM_OUTPUT_SCHEMA` | `seriatim-intermediate` | Output schema used when `--output-schema` is not explicitly provided. Allowed values are `seriatim-minimal`, `seriatim-intermediate`, and `seriatim-full`. The CLI flag takes precedence. |
| `SERIATIM_OVERLAP_WORD_RUN_GAP` | `1.0` | Maximum gap in seconds between adjacent timed words when `resolve-overlaps` builds word-run replacement segments. Must be a positive float. |
| `SERIATIM_OVERLAP_WORD_RUN_REORDER_WINDOW` | `1.0` | Near-start window in seconds for ordering replacement word runs shortest-first. Must be a positive float. |
| `SERIATIM_BACKCHANNEL_MAX_DURATION` | `2.0` | Maximum duration in seconds for `backchannel` classification. Must be a positive float. |
| `SERIATIM_FILLER_MAX_DURATION` | `1.25` | Maximum duration in seconds for `filler` classification. Must be a positive float. |
## Input JSON Format
Each input file must be valid JSON with a top-level `segments` array. The current parser accepts the WhisperX segment subset needed for merging:
```json
{
"segments": [
{
"start": 1.25,
"end": 3.5,
"text": "Hello there.",
"words": [
{"word": "Hello", "start": 1.25, "end": 1.55, "score": 0.98},
{"word": "there.", "start": 1.7, "end": 2.0}
]
}
]
}
```
Required segment fields:
- `start`: number, must be `>= 0`.
- `end`: number, must be `>= start`.
- `text`: string.
Optional word fields:
- `words`: array of word timing objects.
- `words[].word`: string.
- `words[].start`: optional number, must be `>= 0` when present.
- `words[].end`: optional number, must be `>= start` when present with `start`.
- `words[].score`: optional number.
- `words[].speaker`: optional raw speaker label string.
Word-level timing is preserved internally for overlap resolution. If a word is missing `start` or `end`, seriatim keeps the word text, emits a warning in the optional report, and does not use that word as a timing anchor. Word timing is not emitted in the final JSON artifact.
## Speaker Map Format
`speakers.yml` maps input files to canonical speaker names using ordered substring rules:
This file is optional. If `--speakers` is omitted, `seriatim` uses each input file basename as the segment speaker label.
```yaml
match:
- speaker: "Eric Rakestraw"
match:
- "Eric_Rakestraw"
- "Eric"
- speaker: "Mike Brown"
match:
- "Mike_Brown"
- "mb"
```
For each `--input-file`, `seriatim` takes the file basename and evaluates the rules in order. The first rule with a matching substring wins, and no later rules are evaluated.
For example, this input:
```text
samples/raw/2026-04-19-Eric_Rakestraw.json
```
matches this rule because the basename contains `Eric_Rakestraw`:
```yaml
- speaker: "Eric Rakestraw"
match:
- "Eric_Rakestraw"
```
Important details:
- Matching is against the input file basename, not the full path.
- Matching is case-insensitive.
- Rules are evaluated from first to last.
- Each rule must have a non-empty `speaker`.
- Each rule must have at least one non-empty `match` string.
- Duplicate speaker names are invalid.
- Every input file must match at least one rule or the command fails.
Deprecated old format:
```yaml
inputs:
eric.json:
speaker: "Eric Rakestraw"
```
The old `inputs:` direct mapping format is no longer supported.
## Output JSON Format
`--output-modules json` controls the writer. `--output-schema` controls the JSON contract that writer serializes.
The named schemas are stable public contracts. If a consumer depends on a specific shape, it should request that schema explicitly at runtime. The runtime default selection may change in a future release.
The `seriatim-intermediate` schema is the current default selection when neither `--output-schema` nor `SERIATIM_OUTPUT_SCHEMA` is set. It stays close to the minimal schema, but adds optional `categories` on each segment:
```json
{
"metadata": {
"application": "seriatim",
"version": "dev",
"output_schema": "seriatim-intermediate"
},
"segments": [
{
"id": 1,
"start": 1.25,
"end": 3.5,
"speaker": "Eric Rakestraw",
"text": "Hello there.",
"categories": ["backchannel"]
}
]
}
```
The `seriatim-full` schema uses the full seriatim envelope:
```json
{
"metadata": {
"application": "seriatim",
"version": "dev",
"input_reader": "json-files",
"input_files": ["eric.json", "mike.json"],
"preprocessing_modules": ["validate-raw", "normalize-speakers", "trim-text"],
"postprocessing_modules": ["detect-overlaps", "resolve-overlaps", "backchannel", "filler", "resolve-danglers", "coalesce", "detect-overlaps", "autocorrect", "assign-ids", "validate-output"],
"output_modules": ["json"]
},
"segments": [
{
"id": 1,
"source": "eric.json",
"source_segment_index": 0,
"speaker": "Eric Rakestraw",
"start": 1.25,
"end": 3.5,
"text": "Hello there.",
"overlap_group_id": 1
},
{
"id": 2,
"source": "eric.json",
"source_ref": "word-run:1:1:1",
"derived_from": ["eric.json#0"],
"speaker": "Eric Rakestraw",
"start": 2.0,
"end": 2.5,
"text": "Resolved word run",
"categories": ["backchannel"]
}
],
"overlap_groups": [
{
"id": 1,
"start": 1.25,
"end": 4.0,
"segments": ["eric.json#0", "mike.json#0"],
"speakers": ["Eric Rakestraw", "Mike Brown"],
"class": "unknown",
"resolution": "unresolved"
}
]
}
```
The `seriatim-minimal` schema emits minimal metadata and compact ordered segments:
```json
{
"metadata": {
"application": "seriatim",
"version": "dev",
"output_schema": "seriatim-minimal"
},
"segments": [
{
"id": 1,
"start": 1.25,
"end": 3.5,
"speaker": "Eric Rakestraw",
"text": "Hello there."
}
]
}
```
Minimal output intentionally omits categories, overlap groups, source/provenance fields, and pipeline configuration metadata.
Intermediate output intentionally omits overlap groups and source/provenance fields, but keeps optional `categories` and minimal metadata.
Segments are sorted deterministically by:
```text
(start, end, source, source_segment_index/source_ref, speaker)
```
Final segment IDs are assigned after sorting and start at `1`.
The public Go output contract is available from:
```go
import "gitea.maximumdirect.net/eric/seriatim/schema"
```
The same package embeds machine-readable JSON Schemas in `schema/full-output.schema.json`, `schema/intermediate-output.schema.json`, and `schema/minimal-output.schema.json`. The default `validate-output` postprocessor validates the selected output shape and verifies final segment IDs are present, sequential, and start at `1`.
## Overlap Detection
The default postprocessing pipeline detects overlapping segment groups.
Overlap behavior:
- A strict timing overlap is required: `next.start < current_group_end`.
- Segments that only touch at a boundary are not grouped.
- Groups require at least two distinct speakers.
- Transitive overlaps are grouped together.
- Segments in detected groups receive `overlap_group_id`.
- `overlap_groups[].segments` contains stable references in `source#source_segment_index` format.
- `class` is currently `unknown`.
- `resolution` is `unresolved` until `resolve-overlaps` replaces the group.
## Overlap Resolution
The default postprocessing pipeline runs `detect-overlaps`, then `resolve-overlaps`, then `backchannel`, then `filler`, then `resolve-danglers`, then `coalesce`, then a second `detect-overlaps` pass.
For each detected overlap group, `resolve-overlaps` uses preserved WhisperX word timing to build smaller word-run replacement segments:
- The resolution window expands the detected overlap group by `--coalesce-gap` seconds on both sides.
- Nearby same-speaker context segments are included when they intersect the expanded window and their start or end is within `--coalesce-gap` of the original overlap boundary.
- Once a segment is selected for replacement, all timed words from that segment participate in word-run construction; the window controls segment selection, not per-word clipping.
- Context segments that are part of another detected overlap group are not pulled into the current group.
- Untimed words are included in replacement text in original word order when nearby timed words create a replacement run.
- Untimed words do not affect replacement segment start/end times or word-run gap splitting.
- Words for the same speaker are merged into one run when the gap between adjacent words is no greater than `SERIATIM_OVERLAP_WORD_RUN_GAP`.
- The default word-run gap is `1.0` seconds.
- Set `SERIATIM_OVERLAP_WORD_RUN_GAP` to a positive number of seconds to override the default.
- Near-start replacement word runs are reordered so shorter segments come first when adjacent starts are within `SERIATIM_OVERLAP_WORD_RUN_REORDER_WINDOW`.
- The default word-run reorder window is `1.0` seconds.
- Set `SERIATIM_OVERLAP_WORD_RUN_REORDER_WINDOW` to a positive number of seconds to override the default.
- Replacement segment text is built by joining word text with single spaces.
- Replacement segments include `source_ref` and `derived_from`.
- Replacement segments omit `source_segment_index` because they are derived from one or more original segments.
- Resolved overlap groups are removed before the second detection pass.
- Replacement segments are left without `overlap_group_id` until the second detection pass annotates any remaining overlap.
- If a speaker has no usable word timing in a group, that speaker's original segment is kept.
- If no speakers in a group have usable word timing, the original group and annotations remain unchanged.
## Backchannels
The default pipeline runs `backchannel` before `coalesce`. It tags short acknowledgement segments with:
```json
"categories": ["backchannel"]
```
Backchannel matching is case-insensitive, ignores punctuation for matching and word-count purposes, trims surrounding whitespace, and requires a matching acknowledgement phrase, no more than three whitespace-delimited words, and duration no greater than `SERIATIM_BACKCHANNEL_MAX_DURATION` seconds. The default maximum duration is `2.0` seconds.
## Fillers
The default pipeline runs `filler` after `backchannel` and before `coalesce`. It tags short filler utterances with:
```json
"categories": ["filler"]
```
Filler matching is case-insensitive, ignores punctuation for matching and word-count purposes, trims surrounding whitespace, and requires only filler tokens such as `um`, `uh`, `er`, `erm`, `ah`, `eh`, `hmm`, `mm`, or repeated combinations of those tokens. Matching segments must contain no more than three whitespace-delimited words and have duration no greater than `SERIATIM_FILLER_MAX_DURATION` seconds. The default maximum duration is `1.25` seconds.
## Dangler Resolution
The default pipeline runs `resolve-danglers` before `coalesce` and before the second overlap detection pass. It repairs short derived fragments when they share provenance with a nearby segment:
- Dangling-end fragments have no more than two words and end in punctuation.
- Dangling-start fragments have no more than two words.
- Matching uses same-speaker segments with any shared `derived_from` value.
- Merged segments use `source_ref` values such as `resolve-danglers:1`, keep the target segment's transcript position, and union `derived_from`.
## Coalescing
The default pipeline runs `coalesce` after `resolve-danglers` and before the second overlap detection pass. It merges adjacent same-speaker segments in the transcript's current order when `next.start - current.end <= --coalesce-gap`.
Coalesced segments use `source_ref` values such as `coalesce:1`, include `derived_from`, and omit `source_segment_index`.
Different-speaker backchannel and filler segments do not block coalescing of surrounding same-speaker segments. Same-speaker backchannel and filler segments are merged normally when they are within `--coalesce-gap`. When same-speaker segments are coalesced, any `backchannel` or `filler` category from the merged inputs is dropped from the coalesced segment.
## Autocorrect
Autocorrect is included in the default postprocessing pipeline. If `--autocorrect` is omitted, the module leaves transcript text unchanged and records a skip event in the optional report.
Enable corrections by passing `--autocorrect`:
```sh
go run ./cmd/seriatim merge \
--input-file input.json \
--autocorrect autocorrect.yml \
--output-file merged.json
```
`autocorrect.yml` format:
```yaml
autocorrect:
- target: "Hrank"
match:
- "hrank"
- "Frank"
- target: "Mike Brown"
match:
- "Mike Pat"
```
Matching behavior:
- Matching is case-sensitive.
- Matches apply only to whole tokens, not substrings inside larger words.
- Punctuation and whitespace can surround a match.
- Multi-word and hyphenated matches are supported.
- Duplicate match strings are invalid, including duplicates across separate rules.
## Current Limitations
- Only JSON input is supported.
- Overlap resolution depends on WhisperX word timing; groups without usable word timing remain unresolved.
- Alternate output formats are not implemented yet.
## Release Builds
Local builds record version metadata as `dev`. Release builds should inject the release version with `ldflags`:
```sh
go build -ldflags "-X gitea.maximumdirect.net/eric/seriatim/internal/buildinfo.Version=v1.0.0" ./cmd/seriatim
```
# Integration: seriatim
## Purpose
Define Narratio's adapter contract for merge, normalize, and trim subprocess invocations of Seriatim.
## Inputs and Outputs
Inputs:
- `MergeRequest`: raw/normalized transcript inputs, output path, optional report, speaker/autocorrect paths, logs/config
- `NormalizeRequest`: input transcript, output path, schema, optional report, timeout/log/config
- `TrimRequest`: input transcript, output path, keep selector, timeout/log/config
Outputs:
- `MergeResult`, `NormalizeResult`, `TrimResult` with output paths, logs/config paths, exit code, duration, binary provenance, and metadata.
## Boundaries
Owns:
- Validated deterministic CLI invocation construction
- Optional env tuning propagation for merge
- Invocation config file emission
- JSON output validation
Does not own:
- Transcript input selection/promotion logic (stage-owned)
- Bounds computation (scriptorium/trim-stage-owned)
## Config Fields Used
Via `pipeline.seriatim.*` mapped in app/stage wiring:
- `binary`, `timeout`, `output_schema`, `coalesce_gap`, `report`
- `env.overlap_word_run_gap`
- `env.overlap_word_run_reorder_window`
- `env.backchannel_max_duration`
- `env.filler_max_duration`
## External Adapters Used
- Shared subprocess helper (`internal/adapters/subprocess`).
## State and Manifest Behavior
- No direct manifest writes.
- Stage metadata consumes adapter result fields and preserves generated config/log references.
## Skip and Resume Behavior
- Adapter has no skip/resume logic. Runner controls stage execution.
## Failure Behavior
- Constructor fails for invalid binary/timeout/output-schema/coalesce-gap.
- Merge fails on missing output path/inputs/report path (if enabled), subprocess errors, invalid merged output JSON, invalid report JSON.
- Normalize fails on missing input/output, invalid schema, subprocess errors, invalid normalized output JSON shape, invalid report JSON.
- Trim fails on missing input/output/keep selector, subprocess errors, invalid trimmed output JSON shape.
## Tests to Inspect Before Changing
- `internal/adapters/seriatim/subprocess_test.go`
- `internal/adapters/seriatim/fake_test.go`
- `internal/stage/merge_test.go`
- `internal/stage/normalize_test.go`
- `internal/stage/trim_test.go`
## Architectural Invariants
- Supported output schemas are limited to `seriatim-minimal`, `seriatim-intermediate`, `seriatim-full`.
- Normalize/trim outputs must include `segments` arrays.
- Merge/normalize/trim all route through deterministic subprocess invocation.

29
docs/internal/README.md Normal file
View File

@@ -0,0 +1,29 @@
# Internal Documentation Index
## Audience
Developers and LLM coding agents changing Narratio internals.
## Scope
Implementation-accurate contracts for workspace/state, manifests, stages, artifact resolution, adapter boundaries, and restore command behavior.
## Component Docs
- `adapters.md`: external adapter map, runtime wiring, and boundary ownership.
- `storage.md`: remote storage backend contracts and object-store invariants.
- `manifest.md`: session/run manifest schemas, lifecycle transitions, and persistence semantics.
- `artifacts.md`: built-in artifact registry, runtime artifact catalog, and source-resolution behavior.
- `workspace.md`: local state model, manifests, run-local layout, promotion, and cleanup invariants.
- `command-restore.md`: restore command discovery/planning/execution/reporting contract.
- `stage-prepare.md`: input materialization and provenance capture.
- `stage-transcribe.md`: WhisperX transcript generation.
- `stage-merge.md`: Seriatim normalization + merge.
- `stage-polish.md`: Audita transcript polishing.
- `stage-normalize.md`: post-polish normalization.
- `stage-trim.md`: bounds-driven transcript trimming.
- `stage-analyze.md`: dependency-ordered Scriptorium artifact generation for selected configured artifacts.
- `stage-archive.md`: archive upload and current-pointer publish contract.
## External Integration Notes
- `../integrations/README.md`: canonical location for external integration contracts (`audita.md`, `seriatim.md`, `scriptorium.md`).
## Canonical Owner
`docs/internal/` is the canonical home for implemented internals per `docs/documentation/policy.md`.

79
docs/internal/adapters.md Normal file
View File

@@ -0,0 +1,79 @@
# Internal: Adapters
## Purpose
Describe the external adapter boundaries used by Narratio stages and app orchestration, including default runtime wiring.
## Inputs and outputs
Inputs:
- Stage requests passed through adapter interfaces (for example transcription, merge/normalize/trim, polish, artifact generation, object-store operations, notifications).
- Resolved config values used to construct default adapters.
Outputs:
- Adapter-specific result structs (paths, metadata, status/attempt info, duration/exit details).
- Adapter errors returned to stage/app orchestration.
## Boundaries
Owns:
- Transport/process/SDK details at system boundaries (`HTTP`, subprocess CLI invocation, AWS SDK calls).
- Request/response contracts in `internal/adapters/*` packages.
Does not own:
- Stage sequencing, skip/force/resume decisions.
- Manifest transition logic.
- Canonical workspace path policy.
## Config fields used
Default wiring and adapter calls consume:
- `pipeline.whisperx.*`
- `pipeline.seriatim.*`
- `pipeline.audita.*`
- `pipeline.scriptorium.*`
- `pipeline.storage.*` and `pipeline.archive.*` (object-store construction/gating)
- `pipeline.notification.*` (sender boundary exists; placeholder behavior today)
## External adapters used
Runtime env boundary fields (`internal/stage.Env`):
- `whisperx.Client`
- `seriatim.Runner`
- `audita.Runner`
- `scriptorium.Runner`
- `storage.ObjectStore`
- `notify.Sender`
- `analyzer.Runner`
Current execution usage:
- Actively used by implemented stages: `WhisperX`, `Seriatim`, `Audita`, `Scriptorium`, `ObjectStore`, `Notifier`.
- Present but not used by implemented stage set: `Analyzer`, legacy `storage.Backend`.
Default construction in app runner:
- Auto-constructed when not injected: WhisperX HTTP client, Seriatim subprocess runner, Audita subprocess runner, Scriptorium subprocess runner, object store (only when needed), and `notify.NoopSender`.
- Callers can inject test/fake implementations through `app.RunOptions.Env`.
## State and manifest behavior
- Adapters do not directly mutate session/run manifests.
- Stages and runner own manifest writes and stage status transitions.
- Adapter outputs are persisted indirectly through stage result mapping (outputs/logs/generated configs/metadata).
## Skip and resume behavior
- No adapter-level skip/resume semantics.
- Skip/resume/force behavior is decided by app runner using manifest stage state.
## Failure behavior
- Adapter constructors validate config-derived values and fail early on invalid required inputs.
- Adapter run-time failures are returned to stage code with boundary context and are recorded as stage failures by runner logic.
- Subprocess adapters preserve stdout/stderr and generated-config paths to aid diagnosis.
## Tests to inspect before changing
- `internal/adapters/whisperx/http_test.go`
- `internal/adapters/seriatim/subprocess_test.go`
- `internal/adapters/audita/subprocess_test.go`
- `internal/adapters/scriptorium/subprocess_test.go`
- `internal/adapters/storage/*_test.go`
- `internal/adapters/notify/fake_test.go`
- `internal/adapters/analyzer/fake_test.go`
- `internal/app/runner_test.go`
## Architectural invariants
- Stage code depends on adapter interfaces, not transport-specific implementation types.
- External SDK-specific types remain inside adapter implementations.
- Default app wiring must remain deterministic and overrideable via injected env dependencies.

View File

@@ -0,0 +1,87 @@
# Internal: Artifacts
## Purpose
Define Narratio's artifact identity and resolution model for built-in transcript/bounds artifacts and runtime-configured analyze artifacts.
## Inputs and outputs
Inputs:
- artifact sources from config/runtime (`pipeline.scriptorium.artifacts.*.inputs.*.source`)
- session paths and optional session manifest stage outputs
- runtime artifact catalog state for configured artifact sources
Outputs:
- resolved local artifact path and provenance (`ResolvedSessionArtifact`)
- runtime catalog entries for planned/executable/available artifacts
- validation errors for unsupported, missing, or invalid artifact sources
## Boundaries
Owns:
- built-in artifact registry and content validation rules
- runtime artifact catalog for configured artifact source IDs
- source resolution behavior for built-in and configured artifact sources
Does not own:
- artifact generation (stages produce files)
- manifest transition policy
- archive promotion behavior
## Config fields used
- `pipeline.scriptorium.artifacts.<name>.enabled`
- `pipeline.scriptorium.artifacts.<name>.output_path`
- `pipeline.scriptorium.artifacts.<name>.inputs.<key>.source`
## External adapters used
- none
## State and manifest behavior
Built-in registry entries:
| Artifact ID | Canonical file | Producer stage | Output kind |
| --- | --- | --- | --- |
| `narratio.transcript.merged` | `transcripts/merged.json` | `merge` | `transcript_merged` |
| `narratio.transcript.polished` | `transcripts/processed.json` | `polish` | `transcript_processed` |
| `narratio.transcript.full` | `transcripts/normalized.json` | `normalize` | `transcript_normalized` |
| `narratio.transcript.trimmed` | `transcripts/trimmed.json` | `trim` | `transcript_trimmed` |
| `narratio.bounds.session` | `artifacts/session_bounds.json` | `trim` | `session_bounds` |
Runtime catalog entries include built-ins and configured `narratio.artifact.<name>` sources.
Catalog states:
- `planned`: source is registered and known for this run
- `executable`: configured artifact is selected for analyze execution
- `available`: artifact has a usable file path (generated this run or reused from disk)
Resolution behavior:
- built-in sources resolve via manifest producer outputs first, then canonical fallback path
- configured `narratio.artifact.<name>` sources resolve through runtime catalog availability
- configured source lookup requires catalog context
Configured artifact provenance values:
- `generated.current_analyze_run`
- `filesystem.disabled_artifact_output`
Content validation:
- transcript built-ins: JSON with top-level `segments` array
- bounds built-in: valid JSON
- configured artifacts: non-empty text file
## Skip and resume behavior
- resolver and catalog have no direct skip/resume decisions
- stage/runner skip-resume behavior consumes catalog/resolver results
## Failure behavior
- unsupported source -> source validation error
- known source unavailable -> `ErrSessionArtifactNotFound`
- configured source without catalog -> resolution error
- resolved file with invalid content -> validation error
## Tests to inspect before changing
- `internal/artifacts/artifact_resolver_test.go`
- `internal/artifacts/catalog_test.go`
- `internal/stage/analyze_test.go`
- `internal/config/scriptorium_test.go`
## Architectural invariants
- built-in IDs are static and registry-backed
- configured artifact IDs are runtime-derived (`narratio.artifact.<name>`) and catalog-backed
- built-in/source resolution remains deterministic and validation-gated

View File

@@ -0,0 +1,86 @@
# Internal: Command Restore
## Purpose
Define the implemented `narratio restore` command contract: committed remote-state discovery, deterministic planning, safe file installation, conflict policy, and restore reporting.
## Inputs and outputs
Inputs:
- CLI flags: `--config`, `--session`, `--session-id`, `--dry-run`, `--force`, `--include-audio`.
- Resolved/validated `pipeline.yml` and `session.yml`.
- Configured remote object store.
- Remote committed current-state markers (`current/run_id.txt`, `current/manifest.json`).
Outputs:
- Dry-run summary to stdout (plan + counts).
- Non-dry-run completion summary to stdout.
- Local durable session files restored under canonical session root.
- Non-dry-run restore report at `reports/restore-latest.json`.
## Boundaries
Owns:
- Restore command flag parsing and command wiring.
- Remote current-state discovery and identity validation.
- Restore plan construction and conflict classification.
- Restore execution for planned downloads.
- Restore report model and persistence.
Does not own:
- Stage execution orchestration (`run`, `resume`, `run-stage`).
- Archive publish behavior (owned by archive stage).
- Storage transport implementation details (owned by storage adapters).
## Config fields used
- Config/session discovery and templating fields consumed by all commands.
- `pipeline.workspace.root` (local restore target root).
- `pipeline.storage.*` (remote backend + archive identity derivation).
- `pipeline.storage.s3.*` identity components used by archive prefix helpers.
- `session.session_id`
- `session.campaign`
## External adapters used
- `storage.ObjectStore` for `Exists`, `List`, `Download`.
- `artifacts.Store` (`LocalStore`) for layout and session lock management.
- `manifest.LocalStore` for manifest decode/validation and identity checks.
## State and manifest behavior
- Restore is not a pipeline run and does not create a run manifest.
- Restore uses committed remote current state only:
- `current/run_id.txt` must exist and be non-empty.
- `current/manifest.json` must decode and match requested session/campaign.
- Non-dry-run writes restore files to canonical session paths.
- Manifest install behavior:
- validated before replacement.
- installed last among download actions.
- existing local manifest is preserved if restored manifest validation/install fails.
- Non-dry-run report persists summary/action status metadata in `reports/restore-latest.json`.
## Skip and resume behavior
- Restore does not participate in stage skip/resume decisions.
- Restore provides durable local state so subsequent stage commands can resume or rerun based on restored manifest state.
- Dry-run is read-only and returns plan output only.
## Failure behavior
- Fails when storage backend is unavailable or archive identity cannot be resolved.
- Fails when remote current pointer/manifest is missing or invalid.
- Fails when remote manifest identity mismatches requested campaign/session.
- Fails on local conflicts unless `--force` is set.
- Fails fast on session lock acquisition conflict for non-dry-run execution.
- On execution failure, previously installed files remain; no rollback is performed.
## Tests to inspect before changing
- `internal/app/restore_test.go`
- `internal/app/restore_discovery_test.go`
- `internal/app/restore_plan_test.go`
- `internal/app/restore_execution_test.go`
- `internal/app/restore_workflow_test.go`
- `internal/artifacts/archive_identity_test.go`
## Architectural invariants
- Restore relies on centralized archive identity/key helpers (`internal/artifacts`) rather than ad hoc key building.
- `current/run_id.txt` is the remote commit marker; restore must not infer committed state from incidental files.
- Local path mapping is traversal-safe and constrained to session root.
- Restore scope is deterministic and path-classified:
- include `manifest.json`, `transcripts/**`, `artifacts/**`
- include `audio/**` only with `--include-audio`
- exclude `runs/**`, `logs/**`, `reports/**`, `config/**`, `inputs/**`
- Command remains standalone; no implicit `run --restore` behavior.

81
docs/internal/manifest.md Normal file
View File

@@ -0,0 +1,81 @@
# Internal: Manifest
## Purpose
Describe Narratio's durable execution state model for session-level and run-level manifests, including lifecycle transitions and persistence behavior.
## Inputs and outputs
Inputs:
- Session identity and run identity from app orchestration.
- Stage transition events and stage result payloads.
Outputs:
- Session manifest at `{workspace.root}/work/{campaign}/{session_id}/manifest.json`.
- Run manifest at `{workspace.root}/work/{campaign}/{session_id}/runs/{run_id}/manifest.json`.
## Boundaries
Owns:
- Manifest schemas (`Manifest`, `RunManifest`, stage records, error records, input/artifact records).
- Stage status/action transition methods.
- Persistent store contract (`manifest.Store`) and local JSON store implementation.
Does not own:
- Stage implementation details.
- Path construction policy outside manifest file persistence calls.
- CLI command behavior.
## Config fields used
Manifest package itself does not read config directly.
Manifest identity fields are populated by app/stage orchestration from:
- `session.session_id`
- `session.campaign`
- `pipeline.workspace.root`
- `pipeline.storage.s3.*` (when archive/S3 identity is set)
## External adapters used
- No external service adapters.
- Uses local filesystem for persistence via `manifest.LocalStore`.
## State and manifest behavior
Session manifest model:
- Tracks durable per-session stage state and provenance (`pending`, `running`, `succeeded`, `failed`, `skipped`, `stale`, `interrupted`).
- Stores resolved inputs, durable artifacts, stage logs/config refs, and stage metadata.
Run manifest model:
- Tracks one invocation (`run_id`) with requested stages and force mode.
- Tracks per-stage action (`run` or `skip`) and per-stage status.
- Tracks overall run status (`running`, `succeeded`, `failed`).
Persistence behavior:
- Load validates required identity/timestamp fields and normalizes maps/records.
- Save updates `updated_at` and writes JSON atomically (temp file + rename).
- Session and run manifests are saved incrementally before/after stage transitions.
Relationship during execution:
- Runner updates both manifests for every stage transition.
- Session manifest is the durable pipeline-progress ledger.
- Run manifest is invocation history and audit record.
- Analyze stage outputs are persisted as `kind=scriptorium_artifact` with `source_id=narratio.artifact.<name>` for configured artifact identity.
## Skip and resume behavior
- Resume and skip decisions are based on session-manifest stage statuses.
- `--force` reruns selected stages and marks downstream succeeded stages as `stale` in session manifest.
- Run manifest records whether each stage was executed or skipped in that invocation.
## Failure behavior
- Stage failure marks both manifests failed for that stage and records error messages/timestamps.
- Save failures are returned immediately and fail the command.
- Invalid/malformed manifest files fail load with explicit validation/decode errors.
## Tests to inspect before changing
- `internal/manifest/manifest_test.go`
- `internal/manifest/run_manifest_test.go`
- `internal/manifest/store_test.go`
- `internal/app/runner_test.go`
- `internal/app/run_control_test.go`
- `internal/app/resume_run_stage_test.go`
## Architectural invariants
- Session manifest is authoritative for stage progression across invocations.
- Run manifest is invocation-scoped and never replaces session manifest as progress authority.
- Manifest writes are atomic and deterministic (JSON + newline, temp rename pattern).

View File

@@ -0,0 +1,84 @@
# Stage: analyze
## Purpose
Execute selected configured Scriptorium artifacts in deterministic dependency order and promote successful outputs to canonical session artifact paths.
## Inputs and Outputs
Inputs:
- configured artifact definitions from `pipeline.scriptorium.artifacts`
- selected artifact filter from runtime (`--artifacts`) when provided
- resolved artifact input sources declared per artifact (`inputs.*.source`)
- optional previous-session file inputs (`previous_session_artifact`)
Outputs:
- one promoted output file per executed configured artifact at that artifact's configured `output_path`
- stage metadata containing generated artifact entries and reused disabled-artifact entries
## Boundaries
Owns:
- runtime artifact catalog construction for analyze execution
- selected-artifact planning and dependency ordering
- per-artifact input resolution, var resolution, timeout/render-debug resolution
- Scriptorium run/render invocation for each selected artifact
- run-local output generation and canonical promotion
Does not own:
- transcript generation/processing stages
- archive promotion policy
- per-artifact resume semantics
## Config Fields Used
- `session.session_id`
- `session.campaign`
- `pipeline.workspace.root`
- `pipeline.scriptorium.binary`
- `pipeline.scriptorium.config_path`
- `pipeline.scriptorium.timeout`
- `pipeline.scriptorium.render_debug`
- `pipeline.scriptorium.artifacts.<name>.*`
- `enabled`
- `depends_on`
- `prompt_id`
- `profile_id`
- `timeout`
- `output_path`
- `render_debug`
- `inputs`
- `vars`
## External Adapters Used
- Scriptorium adapter:
- optional `RenderArtifact` (render debug)
- `RunArtifact` (artifact generation)
## State and Manifest Behavior
- If `pipeline.scriptorium` is absent, stage returns success metadata with `skipped=true`.
- If no artifacts are configured, stage returns success metadata with `skipped=true`.
- If zero artifacts are executable after `enabled` + `--artifacts` filtering, stage returns success metadata with `skipped=true`.
- Builds runtime catalog with built-ins and configured artifacts.
- Non-executable configured artifacts are marked available only when their configured output file exists and is valid on disk.
- Executes selected configured artifacts in topological order with deterministic tie-breaking.
- For each generated artifact, records metadata fields including `name`, `source_id`, `output_kind`, `path`, `prompt_id`, `profile_id`, and `provenance`.
- Reused disabled artifacts are recorded separately in `reused_artifacts` with provenance `filesystem.disabled_artifact_output`.
## Skip and Resume Behavior
- Runner-level skip applies when analyze is already `succeeded` and `--force` is not set.
- Analyze remains stage-scoped for resume/skip; there is no per-artifact resume state.
- `--artifacts` filters which configured artifacts are executable when analyze runs; it does not imply `--force`.
## Failure Behavior
- Fails on invalid dependency ordering, unavailable required configured inputs, invalid built-in input prerequisites, render/run adapter failures, validation-failed adapter results, or missing/empty outputs.
- Required configured dependency missing from catalog availability fails clearly before invocation.
- Optional missing inputs are omitted.
## Tests to Inspect Before Changing
- `internal/stage/analyze_test.go`
- `internal/artifacts/catalog_test.go`
- `internal/artifacts/artifact_resolver_test.go`
- `internal/adapters/scriptorium/subprocess_test.go`
## Architectural Invariants
- Configured artifacts are identified by `narratio.artifact.<name>` source IDs.
- Artifact-to-artifact references rely on explicit `depends_on` declarations validated in config.
- Generated analyze outputs are treated uniformly as Scriptorium artifacts.
- Successful outputs must exist and be non-empty before promotion.

View File

@@ -0,0 +1,68 @@
# Stage: archive
## Purpose
Publish run records and promoted session artifacts to object storage, then atomically advance the remote current pointer.
## Inputs and Outputs
Inputs:
- session manifest and prerequisite stage records
- run root contents under `runs/{run_id}/`
- promotion rules with artifact `source` IDs and archive `dest` paths (`archive.promote_artifacts`)
Outputs:
- uploaded run files under `{session_prefix}/runs/{run_id}/...`
- uploaded promoted artifacts under `{session_prefix}/...`
- `{session_prefix}/current/manifest.json`
- `{session_prefix}/current/run_id.txt` written last
## Boundaries
Owns:
- Archive enable/disable gate behavior
- Prerequisite stage success enforcement
- Run file collection and upload (excluding `audio/`)
- Promotion rule resolution and upload
- Commit pointer publish order
Does not own:
- Stage execution before archive
- Post-archive local cleanup policy execution (handled by app cleanup logic)
## Config Fields Used
- `pipeline.archive.enabled`
- `pipeline.archive.upload_run`
- `pipeline.archive.promote_artifacts`
- `pipeline.storage.s3.bucket`
- `pipeline.storage.s3.root_prefix`
- `pipeline.workspace.root`
- `session.campaign`
- `session.session_id`
## External Adapters Used
- Object storage backend (`env.ObjectStore`) for upload/list primitives.
## State and Manifest Behavior
- Requires `prepare`, `transcribe`, `merge`, `polish`, `normalize`, `trim`, and `analyze` status `succeeded`.
- Resolves bucket/prefix from manifest identity first, then config fallback.
- Writes metadata including:
- upload counts/paths
- `current_manifest_key`
- `current_run_id_key`
- `current_pointer_written`
- On skipped archive path, returns metadata with `skipped=true` and pointer not written.
## Skip and Resume Behavior
- Stage may self-skip (metadata skip) when archive disabled or run upload disabled.
- Runner-level skip also applies for previously succeeded stage unless forced.
## Failure Behavior
- Fails on missing prerequisite success, missing object store when required, missing run root, missing required promotion source, upload failures, or pointer write failures.
- Pointer semantics are fail-safe: `current/run_id.txt` is not written if prior required uploads fail.
## Tests to Inspect Before Changing
- `internal/stage/archive_test.go`
- `internal/app/post_archive_cleanup_test.go`
## Architectural Invariants
- Run upload excludes `audio/` subtree.
- `current/manifest.json` uploads before `current/run_id.txt`.
- `current/run_id.txt` is the remote publish commit marker.

View File

@@ -0,0 +1,63 @@
# Stage: merge
## Purpose
Normalize per-speaker raw transcripts and merge them into one merged transcript via Seriatim.
## Inputs and Outputs
Inputs:
- `transcripts/raw/*.json`
- `inputs/speakers.yml`
- `inputs/autocorrect.yml`
Outputs:
- `transcripts/merged.json`
- optional `artifacts/seriatim.report.json` (when report enabled)
## Boundaries
Owns:
- Raw transcript discovery/validation
- Per-input normalize calls to Seriatim
- Final merge call to Seriatim
- Run-local log/config/report path wiring
- Promotion of merged/report outputs to canonical paths
Does not own:
- Transcript polishing or downstream artifact generation
## Config Fields Used
- `session.session_id`
- `session.campaign`
- `pipeline.workspace.root`
- `pipeline.seriatim.binary`
- `pipeline.seriatim.timeout`
- `pipeline.seriatim.output_schema`
- `pipeline.seriatim.coalesce_gap`
- `pipeline.seriatim.report`
- `pipeline.seriatim.env.*`
## External Adapters Used
- Seriatim adapter:
- `Normalize` for each raw input
- `Run` for final merge
## State and Manifest Behavior
- Reads transcript inputs from transcribe stage outputs in manifest when present; falls back to canonical raw directory.
- Writes run-local outputs/logs/config under `runs/{run_id}/merge/...` when enabled.
- Promotes canonical merged transcript and optional report.
- Records normalized-input provenance and adapter metadata in stage metadata.
## Skip and Resume Behavior
- Runner-level skip applies when already succeeded and not forced.
- Forced rerun of this or upstream stages can stale downstream succeeded stages via runner invalidation.
## Failure Behavior
- Fails on missing/invalid raw transcripts, missing speakers/autocorrect files, normalize failure, merge failure, invalid merged output JSON, or invalid report JSON when enabled.
## Tests to Inspect Before Changing
- `internal/stage/merge_test.go`
- `internal/adapters/seriatim/subprocess_test.go`
## Architectural Invariants
- Merge consumes normalized forms of each raw transcript.
- Merged transcript must validate before promotion.
- Report output is optional and gated by config.

View File

@@ -0,0 +1,56 @@
# Stage: normalize
## Purpose
Normalize the processed transcript into a deterministic intermediate schema for trim and optionally emit a normalize report.
## Inputs and Outputs
Inputs:
- `transcripts/processed.json`
Outputs:
- `transcripts/normalized.json` (or configured normalize output path)
- optional `artifacts/seriatim.normalize.report.json`
## Boundaries
Owns:
- Processed transcript discovery/validation
- Normalize request construction and invocation
- Optional normalize report wiring
- Promotion of normalized transcript and optional report
Does not own:
- Bounds detection or segment trimming
## Config Fields Used
- `session.session_id`
- `session.campaign`
- `pipeline.workspace.root`
- `pipeline.normalize.output_path`
- `pipeline.normalize.output_schema`
- `pipeline.normalize.report`
- `pipeline.seriatim.binary`
- `pipeline.seriatim.timeout`
## External Adapters Used
- Seriatim adapter (`Normalize`).
## State and Manifest Behavior
- Reads processed transcript from polish outputs in manifest when present; falls back to canonical path.
- Uses run-local output/report/log/config paths when run layout is enabled.
- Promotes canonical normalized transcript and optional normalize report.
- Records adapter/result metadata including source path selection.
## Skip and Resume Behavior
- Runner-level skip applies when already succeeded and not forced.
- Forced reruns can stale downstream succeeded stages.
## Failure Behavior
- Fails on missing/invalid processed transcript, adapter error, invalid normalized output, or invalid report output when report enabled.
## Tests to Inspect Before Changing
- `internal/stage/normalize_test.go`
- `internal/adapters/seriatim/subprocess_test.go`
## Architectural Invariants
- Normalized output must validate as processed-transcript-compatible JSON (`segments` array required).
- Default normalize config is applied when `pipeline.normalize` is unset.

View File

@@ -0,0 +1,69 @@
# Stage: polish
## Purpose
Polish merged transcript with Audita and produce a processed transcript for downstream normalization/analyze.
## Inputs and Outputs
Inputs:
- `transcripts/merged.json`
- `inputs/glossary.yml`
Outputs:
- `transcripts/processed.json`
- optional `artifacts/audita.report.json` (when report enabled)
## Boundaries
Owns:
- Merged transcript discovery/validation
- Audita invocation request construction
- Run-local logs/config/work-dir/report wiring
- Promotion of processed transcript and optional report
Does not own:
- Upstream merge normalization
- Downstream normalize/trim/analyze logic
## Config Fields Used
- `session.session_id`
- `session.campaign`
- `pipeline.workspace.root`
- `pipeline.audita.binary`
- `pipeline.audita.timeout`
- `pipeline.audita.llm_api_key_env`
- `pipeline.audita.modules`
- `pipeline.audita.base_url`
- `pipeline.audita.model`
- `pipeline.audita.transcript_description`
- `pipeline.audita.config_path`
- `pipeline.audita.output_schema`
- `pipeline.audita.work_dir_retention`
- `pipeline.audita.total_llm_concurrency`
- `pipeline.audita.proposal_llm_concurrency`
- `pipeline.audita.validation_model`
- `pipeline.audita.validation_llm_concurrency`
- `pipeline.audita.report`
## External Adapters Used
- Audita adapter (`env.Audita.Run`).
## State and Manifest Behavior
- Reads merged transcript from merge manifest outputs when available; falls back to canonical merged path.
- Uses run-local output/report/log/config/scratch paths when run layout is enabled.
- Promotes canonical `transcripts/processed.json` and optional report.
- Records adapter invocation metadata, credential presence signal, and output provenance in stage metadata.
## Skip and Resume Behavior
- Runner-level skip applies when already succeeded and not forced.
- Forced rerun can stale downstream succeeded stages via runner invalidation.
## Failure Behavior
- Fails on missing/invalid merged transcript, missing glossary, adapter error, invalid processed output shape (`segments` array required), or invalid report JSON when enabled.
## Tests to Inspect Before Changing
- `internal/stage/polish_test.go`
- `internal/adapters/audita/subprocess_test.go`
## Architectural Invariants
- Processed transcript must contain a top-level `segments` array.
- Report behavior is strictly config-gated.
- Stage output canonicalization always ends at `transcripts/processed.json`.

View File

@@ -0,0 +1,74 @@
# Stage: prepare
## Purpose
Materialize all required session inputs into canonical local workspace paths and record input provenance in the session manifest.
## Inputs and Outputs
Inputs:
- `session.yml` (resolved session config)
- `pipeline.resolved.yml` (materialized from resolved pipeline config)
- `speakers.yml`
- `autocorrect.yml`
- `glossary.yml`
- audio source:
- local (`session.inputs.audio_dir` or `session.inputs.audio_files`), or
- S3 (`session.inputs.audio_s3.prefix`)
Outputs:
- `inputs/session.yml`
- `inputs/pipeline.resolved.yml`
- `inputs/speakers.yml`
- `inputs/autocorrect.yml`
- `inputs/glossary.yml`
- `audio/*.flac` in session workdir
- `manifest.Inputs` records with checksums and source metadata
## Boundaries
Owns:
- Input path resolution and validation
- Local copy/materialization of configs and audio files
- S3 audio download to run-scoped spool, then copy into work audio dir
Does not own:
- Transcript generation/processing
- Archive publish behavior
## Config Fields Used
- `session.session_id`
- `session.campaign`
- `session.inputs.speakers_file`
- `session.inputs.autocorrect_file`
- `session.inputs.glossary_file`
- `session.inputs.audio_dir`
- `session.inputs.audio_files`
- `session.inputs.audio_s3.prefix`
- `pipeline.workspace.root`
- `pipeline.spool.root`
- `pipeline.storage.s3.bucket`
- `pipeline.storage.s3.root_prefix`
## External Adapters Used
- Object storage backend (`env.ObjectStore`) for S3 audio list/download when `audio_s3` is configured.
## State and Manifest Behavior
- Ensures workspace layout exists.
- Writes resolved config and input files to canonical `inputs/` paths.
- Records all prepared inputs into `manifest.Inputs` (sorted deterministically by kind/path).
- For S3 audio, records `S3Bucket`, `S3Key`, `S3Size`, `S3ETag`, and `SpoolPath` in each audio input record.
## Skip and Resume Behavior
- Runner-level skip applies when stage already `succeeded` and `--force` is not set.
- Stage itself is deterministic/idempotent for unchanged inputs (`copyFileIfChanged`, `writeBytesIfChanged`).
## Failure Behavior
- Fails on missing required files, invalid audio source combinations, no discoverable `.flac` files, duplicate audio basenames, missing object store for S3 mode, or S3 list/download failures.
## Tests to Inspect Before Changing
- `internal/stage/prepare_test.go`
- `internal/app/session_cli_test.go`
- `internal/config/load_validate_test.go`
## Architectural Invariants
- `audio_dir`/`audio_files` and `audio_s3` are mutually exclusive.
- Audio files must be `.flac`.
- Canonical `inputs/*` and `audio/*` paths are the durable source for downstream stages.

View File

@@ -0,0 +1,58 @@
# Stage: transcribe
## Purpose
Generate per-speaker raw transcripts from prepared audio using WhisperX.
## Inputs and Outputs
Inputs:
- `audio/*.flac` prepared by `prepare`
Outputs:
- `transcripts/raw/<speaker>.json` for each input audio file
## Boundaries
Owns:
- Discovering prepared audio inputs
- Deriving speaker ids from audio basenames
- Parallel WhisperX invocation with bounded concurrency
- Validating produced JSON and promoting run-local outputs
Does not own:
- Transcript merge/polish/normalize/trim/analyze
## Config Fields Used
- `session.session_id`
- `session.campaign`
- `pipeline.workspace.root`
- `pipeline.whisperx.transcribe_url`
- `pipeline.whisperx.language`
- `pipeline.whisperx.timeout`
- `pipeline.whisperx.retries`
- `pipeline.whisperx.retry_delay`
- `pipeline.whisperx.concurrency`
## External Adapters Used
- WhisperX adapter (`env.WhisperX.Transcribe`).
## State and Manifest Behavior
- Uses run-local output paths under `runs/{run_id}/transcribe/outputs/...` when run layout is enabled.
- Validates each generated transcript JSON before promotion.
- Promotes canonical outputs to `transcripts/raw/*.json`.
- Records per-file metadata (attempts/status/duration/output path) in stage metadata.
## Skip and Resume Behavior
- Runner-level skip applies for previously succeeded stage unless forced.
- On forced upstream reruns, downstream succeeded stages can be marked `stale` by runner logic.
## Failure Behavior
- Fails if no prepared audio exists, duplicate speaker basenames are detected, adapter output path mismatches expected path, any output JSON is invalid, or one worker fails.
- Cancels in-flight workers after first terminal error.
## Tests to Inspect Before Changing
- `internal/stage/transcribe_test.go`
- `internal/app/whisperx_wiring_test.go`
## Architectural Invariants
- Speaker identity is derived from `.flac` basename and must be unique.
- Every successful speaker output must be valid JSON before promotion.
- Canonical raw transcript set is the only supported merge input surface.

View File

@@ -0,0 +1,75 @@
# Stage: trim
## Purpose
Optionally trim the normalized transcript to session bounds; always produce a durable trimmed transcript.
## Inputs and Outputs
Inputs:
- `transcripts/normalized.json`
Outputs:
- `transcripts/trimmed.json` (or configured trim output path)
- when trim enabled: `artifacts/session_bounds.json`
## Boundaries
Owns:
- Trim-enabled switch behavior
- Bounds generation via Scriptorium artifact run
- Bounds validation against normalized transcript
- Keep-selector derivation and Seriatim trim invocation
- Copy-through behavior when disabled or bounds indicate unchanged transcript
Does not own:
- Upstream normalization
- Downstream artifact analysis
## Config Fields Used
- `session.session_id`
- `session.campaign`
- `pipeline.workspace.root`
- `pipeline.trim.enabled`
- `pipeline.trim.output_path`
- `pipeline.trim.bounds.prompt_id`
- `pipeline.trim.bounds.profile_id`
- `pipeline.trim.bounds.timeout`
- `pipeline.trim.bounds.output_path`
- `pipeline.trim.bounds.transcript_input_name`
- `pipeline.trim.bounds.render_debug`
- `pipeline.trim.bounds.render_output_path`
- `pipeline.seriatim.binary`
- `pipeline.seriatim.timeout`
- `pipeline.scriptorium.binary`
- `pipeline.scriptorium.config_path`
- `pipeline.scriptorium.timeout`
## External Adapters Used
- Scriptorium adapter:
- optional `RenderArtifact` for bounds debug render
- `RunArtifact` for bounds output
- Seriatim adapter:
- `Trim` when bounds indicate trimming is required
## State and Manifest Behavior
- Reads normalized transcript from normalize manifest outputs when available; falls back to canonical path.
- Uses run-local outputs/logs/reports/config/scratch paths when run layout is enabled.
- Promotes canonical trimmed transcript; promotes session bounds when trim enabled.
- Records bounds diagnostics, trim action, keep selector, and adapter metadata.
## Skip and Resume Behavior
- Runner-level skip applies when already succeeded and not forced.
- Forced reruns can stale downstream succeeded stages.
- When `trim.enabled=false`, stage still succeeds by copying normalized to trimmed output.
## Failure Behavior
- Fails on missing/invalid normalized transcript.
- With trim enabled, fails on missing adapters/config, bounds generation/validation errors, invalid bounds JSON, invalid range/segment ids, trim adapter failures, or invalid trimmed output.
## Tests to Inspect Before Changing
- `internal/stage/trim_test.go`
- `internal/adapters/scriptorium/subprocess_test.go`
- `internal/adapters/seriatim/subprocess_test.go`
## Architectural Invariants
- Trim never falls back to processed transcript; normalized transcript is required input.
- `session_bounds` output exists only for enabled trim path.
- Render-debug artifacts are diagnostics and not declared stage outputs.

71
docs/internal/storage.md Normal file
View File

@@ -0,0 +1,71 @@
# Internal: Storage
## Purpose
Document Narratio's remote storage backend contracts and implementations under `internal/adapters/storage`.
## Inputs and outputs
Inputs:
- Resolved storage config (`pipeline.storage.*`).
- Bucket-relative object keys and local file paths from stage/app orchestration.
Outputs:
- Listed/downloaded/uploaded object metadata (`ObjectInfo`).
- Existence checks and storage-layer errors.
## Boundaries
Owns:
- Remote object-store interface and implementation details.
- S3 client wiring and API calls.
- Object key normalization and upload/download/list primitives.
Does not own:
- Session/run prefix semantics.
- Archive commit order semantics.
- Manifest updates.
## Config fields used
- `pipeline.storage.backend`
- `pipeline.storage.s3.bucket`
- `pipeline.storage.s3.region`
- `pipeline.storage.s3.endpoint`
- `pipeline.storage.s3.force_path_style`
- `pipeline.storage.s3.access_key_id_env`
- `pipeline.storage.s3.secret_access_key_env`
## External adapters used
Storage package contracts:
- `ObjectStore` (active remote object-store boundary): `List`, `Download`, `Upload`, `Exists`.
- `Backend` (archive request boundary): currently implemented with `NoopBackend` only.
Implementations:
- `S3Backend`: AWS SDK-backed `ObjectStore` implementation.
- `FakeBackend`: deterministic test `ObjectStore` and archive backend.
- `NoopBackend`: deterministic no-op archive backend for compatibility wiring.
## State and manifest behavior
- Storage implementations are stateless with respect to manifest/session lifecycle.
- Caller supplies fully-qualified bucket-relative keys.
- Storage layer does not infer campaign/session/run/root-prefix semantics.
- Caller controls publish ordering; storage layer executes individual operations in the order invoked.
## Skip and resume behavior
- No storage-level skip/resume behavior.
- Skip/resume decisions are made by stage/app logic before storage calls occur.
## Failure behavior
- `NewObjectStoreFromConfig` fails when no remote backend is configured or required S3 config is missing.
- `S3Backend` constructor fails when required bucket is missing or AWS client setup fails.
- CRUD operations return contextual errors (including not-found behavior via `Exists`).
- Key normalization is applied before operations (`\\` to `/`, leading slash trimmed).
## Tests to inspect before changing
- `internal/adapters/storage/factory_test.go`
- `internal/adapters/storage/s3_backend_test.go`
- `internal/adapters/storage/fake_test.go`
- `internal/adapters/storage/keys_test.go`
- `internal/adapters/storage/archive.go` + consumers in stage tests (`prepare`, `archive`)
## Architectural invariants
- Callers pass full bucket-relative keys.
- Storage backends must not prepend or infer narratio prefixes.
- Remote transport details remain isolated to storage adapter implementations.

View File

@@ -0,0 +1,68 @@
# Workspace internals
## Purpose
Define the local durable and run-local workspace model used by stages, manifests, resume, and archive.
## Inputs and Outputs
Inputs:
- `pipeline.workspace.root`
- `session.campaign`
- `session.session_id`
- generated `run_id`
Outputs:
- Session manifest at `{workspace.root}/work/{campaign}/{session_id}/manifest.json`
- Run manifest at `{workspace.root}/work/{campaign}/{session_id}/runs/{run_id}/manifest.json`
- Canonical durable session directories and run-local stage trees
## Boundaries
Owns:
- Session-level path layout (`inputs/`, `audio/`, `transcripts/`, `artifacts/`, `reports/`, `logs/`, `config/`, `current/`, `runs/`)
- Run-local stage sandbox layout under `runs/{run_id}/{stage}/`
- Session lock acquisition/release (`.lock`)
Does not own:
- Stage business logic
- Remote archive semantics (documented in `stage-archive.md`)
- CLI argument parsing
## Config Fields Used
- `pipeline.workspace.root`
- `pipeline.workspace.cleanup_after_archive`
- `pipeline.spool.root`
- `pipeline.spool.delete_audio_after_archive`
- `session.campaign`
- `session.session_id`
## External Adapters Used
None directly in this subsystem. Stages may use object storage adapters and then write local outputs into this layout.
## State and Manifest Behavior
- Session state is persisted in the session manifest (`manifest.Manifest`).
- Invocation history is persisted per run in run manifests under `runs/{run_id}/manifest.json`.
- During each run, stage outputs are often written run-local first (`runs/{run_id}/{stage}/outputs/...`) and promoted to canonical session paths after stage success.
- `manifest.Artifacts` entries record `ProducerRunID` for durable outputs.
- For S3 audio sessions, `prepare` records spool/work paths and S3 provenance in `manifest.Inputs`.
## Skip and Resume Behavior
- Skip/resume decisions are made in `internal/app` (`run_control.go`, `resume.go`) using stage status in the session manifest.
- `--force` reruns selected stages and marks downstream previously-succeeded stages as `stale`.
- Workspace layout is idempotent (`EnsureLayoutFor`) and reused across runs.
## Failure Behavior
- Failures preserve manifests and run-local files for inspection.
- Lock conflicts fail fast via `ErrLockConflict`.
- Cleanup can fail post-archive; failure is recorded in archive stage metadata and returned by the run.
## Tests to Inspect Before Changing
- `internal/artifacts/local_test.go`
- `internal/stage/run_local_test.go`
- `internal/app/run_control_test.go`
- `internal/app/resume_run_stage_test.go`
- `internal/app/post_archive_cleanup_test.go`
## Architectural Invariants
- Session root is campaign-aware: `{workspace.root}/work/{campaign}/{session_id}`.
- Run roots are always nested: `runs/{run_id}` under the session root.
- Run-local output promotion must end in canonical session paths.
- Cleanup only targets run-scoped directories and must never delete configured root directories.

213
docs/operations.md Normal file
View File

@@ -0,0 +1,213 @@
# Operations
This guide describes the implemented operator lifecycle for Narratio.
For field-level configuration, see [docs/config.md](./config.md). For full command/flag reference, see [docs/cli.md](./cli.md).
## Normal workflow (S3-first path)
1. Upload session `.flac` files to object storage under the configured session audio prefix.
2. Run Narratio:
```bash
narratio run --session-id 2026-04-04
```
3. Read success output:
- `narratio run: session <session_id>; executed=<n> skipped=<n>; manifest=<path>`
- use `manifest=<path>` with `status` for inspection.
Notes:
- default config/session discovery applies unless `--config` and `--session` are passed.
- S3 audio mode requires `session.inputs.audio_s3.prefix` and valid object-store access.
## Restore workflow
Use restore when local durable session state is missing or stale and archive current state is authoritative.
Dry-run (no local writes):
```bash
narratio restore --session-id 2026-04-04 --dry-run
```
Execution:
```bash
narratio restore --session-id 2026-04-04
```
Post-restore analyze rerun pattern:
```bash
narratio run-stage --session-id 2026-04-04 --force analyze
```
Restore source-of-truth:
- remote commit marker: `current/run_id.txt`
- remote current manifest: `current/manifest.json`
Restore default scope:
- includes `manifest.json`, `transcripts/**`, `artifacts/**`
- includes `audio/**` only with `--include-audio`
- excludes `runs/**`, `logs/**`, `reports/**`, `config/**`, `inputs/**`, and `current/**` (except remote `current/manifest.json` as source)
## Local filesystem layout and state artifacts
Session root:
- `{workspace.root}/work/{campaign}/{session_id}/`
Primary state:
- `manifest.json`: session-level stage state.
- `runs/{run_id}/manifest.json`: invocation-level state.
- `.lock`: session lock while a modifying command is active.
Canonical session directories:
- `inputs/`
- `audio/`
- `transcripts/`
- `artifacts/`
- `reports/`
- `logs/`
- `config/`
- `current/`
- `runs/`
Run-local stage directories:
- `runs/{run_id}/{stage}/` with stage-local `outputs/`, `logs/`, `reports/`, `config/`, `scratch/`.
Behavior:
- directory creation is idempotent.
- stage outputs are generally generated run-local first, then promoted to canonical paths on success.
- restore installs downloaded files to canonical session paths and does not recreate historical run sandboxes.
## Analyze artifact execution lifecycle
Analyze executes configured artifacts from `pipeline.scriptorium.artifacts`.
Execution model:
- executable set = enabled artifacts, filtered by `--artifacts` when provided.
- artifact-to-artifact dependencies are declared via `depends_on`.
- selected artifacts run in deterministic dependency order.
- after each successful artifact run, output is promoted to configured canonical `output_path`.
Configured artifact source reuse:
- a non-executable configured artifact can satisfy inputs if its configured output file already exists and is valid.
- reused configured artifact provenance is `filesystem.disabled_artifact_output`.
`--artifacts` behavior:
- accepted on `run`, `resume`, and `run-stage analyze`.
- filters analyze execution only; does not force stage rerun.
## Remote archive layout and publish contract
When archive is enabled and run upload is enabled, archive publishes under:
- session prefix: `{root_prefix}/campaigns/{campaign}/sessions/{session_id}/`
- run prefix: `{session_prefix}/runs/{run_id}/`
Archive uploads:
- run record files from run root (excluding `audio/`).
- promoted files from explicit `archive.promote_artifacts` rules.
Publish order:
1. upload `current/manifest.json`
2. upload `current/run_id.txt` last
`current/run_id.txt` is the remote commit marker.
Archive promotion is explicit and source-based:
- Narratio does not auto-promote all generated analyze artifacts.
- each rule resolves `source` through the artifact resolver/catalog model, then uploads to `dest`.
- missing required promotion sources fail archive stage.
- missing optional promotion sources are skipped.
- invalid resolved artifacts fail archive stage.
## Resume, retry, restore, and safe rerun behavior
Default skip:
- `run` and `run-stage` skip already-succeeded stages unless `--force` is set.
Resume:
- `resume` starts at first non-succeeded stage.
- `resume --force` runs full stage order.
Restore conflict policy:
- restore classifies local differences as conflicts.
- without `--force`, restore fails when conflicts exist.
- with `--force`, conflicting local files are overwritten by remote archive files.
Forced reruns:
- force-rerunning an upstream succeeded stage marks downstream succeeded stages as `stale`.
Safe rerun pattern:
1. rerun the changed stage with `--force`.
2. run `resume` to rebuild downstream stages.
## Cleanup behavior
Cleanup is considered only when archive stage executed and succeeded.
Cleanup toggles:
- `pipeline.spool.delete_audio_after_archive=true` deletes run-scoped spool audio.
- `pipeline.workspace.cleanup_after_archive=true` deletes run-scoped local run directory.
Cleanup eligibility gates:
- archive enabled
- archive run upload enabled
- run record upload completed
- current pointer write completed (`current/run_id.txt` written)
No cleanup for failed/incomplete/unarchived/archive-skipped runs.
## Failure and recovery playbooks
After run failure, Narratio keeps:
- session manifest
- run manifest
- run-local artifacts/logs/config/reports
Failed or incomplete runs remain local-only.
After restore failure:
- already-installed restore files remain in place.
- restore does not roll back prior successful installs.
- existing local manifest is preserved if restored manifest validation/install fails.
Recommended recovery:
1. inspect state:
```bash
narratio status --manifest <manifest-path>
```
2. for restore-specific checks, run:
```bash
narratio restore --session-id 2026-04-04 --dry-run
```
3. fix root cause (config/input/credentials/storage/service availability).
4. continue with `resume`, or targeted `run-stage --force` followed by `resume`.
## Restore report
Non-dry-run restore writes a durable report at:
- `reports/restore-latest.json`
Report content includes:
- identity (`campaign`, `session_id`, `run_id`)
- mode flags (`dry_run`, `force`, `include_audio`)
- plan counts and execution counts
- per-action status
Dry-run does not write restore report files.
## Operational caveats
- `status` requires explicit `--manifest`; there is no session-id lookup command.
- local and S3 audio input modes are mutually exclusive.
- archive publish requires upstream stages through `analyze` to be `succeeded`.
- required promotion rules can fail when selected analyze artifacts did not generate a required file path.
- restore requires configured remote object storage and committed remote current state.

715
docs/roadmap/restore.md Normal file
View File

@@ -0,0 +1,715 @@
# Roadmap: `narratio restore` Subcommand
## Status
Implemented through Step 8. This document remains as roadmap and design history for the restore feature, and as the home for future restore-related ideas (for example `run --restore`).
## Summary
Add a new `narratio restore` subcommand that hydrates a local session workspace from the current committed remote archive state.
The primary operator workflow is:
```bash
narratio restore --session-id 2026-04-04
narratio run-stage --force analyze
```
This should allow a new machine with no local workspace state to restore the durable session manifest, transcripts, and generated artifacts from S3, then generate new Scriptorium artifacts without re-running transcription, merge, normalize, polish, or trim.
This is intentionally a separate command. Do not fold this behavior into the `prepare` stage. The existing `prepare` stage should remain focused on materializing configured local/S3 inputs for a pipeline run.
## Goals
- Add a first-class `narratio restore` command.
- Restore the current committed remote session state into the canonical local session workspace.
- Use the existing object storage adapter boundary.
- Preserve archive commit semantics: only restore from a remote state that has a valid current commit marker.
- Restore durable session-level outputs needed for downstream stages, especially `analyze`.
- Provide safe conflict behavior by default.
- Support `--dry-run`, `--force`, and `--include-audio`.
- Keep the implementation explicit, testable, and narrow.
## Non-goals
- Do not make `restore` a pipeline stage.
- Do not change the `prepare` stage behavior as part of this work.
- Do not add implicit restore behavior to `narratio run` in this implementation.
- Do not restore historical run-local sandboxes by default.
- Do not implement a generic remote synchronization engine.
- Do not implement bidirectional sync.
- Do not delete local files merely because they are absent remotely.
- Do not merge remote and local manifests in the first implementation.
- Do not require live S3 for the ordinary unit test suite.
## Future work explicitly out of scope
A future change may add:
```bash
narratio run --restore
```
That future flag should run `narratio restore` before starting the normal pipeline. Mention this as future work in roadmap/docs if useful, but do not implement it now.
## Existing architecture to preserve
### `prepare` remains input materialization
The `prepare` stage currently materializes required session inputs into canonical local workspace paths and records input provenance. It owns local copying/materialization of config and audio inputs, including S3 audio download when `session.inputs.audio_s3.prefix` is configured. It does not own transcript generation/processing or archive publish behavior.
`restore` should not be implemented by expanding `prepare`. It should be an app-level command that reuses shared helpers where appropriate.
### Workspace model
The local durable session workspace is campaign-aware:
```text
{workspace.root}/work/{campaign}/{session_id}/
```
It contains durable session paths such as:
```text
manifest.json
inputs/
audio/
transcripts/
artifacts/
reports/
logs/
config/
current/
runs/
```
Run-local sandboxes live below:
```text
runs/{run_id}/
```
Restore should target durable session-level paths, not old run-local stage sandboxes.
### Storage boundary
The storage adapter owns object-store primitives only: `List`, `Download`, `Upload`, and `Exists`.
The storage adapter must not infer root prefixes, campaign names, session IDs, run IDs, or archive layout. Restore code must construct full bucket-relative keys before calling storage.
### Archive commit boundary
A remote run is current only after the archive stage has uploaded the run record, promoted outputs, `current/manifest.json`, and finally `current/run_id.txt`.
`current/run_id.txt` is the final remote commit marker and must be written last.
Restore must not treat incomplete, skipped, failed, or uncommitted archive attempts as current remote state.
## User-facing command
Add:
```bash
narratio restore [flags]
```
The command should use the same configuration/session discovery conventions as `run`, `plan`, `resume`, and `run-stage` where practical:
```bash
narratio restore --config /path/to/pipeline.yml --session ./session.yml --session-id 2026-04-04
```
Required effective inputs:
- resolved pipeline config;
- resolved session config;
- `session.campaign`;
- `session.session_id`;
- configured remote storage backend.
Supported flags:
```text
--config <path> Existing pipeline config path behavior.
--session <path> Existing session config path behavior.
--session-id <value> Existing session template behavior.
--dry-run Plan restore actions without writing local files.
--force Overwrite conflicting local files with remote files.
--include-audio Include archived session-level audio files.
```
Do not add `--restore` to `run` in this implementation.
## Default restore scope
By default, restore:
1. Validates and reads the current remote commit marker.
2. Downloads the current remote manifest into the local session manifest path.
3. Downloads durable transcript files.
4. Downloads durable generated artifact files.
Default included remote/local durable paths:
```text
manifest.json from remote current manifest
transcripts/**
artifacts/**
```
Default excluded paths:
```text
audio/** unless --include-audio is passed
runs/** always excluded for this implementation
logs/** excluded for this implementation
reports/** excluded for this implementation unless needed for current manifest validation
config/** excluded for this implementation
inputs/** excluded for this implementation
current/** remote control metadata only; do not mirror blindly
```
If the existing archive implementation stores promoted files in a different remote layout, use the existing archive/path helpers and current archive semantics rather than inventing a parallel layout.
## Remote state discovery
Implement restore around the current committed archive state.
Expected algorithm:
1. Resolve pipeline/session config.
2. Ensure storage is configured.
3. Ensure local workspace layout exists.
4. Acquire the session lock.
5. Build the remote session archive prefix using the same helpers/policy used by archive code.
6. Check for the remote `current/run_id.txt` commit marker.
7. Read the committed run ID.
8. Download `current/manifest.json` to a temporary file.
9. Validate that the manifest is parseable and belongs to the requested campaign/session.
10. Build a restore plan from the committed remote state.
11. Execute the restore plan unless `--dry-run` is set.
12. Emit a concise summary.
Important: `current/run_id.txt` is the commit marker. Do not restore from a remote session prefix merely because files exist under `transcripts/` or `artifacts/`.
## Restore planning
Create a planning layer before writing files.
A restore plan entry should include at least:
```go
type RestoreAction struct {
Kind RestoreActionKind
RemoteKey string
LocalPath string
Size int64
ETag string
ExistsLocal bool
SameLocal bool
Conflict bool
Reason string
}
```
Suggested action kinds:
```text
download
skip_same
skip_missing_optional
conflict
```
The restore planner should be deterministic:
- sort remote objects by key;
- sort planned actions by local path or stable restore priority;
- write/report stable output for tests.
## Conflict and overwrite policy
Default behavior should be safe.
For each planned file:
```text
local absent:
download
local present and same as remote:
skip
local present and different:
conflict; fail restore unless --force is set
--force:
overwrite local conflicting files with remote versions
--dry-run:
do not write any files; report what would happen
```
The first implementation may use size and checksum/hash comparison where available. If remote ETag cannot be treated as a content hash, compare by downloading to a temporary file and hashing locally before deciding whether a local file is the same. Prefer correctness over assuming provider-specific ETag semantics.
Do not delete local files that are not present remotely.
## File writing and transactionality
Restore should avoid partial writes.
Implementation requirements:
- download each remote object to a temporary file under the session workspace or OS temp dir;
- validate downloaded content where possible before replacing local files;
- create parent directories as needed;
- atomically rename/copy into place only after successful download;
- do not overwrite local files unless `--force` is set;
- if a later file fails, preserve already-restored files but return a failure summary;
- never corrupt an existing local manifest on failed manifest download/parse.
Manifest restore is especially sensitive:
- download remote `current/manifest.json` to a temporary file;
- parse and validate it;
- if no local manifest exists, install it;
- if a local manifest exists and is equivalent, skip;
- if a local manifest exists and differs, fail unless `--force` is set;
- with `--force`, replace the local manifest with the remote manifest after validation;
- do not attempt a manifest merge in the initial implementation.
## Manifest semantics
`restore` is not a pipeline run and should not mark stages as running/succeeded/failed.
The restored remote manifest becomes the local session manifest. That is what allows a subsequent command such as:
```bash
narratio run-stage --force analyze
```
to see existing upstream stage state and canonical durable outputs.
Do not create a new run manifest for `restore`.
It is acceptable to write a restore diagnostic report outside the manifest, for example:
```text
reports/restore-latest.json
```
or a timestamped report, if that pattern fits the existing codebase. The report must not contain secrets.
## Local workspace locking
`restore` should acquire the same session lock used by ordinary pipeline operations before modifying session workspace state.
If the lock is held, fail fast with the same lock-conflict behavior used elsewhere.
`--dry-run` may still acquire the lock for consistency, but it is acceptable to avoid the lock if the codebase already has a clear read-only command pattern. Prefer safety and simplicity.
## Audio behavior
By default, do not restore audio.
If `--include-audio` is passed:
- restore archived durable session-level audio files only;
- do not use run-scoped spool paths;
- do not mutate or delete spool state;
- do not infer original `session.inputs.audio_s3.prefix` behavior;
- respect the same conflict/force/dry-run behavior used for transcripts/artifacts.
If the archive does not contain durable audio files, `--include-audio` should report that no archived audio was found rather than failing, unless the final implementation chooses to treat explicit audio restore as required. Prefer non-failure for absent archived audio unless tests or existing archive semantics suggest otherwise.
## Remote object selection
Prefer using manifest/artifact metadata when it reliably identifies durable outputs.
Also support listing committed durable archive prefixes so restore can retrieve all top-level session artifacts that may not yet be fully represented in manifest metadata.
The implementation should inspect existing archive code before choosing the final object-selection method. Do not duplicate archive path construction.
Recommended selection priority:
1. Remote current manifest path.
2. Durable promoted transcript/artifact outputs recorded in the manifest or archive metadata, if available.
3. Objects under committed durable `transcripts/` and `artifacts/` archive prefixes.
4. Objects under durable `audio/` only when `--include-audio` is passed.
Always exclude:
```text
runs/**
```
for the first implementation.
## Package and file organization
Expected areas to inspect and update:
```text
cmd/narratio/
internal/app/
internal/adapters/storage/
internal/artifacts/
internal/manifest/
docs/
examples/
```
Suggested implementation shape:
```text
internal/app/restore.go
internal/app/restore_test.go
internal/archive/restore/
planner.go
executor.go
report.go
keys.go
*_test.go
```
The exact package name may vary. Use whatever best fits the existing repository, but keep these boundaries clear:
- `internal/app` owns CLI command handling, config/session loading, lock acquisition, and wiring.
- Restore planning/execution owns remote key discovery, conflict detection, downloads, and reporting.
- `internal/adapters/storage` remains a transport boundary only.
- Workspace/path helpers remain centralized; do not scatter string concatenation.
If the repository already has an `internal/archive` or archive-stage helper package, prefer extending that rather than creating a conflicting package layout.
## CLI output
`narratio restore` should print a concise operator summary.
Example successful output:
```text
Restored session archive for sample-campaign/2026-04-04
Remote run: 20260504T031500Z-a1b2c3
Downloaded: 4
Skipped unchanged: 2
Conflicts: 0
```
Example dry run:
```text
Restore plan for sample-campaign/2026-04-04
Remote run: 20260504T031500Z-a1b2c3
Would download: transcripts/processed.json
Would download: transcripts/trimmed.json
Would skip unchanged: artifacts/session_recap.md
```
Example conflict:
```text
restore conflict: local artifacts/session_recap.md differs from remote archive; rerun with --force to overwrite
```
Do not print transcript or artifact content.
## Error behavior
Fail clearly when:
- storage backend is not configured;
- S3 bucket/config is missing or invalid;
- remote current commit marker is missing;
- remote current manifest is missing;
- remote manifest is invalid;
- remote manifest does not match requested campaign/session;
- local file differs from remote and `--force` is not set;
- a required remote object download fails;
- a local path would escape the session workspace;
- a remote key maps to an unsafe local path.
Skip or report non-fatal conditions when:
- optional audio restore finds no archived audio;
- an included prefix has no objects;
- a local file already matches the remote file.
## Path safety
Every restored file must map to a safe path under the session root.
Validation rules:
- local restore paths must be relative to the session root;
- reject absolute paths;
- reject `..` traversal;
- reject paths that escape through symlinks if the codebase has symlink-safe path checks;
- do not restore remote keys directly without mapping/classification;
- do not mirror arbitrary remote keys.
## Testing plan
Add focused unit tests. Do not require live S3.
### CLI tests
Add or update `internal/app` command tests for:
- `narratio restore --help`;
- restore accepts `--config`, `--session`, and `--session-id`;
- restore accepts `--dry-run`;
- restore accepts `--force`;
- restore accepts `--include-audio`;
- restore fails when storage is not configured;
- restore does not run pipeline stages.
### Restore planner tests
Test:
- missing `current/run_id.txt` fails;
- missing `current/manifest.json` fails;
- invalid manifest fails;
- wrong campaign/session manifest fails;
- default scope includes manifest/transcripts/artifacts;
- default scope excludes audio/logs/reports/config/runs;
- `--include-audio` includes durable audio;
- run-local keys are excluded;
- keys are sorted deterministically;
- unsafe remote-to-local paths are rejected.
### Conflict policy tests
Test:
- absent local file downloads;
- matching local file skips;
- differing local file conflicts by default;
- `--force` overwrites conflicts;
- `--dry-run` writes nothing;
- partial failure does not corrupt an existing local manifest.
### Storage/fake tests
Use fake storage to simulate:
- object listing;
- object download;
- missing objects;
- download failures;
- metadata/ETag behavior.
### Workspace/lock tests
Test:
- session layout is created before restore;
- session lock conflict fails;
- restored files land under the expected campaign/session workspace;
- no files are written outside the session root.
### Follow-up command workflow test
Add at least one test that simulates:
```bash
narratio restore --session-id 2026-04-04
narratio run-stage --force analyze
```
The test does not need to run real Scriptorium. Use existing fake/stub behavior to verify that restored transcripts and manifest state are sufficient for analyze-stage input resolution.
## Documentation updates when implemented
When the feature is implemented, update current-behavior docs:
```text
docs/cli.md
docs/operations.md
docs/internal/storage.md or docs/internal/archive/restore.md
```
If the documentation set does not yet have an internal restore document, add one consistent with the existing internal-doc style:
```text
docs/internal/command-restore.md
```
or:
```text
docs/internal/archive-restore.md
```
Do not document future `narratio run --restore` behavior outside `docs/roadmap/` until implemented.
## Implementation phases
### Phase 1: Audit existing archive and path helpers (completed)
Before coding behavior, inspect:
```text
internal/app/
internal/stage/archive*
internal/adapters/storage/
internal/artifacts/
internal/manifest/
docs/internal/stage-archive.md, if present
```
Determine:
- exact remote archive key layout;
- how root prefix/campaign/session are modeled;
- how current commit marker keys are built;
- how current manifest is uploaded;
- where promoted outputs are uploaded;
- whether helper functions already exist for remote archive keys;
- whether local workspace path helpers can safely map restore destinations.
Deliverable:
- small code comments or internal helper selection;
- no large behavior change yet unless required by tests.
### Phase 2: Add CLI surface and command wiring (completed)
Add `narratio restore` command parsing.
Wire flags:
```text
--config
--session
--session-id
--dry-run
--force
--include-audio
```
Use the existing config/session load path where practical.
Deliverable:
- command exists;
- help output is sensible;
- command validates basic inputs;
- command returns a clear “not yet implemented” or calls an empty planner if phased commits are desired;
- CLI tests pass.
### Phase 3: Implement remote current-state discovery (completed)
Add restore code that:
- creates an object store from resolved config;
- builds remote current marker key;
- reads `current/run_id.txt`;
- reads/downloads `current/manifest.json`;
- validates manifest identity;
- returns remote current-state metadata.
Deliverable:
- fake-storage tests for current-state discovery;
- no local file writes beyond temporary files.
### Phase 4: Implement restore planning (completed)
Build deterministic restore plans for default scope and `--include-audio`.
Deliverable:
- plan lists manifest, transcript, artifact files;
- plan excludes run-local data;
- plan detects local same/conflict/missing states;
- dry-run output works;
- no real file overwrite yet except temp comparisons as needed.
### Phase 5: Implement restore execution (completed)
Execute the plan safely:
- create directories;
- download to temporary files;
- validate content where practical;
- atomically install files;
- enforce default conflict failure;
- support `--force`;
- preserve existing manifest unless safe to replace.
Deliverable:
- restore works end-to-end against fake storage;
- failures are clear and do not corrupt existing local manifest.
### Phase 6: Add restore report and operator summary (completed)
Add concise stdout summary and optional JSON restore report if consistent with project diagnostics.
Deliverable:
- user-friendly output;
- durable diagnostic report if implemented;
- no content leakage.
### Phase 7: Workflow integration test (completed)
Add a test for restoring a previous session and then forcing `analyze`.
Deliverable:
- restored manifest/transcripts/artifacts are sufficient for analyze input resolution;
- no upstream stages rerun;
- no reliance on live subprocesses or S3.
### Phase 8: Documentation update (completed)
Once implemented, update current-behavior docs and internal command docs.
Also leave future `narratio run --restore` in roadmap only.
## Definition of done
The feature is complete when:
- `narratio restore` exists and is documented.
- It uses the same config/session discovery semantics as other commands where practical.
- It requires configured remote storage.
- It restores only from a committed current archive state.
- It restores the current manifest, transcripts, and artifacts by default.
- It restores audio only with `--include-audio`.
- It excludes run-local sandboxes.
- It fails on local/remote conflicts by default.
- `--force` overwrites conflicts.
- `--dry-run` writes nothing.
- It uses fake storage in tests.
- It does not change `prepare` behavior.
- It does not implement `narratio run --restore`.
- It avoids AWS SDK leakage outside the storage adapter.
- It uses centralized path/key helpers rather than scattered string concatenation.
- `go test ./...` passes.
## Suggested test commands
Run focused tests first:
```bash
go test ./internal/app -run TestExecute -v
go test ./internal/adapters/storage -v
go test ./internal/artifacts -v
go test ./internal/manifest -v
```
Then run the full suite:
```bash
go test ./...
```
## Suggested commit message
```text
Add restore subcommand roadmap
```

View File

@@ -1,119 +0,0 @@
# Archive Storage
This document describes implemented archive-stage publish behavior.
## S3 Paths
Session root:
`{root_prefix}/campaigns/{campaign}/sessions/{session_id}/`
Run prefix:
`{root_prefix}/campaigns/{campaign}/sessions/{session_id}/runs/{run_id}/`
## Scope
Implemented:
- archive uploads successful run records to remote object storage through the storage backend abstraction.
- archive uploads configured promoted outputs to session-level keys.
- archive uploads `current/manifest.json`.
- archive uploads `current/run_id.txt` last as the effective commit marker.
- optional post-archive local cleanup:
- `pipeline.spool.delete_audio_after_archive: true` removes only the run-scoped spool audio directory
- `pipeline.workspace.cleanup_after_archive: true` removes only the run-scoped local workdir
- tests use fake storage and do not require live S3.
Future work:
- `notify` stage behavior
- stale detection
- optional future source-audio upload mode
- additional artifact generation beyond current implemented set
## Prerequisites
Archive verifies these stages succeeded before upload:
- `prepare`
- `transcribe`
- `merge`
- `polish`
- `normalize`
- `trim`
- `analyze`
If any prerequisite is missing or not succeeded, archive fails and does not upload.
Failed or incomplete runs remain local only.
## Run Upload
Archive uploads existing files from the run workdir when present:
- `inputs/`
- `transcripts/`
- `artifacts/`
- `reports/` (optional)
- `config/`
- `logs/`
- `manifest.json`
Relative paths are preserved under `runs/{run_id}/`.
## Promotion Rules
Archive applies `archive.promote_artifacts` in config order.
Rule behavior:
- `from`: local workdir-relative source path
- `to`: session-root-relative destination key
- `required: true`: missing source fails archive
- `required: false`: missing source is skipped and recorded
Default promoted outputs:
- `transcripts/trimmed.json`
- `artifacts/session_recap.md`
## Current Pointers
Archive writes:
1. `current/manifest.json` (after run upload + promotions)
2. `current/run_id.txt` last
`current/run_id.txt` contains exactly:
- `{run_id}` plus trailing newline
Writing `current/run_id.txt` last makes it the effective commit marker for published session state.
If any required run upload, promotion upload, or current-manifest upload fails, archive returns failure and does not write `current/run_id.txt`.
Cleanup runs only after this commit-marker write has succeeded.
## Audio Upload Policy
Archive does not upload local `audio/` by default.
Original audio is expected at the session-level audio prefix and is not duplicated under `runs/{run_id}/`.
## Config Controls
- `archive.enabled: false` skips archive cleanly.
- `archive.upload_run: false` skips run upload cleanly.
- both skip cases also skip post-archive local cleanup.
## Metadata
Archive stage metadata includes non-secret upload context (for example):
- `s3_bucket`
- `s3_run_prefix`
- run upload counts/paths
- promoted upload counts/paths
- skipped optional promotions
- `current_manifest_key`
- `current_run_id_key`
- `current_pointer_written`
- `audio_upload_skipped`

View File

@@ -1,84 +0,0 @@
# S3 Audio Input
This document describes implemented S3 audio input behavior in `prepare`.
## Scope
Implemented:
- `prepare` can acquire source audio from S3 when `session.inputs.audio_s3.prefix` is configured.
- object listing and download go through the storage backend abstraction.
- tests use fake storage; no live S3 service is required for test runs.
Not implemented:
- uploads of failed runs
## Required Configuration
`pipeline.yml`:
- `storage.s3.bucket` must be set when S3 audio input is used.
- `storage.s3.root_prefix` defaults to `dnd`.
- `storage.s3.access_key_id_env` defaults to `OBJECT_STORAGE_KEY_ID`.
- `storage.s3.secret_access_key_env` defaults to `OBJECT_STORAGE_KEY`.
- `spool.root` defaults to `/var/spool/narratio`.
`session.yml`:
- configure `session.campaign` and `session.session_id`.
- configure `session.inputs.audio_s3.prefix` for S3 audio input.
- do not configure `inputs.audio_dir` or `inputs.audio_files` at the same time as `inputs.audio_s3`.
## Prefix Shape
Session S3 root:
`{root_prefix}/campaigns/{campaign}/sessions/{session_id}/`
Audio prefix:
`{session_root}/{audio_s3.prefix}`
Example:
`dnd/campaigns/forsaken/sessions/2026-04-19/audio/`
Audio files must already exist in S3 before running Narratio.
## Prepare Behavior
When `inputs.audio_s3.prefix` is configured, `prepare`:
1. lists objects under the computed S3 audio prefix
2. filters to `.flac` objects
3. fails when no `.flac` objects are found
4. downloads selected objects to spool audio:
- `{spool.root}/{campaign}/{session_id}/{run_id}/audio/`
5. materializes audio into workdir audio:
- `{workspace.root}/work/{campaign}/{session_id}/{run_id}/audio/`
6. records input provenance in the manifest (bucket, key, metadata, local paths, checksum)
Notes:
- `.flac` filtering is case-insensitive.
- ETag is recorded as provider metadata only and is not treated as a checksum.
## Local Audio Development
Local audio workflows remain supported:
- `inputs.audio_dir`
- `inputs.audio_files`
These options are mutually exclusive with `inputs.audio_s3`.
## Archive Boundary
Current archive behavior relevant to S3 audio input:
- successful runs are uploaded by archive under `runs/{run_id}/`
- configured promotions are uploaded to session-level destinations
- `current/manifest.json` and `current/run_id.txt` are published
- local source audio is not re-uploaded by default
- failed or incomplete runs are not uploaded

380
docs/troubleshooting.md Normal file
View File

@@ -0,0 +1,380 @@
# Troubleshooting
## Purpose
Canonical operator troubleshooting guide for recurring implemented Narratio failures.
## Config file discovery failure
Symptom:
- `run`, `plan`, `resume`, `run-stage`, or `restore` fails with config/session not found.
Likely Cause:
- `pipeline.yml` or `session.yml` is missing from discovery paths.
- wrong working directory when relying on `./session.yml`.
Diagnostics:
```bash
pwd
ls -l ./session.yml
ls -l /usr/local/etc/narratio/pipeline.yml /etc/narratio/pipeline.yml
```
Safe Fix:
- pass explicit `--config` and `--session`.
- or place files in documented discovery paths.
Links:
- [docs/config.md](./config.md)
- [docs/cli.md](./cli.md)
## Session template rendering failure
Symptom:
- load fails with unresolved placeholder or `session_id` mismatch.
Likely Cause:
- templated `session.yml` used without `--session-id`.
- rendered `session_id` differs from passed `--session-id`.
Diagnostics:
```bash
narratio plan --session ./session.yml --session-id 2026-04-04
```
Safe Fix:
- pass `--session-id` when template placeholders are present.
- ensure rendered `session_id` matches intended run session id.
Links:
- [docs/config.md](./config.md)
## Strict YAML decode or validation failure
Symptom:
- config load fails with unknown field or validation error.
Likely Cause:
- typo/stale field name.
- missing required fields or invalid constraints.
Diagnostics:
```bash
narratio plan --config /path/to/pipeline.yml --session /path/to/session.yml --session-id 2026-04-04
```
Safe Fix:
- align fields/values to canonical config reference and examples.
Links:
- [docs/config.md](./config.md)
- [examples/](../examples/)
## `--artifacts` selection failure
Symptom:
- `run`/`resume`/`run-stage` fails with invalid or unknown artifact selection.
Likely Cause:
- `--artifacts` contains blank names or unknown artifact keys.
- `pipeline.scriptorium.artifacts` missing while using `--artifacts`.
Diagnostics:
```bash
narratio run --config /path/to/pipeline.yml --session /path/to/session.yml --session-id 2026-04-04 --artifacts player_handout
```
Safe Fix:
- use configured artifact keys only.
- ensure `pipeline.scriptorium.artifacts` is defined.
Links:
- [docs/cli.md](./cli.md)
- [docs/config.md](./config.md)
## `run-stage --artifacts` on non-analyze stage
Symptom:
- `run-stage` fails with `--artifacts is only supported for stage "analyze"`.
Likely Cause:
- `--artifacts` was used with a non-`analyze` stage.
Diagnostics:
```bash
narratio run-stage --config /path/to/pipeline.yml --session /path/to/session.yml --session-id 2026-04-04 --artifacts session_recap polish
```
Safe Fix:
- use `--artifacts` only with `run-stage ... analyze`.
Links:
- [docs/cli.md](./cli.md)
## Configured artifact dependency/input validation failure
Symptom:
- config validation fails for `depends_on`, `narratio.artifact.<name>` source, or artifact output path.
Likely Cause:
- `narratio.artifact.<name>` source missing matching `depends_on` key.
- dependency references unknown artifact key.
- dependency self-reference or enabled dependency cycle.
- artifact output path missing/invalid/outside `artifacts/` root.
Diagnostics:
```bash
narratio plan --config /path/to/pipeline.yml --session /path/to/session.yml --session-id 2026-04-04
```
Safe Fix:
- ensure artifact-to-artifact inputs have explicit `depends_on` entries using artifact keys.
- ensure referenced artifacts exist and define valid `output_path` values.
- keep output paths relative and under `artifacts/`.
Links:
- [docs/config.md](./config.md)
- [docs/internal/stage-analyze.md](./internal/stage-analyze.md)
## Required configured artifact input unavailable at analyze time
Symptom:
- analyze fails because configured input source is unavailable.
Likely Cause:
- required upstream configured artifact was not selected/executed this run.
- non-executable dependency output file is missing or invalid on disk.
Diagnostics:
```bash
narratio status --manifest /path/to/manifest.json
narratio run-stage --config /path/to/pipeline.yml --session /path/to/session.yml --session-id 2026-04-04 --artifacts player_handout analyze
```
Safe Fix:
- run analyze with needed artifacts selected.
- or ensure dependency output file exists at configured path and is valid.
Links:
- [docs/operations.md](./operations.md)
- [docs/config.md](./config.md)
## Manifest/status path failure
Symptom:
- `status` fails because manifest path is missing, unreadable, or invalid.
Likely Cause:
- wrong manifest path.
- manifest removed after cleanup.
- `--manifest` omitted.
Diagnostics:
```bash
narratio status --manifest /path/to/manifest.json
ls -l /path/to/manifest.json
```
Safe Fix:
- use manifest path printed by `run`, `resume`, or `run-stage`.
Links:
- [docs/cli.md](./cli.md)
- [docs/operations.md](./operations.md)
## Session lock conflict (`.lock`)
Symptom:
- `run`, `resume`, `run-stage`, or `restore` fails with lock conflict for session workdir.
Likely Cause:
- another Narratio process is running same session.
- stale lock from interrupted prior run.
Diagnostics:
```bash
ls -l {workspace.root}/work/{campaign}/{session_id}/.lock
cat {workspace.root}/work/{campaign}/{session_id}/.lock
ps aux | grep narratio
```
Safe Fix:
- wait for active process to finish.
- if no process is active, remove only stale session `.lock` file.
Links:
- [docs/operations.md](./operations.md)
- [docs/internal/workspace.md](./internal/workspace.md)
## Restore remote current pointer or manifest missing
Symptom:
- `restore` fails with remote current pointer or current manifest errors.
Likely Cause:
- `current/run_id.txt` was never published.
- `current/manifest.json` is missing for the session prefix.
- archive commit did not complete.
Diagnostics:
```bash
narratio restore --config /path/to/pipeline.yml --session /path/to/session.yml --session-id 2026-04-04 --dry-run
```
Safe Fix:
- verify archive stage succeeded for the target session.
- rerun/archive from a healthy source workspace so current pointers are published.
Links:
- [docs/operations.md](./operations.md)
- [docs/internal/stage-archive.md](./internal/stage-archive.md)
## Restore manifest identity mismatch
Symptom:
- `restore` fails because remote manifest session or campaign does not match requested values.
Likely Cause:
- wrong `--session-id` or wrong session config selected.
- archive prefix points to a different campaign/session.
Diagnostics:
```bash
narratio restore --config /path/to/pipeline.yml --session /path/to/session.yml --session-id 2026-04-04 --dry-run
```
Safe Fix:
- use the correct session config and `--session-id`.
- verify campaign/session identity in local config before restore.
Links:
- [docs/config.md](./config.md)
- [docs/operations.md](./operations.md)
## Restore conflict without `--force`
Symptom:
- `restore` fails with `restore conflict` and conflict counts.
Likely Cause:
- local durable file differs from remote file for one or more planned restore paths.
Diagnostics:
```bash
narratio restore --config /path/to/pipeline.yml --session /path/to/session.yml --session-id 2026-04-04 --dry-run
```
Safe Fix:
- review planned conflicts.
- rerun with `--force` only when remote state should overwrite local state.
Links:
- [docs/cli.md](./cli.md)
- [docs/operations.md](./operations.md)
## Restore report expectations
Symptom:
- operator expects restore report file but does not find one.
Likely Cause:
- restore was executed in `--dry-run` mode.
- restore failed before report persistence path (for example lock acquisition failure).
Diagnostics:
```bash
ls -l {workspace.root}/work/{campaign}/{session_id}/reports/restore-latest.json
```
Safe Fix:
- run non-dry-run restore for durable report output.
- resolve lock or early preflight failures and retry.
Links:
- [docs/operations.md](./operations.md)
## Secrets env-dir or credential-env failure
Symptom:
- startup fails loading secrets directory, or stage fails due to missing credential env vars.
Likely Cause:
- invalid `pipeline.secrets.env_dir` path/permissions.
- required credential env var unset/empty.
Diagnostics:
```bash
ls -la /path/to/secrets_dir
env | grep -E 'AUDITA|OBJECT_STORAGE|AWS|SCRIPTORIUM'
```
Safe Fix:
- fix secrets directory and credential env vars.
- keep secret values out of YAML.
Links:
- [docs/config.md](./config.md)
## S3-audio prepare failure
Symptom:
- `prepare` fails in S3 mode (listing/downloading/no audio/backend error).
Likely Cause:
- wrong `session.inputs.audio_s3.prefix`.
- no `.flac` files at resolved prefix.
- invalid/missing object-store credentials or backend config.
- mixed local+S3 audio input config.
Diagnostics:
```bash
narratio run-stage --config /path/to/pipeline.yml --session /path/to/session.yml --session-id 2026-04-04 prepare
```
Safe Fix:
- configure exactly one audio source mode.
- verify `.flac` files and storage access.
Links:
- [docs/config.md](./config.md)
- [docs/operations.md](./operations.md)
## Archive promotion/current-pointer failure
Symptom:
- archive fails on required promotion source missing or pointer write failure.
Likely Cause:
- required promoted file absent (including analyze outputs not generated for this run).
- storage upload failed before `current/run_id.txt` commit marker write.
Diagnostics:
```bash
narratio status --manifest /path/to/manifest.json
narratio run-stage --config /path/to/pipeline.yml --session /path/to/session.yml --session-id 2026-04-04 archive
```
Safe Fix:
- rerun or resume upstream stages to generate required files.
- adjust promotion `source`/`dest` rules to match artifacts that must exist.
- retry after storage issue is resolved.
Links:
- [docs/operations.md](./operations.md)
- [docs/config.md](./config.md)
- [docs/internal/stage-archive.md](./internal/stage-archive.md)

View File

@@ -1,26 +0,0 @@
workspace:
root: ./tmp/narratio-workspace
whisperx:
transcribe_url: "https://transcription.example.com/transcribe"
seriatim:
binary: "seriatim"
timeout: "10m"
output_schema: "seriatim-intermediate"
coalesce_gap: 3.0
audita:
binary: "audita"
timeout: "3h"
base_url: "https://openrouter.ai/api/v1"
model: "openrouter/google/gemma-4-31b-it"
llm_api_key_env: "AUDITA_LLM_API_KEY"
modules: ["glossary", "homophones", "spoken_word", "grammar"]
output_schema: "audita-v1"
work_dir_retention: "auto"
total_llm_concurrency: 2
proposal_llm_concurrency: 1
validation_model: "openrouter/google/gemma-4-31b-it"
validation_llm_concurrency: 1
report: true

View File

@@ -0,0 +1,182 @@
# Full annotated pipeline example for implemented Narratio config fields.
# Values are safe placeholders and must be adapted per environment.
workspace:
# Optional: defaults to /var/lib/narratio.
root: /var/lib/narratio/workspace
# Optional: remove run-scoped workdir after successful archive commit.
cleanup_after_archive: false
# Optional: local secret file loader (directory of ENV_VAR_NAME files).
# secrets:
# env_dir: ./secrets
storage:
# Optional storage backend selector; use "s3" for archive + S3 audio workflows.
backend: s3
# Compatibility fields retained in schema.
bucket: ""
prefix: ""
s3:
# Required when using S3 audio or S3 archive uploads.
bucket: my-dnd-archive
# Optional; defaults to "dnd".
root_prefix: dnd
# Optional region/endpoint settings.
region: us-east-1
endpoint: ""
force_path_style: false
# Optional; defaults shown explicitly.
access_key_id_env: OBJECT_STORAGE_KEY_ID
secret_access_key_env: OBJECT_STORAGE_KEY
spool:
# Optional; defaults to /var/spool/narratio.
root: /var/spool/narratio
# Optional cleanup of run-scoped spool audio after successful archive commit.
delete_audio_after_archive: false
archive:
# Optional booleans; defaults are true.
enabled: true
upload_run: true
# Optional promotion rules; sources use Narratio artifact source IDs.
promote_artifacts:
- source: narratio.transcript.trimmed
dest: transcripts/trimmed.json
required: true
- source: narratio.artifact.session_recap
dest: artifacts/session_recap.md
required: true
- source: narratio.artifact.player_handout
dest: artifacts/player_handout.md
required: false
whisperx:
# Required.
transcribe_url: "https://transcription.example.com/transcribe"
# Optional overrides; defaults shown explicitly.
language: en
timeout: 30m
retries: 3
retry_delay: 2s
concurrency: 2
seriatim:
# Optional overrides; defaults shown explicitly.
binary: seriatim
timeout: 10m
output_schema: seriatim-intermediate
coalesce_gap: 3.0
report: true
env:
# Optional advanced tuning; set only when needed.
overlap_word_run_gap: 1.0
overlap_word_run_reorder_window: 1.0
backchannel_max_duration: 2.0
filler_max_duration: 1.25
audita:
# Optional overrides; defaults shown explicitly where applicable.
binary: audita
timeout: 3h
llm_api_key_env: AUDITA_LLM_API_KEY
modules: [glossary, homophones, spoken_word, grammar]
base_url: ""
model: ""
total_llm_concurrency: 2
proposal_llm_concurrency: 1
validation_model: ""
validation_llm_concurrency: 1
transcript_description: ""
config_path: /usr/local/etc/audita/config.yml
output_schema: audita-v1
work_dir_retention: auto
report: true
normalize:
# Optional; defaults shown explicitly.
output_path: transcripts/normalized.json
output_schema: seriatim-intermediate
report: true
trim:
# Keep disabled unless bounds prompt integration is configured.
enabled: false
output_path: transcripts/trimmed.json
bounds:
prompt_id: dnd.session_bounds
profile_id: local-fast
transcript_input_name: transcript
output_path: reports/session_bounds.json
timeout: 10m
render_debug: false
render_output_path: reports/session_bounds.render.json
seriatim:
report: false
scriptorium:
binary: scriptorium
config_path: /usr/local/etc/scriptorium/config.yml
timeout: 10m
render_debug: false
artifacts:
# Configured artifact keys map to source IDs narratio.artifact.<key>.
session_recap:
enabled: true
prompt_id: dnd.session_recap
profile_id: local-fast
output_path: artifacts/session_recap.md
timeout: 10m
inputs:
transcript:
source: narratio.transcript.trimmed
required: true
previous_recap:
source: previous_session_artifact
artifact: session_recap
path: ""
required: false
vars:
session_id: true
session_date: true
campaign_name: true
previous_session_id: true
output_kind: session_recap
# Example dependent artifact:
# - depends_on entries use artifact keys.
# - narratio.artifact.<key> sources require matching depends_on membership.
player_handout:
enabled: true
depends_on:
- session_recap
prompt_id: dnd.player_handout
profile_id: local-fast
output_path: artifacts/player_handout.md
timeout: 10m
inputs:
recap:
source: narratio.artifact.session_recap
required: true
transcript:
source: narratio.transcript.trimmed
required: true
vars:
session_id: true
campaign_name: true
output_kind: player_handout
analyzer:
# Optional adapter settings.
binary_path: ""
timeout: 2m
artifacts:
output_dir: ""
types: []
notification:
# Optional notification settings.
backend: ""
recipient: ""
timeout: 30s

View File

@@ -1,55 +1,2 @@
workspace:
root: ./tmp/narratio-workspace
cleanup_after_archive: false
storage:
backend: s3
s3:
bucket: "my-dnd-archive"
root_prefix: "dnd"
region: "us-east-1"
# Optional credential env-var names (defaulted when omitted):
# access_key_id_env: "OBJECT_STORAGE_KEY_ID"
# secret_access_key_env: "OBJECT_STORAGE_KEY"
spool:
root: "/var/spool/narratio"
delete_audio_after_archive: false
archive:
enabled: true
upload_run: true
whisperx:
transcribe_url: "https://transcription.example.com/transcribe"
# Optional. When omitted entirely, Narratio defaults to seriatim binary + runtime defaults.
seriatim: {}
# Optional runtime overrides. Model/provider can be owned by Audita runtime config.
audita:
config_path: "/usr/local/etc/audita/config.yml"
llm_api_key_env: "AUDITA_LLM_API_KEY"
# Optional Scriptorium integration for analyze artifacts.
scriptorium:
config_path: "/usr/local/etc/scriptorium/config.yml"
artifacts:
session_recap:
enabled: true
prompt_id: "dnd.session_recap"
output_path: "artifacts/session_recap.md"
inputs:
transcript:
source: "trimmed_transcript"
required: true
previous_recap:
source: "previous_session_artifact"
artifact: "session_recap"
required: false
vars:
session_id: true
session_date: true
campaign_name: true
previous_session_id: true
output_kind: "session_recap"

View File

@@ -0,0 +1,116 @@
workspace:
root: /var/lib/narratio/workspace
cleanup_after_archive: true
storage:
backend: s3
s3:
bucket: my-dnd-archive
root_prefix: dnd
region: us-east-1
access_key_id_env: OBJECT_STORAGE_KEY_ID
secret_access_key_env: OBJECT_STORAGE_KEY
spool:
root: /var/spool/narratio
delete_audio_after_archive: true
archive:
enabled: true
upload_run: true
promote_artifacts:
- source: narratio.transcript.trimmed
dest: transcripts/trimmed.json
required: true
- source: narratio.artifact.session_recap
dest: artifacts/session_recap.md
required: true
- source: narratio.artifact.player_handout
dest: artifacts/player_handout.md
required: false
whisperx:
transcribe_url: "https://transcription.example.com/transcribe"
language: en
timeout: 45m
retries: 3
retry_delay: 3s
concurrency: 2
seriatim:
binary: seriatim
timeout: 10m
output_schema: seriatim-intermediate
coalesce_gap: 3.0
report: true
audita:
binary: audita
timeout: 3h
llm_api_key_env: AUDITA_LLM_API_KEY
modules: [glossary, homophones, spoken_word, grammar]
output_schema: audita-v1
work_dir_retention: auto
total_llm_concurrency: 2
proposal_llm_concurrency: 1
validation_llm_concurrency: 1
report: true
normalize:
output_path: transcripts/normalized.json
output_schema: seriatim-intermediate
report: true
trim:
enabled: false
scriptorium:
binary: scriptorium
config_path: /usr/local/etc/scriptorium/config.yml
timeout: 10m
render_debug: false
artifacts:
session_recap:
enabled: true
prompt_id: dnd.session_recap
profile_id: local-fast
output_path: artifacts/session_recap.md
timeout: 10m
inputs:
transcript:
source: narratio.transcript.trimmed
required: true
previous_recap:
source: previous_session_artifact
artifact: session_recap
required: false
vars:
session_id: true
session_date: true
campaign_name: true
previous_session_id: true
output_kind: session_recap
player_handout:
enabled: true
depends_on:
- session_recap
prompt_id: dnd.player_handout
profile_id: local-fast
output_path: artifacts/player_handout.md
timeout: 10m
inputs:
recap:
source: narratio.artifact.session_recap
required: true
transcript:
source: narratio.transcript.trimmed
required: true
vars:
session_id: true
output_kind: player_handout
analyzer:
timeout: 2m
notification:
timeout: 30s

View File

@@ -0,0 +1,9 @@
session_id: 2026-05-03
campaign: sample-campaign
date: 2026-05-03
title: Sample Session
inputs:
audio_dir: ./audio
speakers_file: ./examples/speakers.yml
autocorrect_file: ./examples/autocorrect.yml
glossary_file: ./examples/glossary.yml

View File

@@ -1,13 +0,0 @@
session_id: 2026-05-03
campaign: sample-campaign
date: 2026-05-03
title: Sample Session
inputs:
audio_dir: ./audio
# Optional S3 input alternative. Do not configure with audio_dir/audio_files.
# Narratio prepare lists this prefix and downloads .flac files.
# audio_s3:
# prefix: "audio/"
speakers_file: ./speakers.yml
autocorrect_file: ./autocorrect.yml
glossary_file: ./glossary.yml

View File

@@ -0,0 +1,10 @@
session_id: 2026-05-03
campaign: sample-campaign
date: 2026-05-03
title: Sample Session
inputs:
audio_s3:
prefix: audio/
speakers_file: ./examples/speakers.yml
autocorrect_file: ./examples/autocorrect.yml
glossary_file: ./examples/glossary.yml

View File

@@ -1,12 +1,7 @@
session_id: "{{ session_id }}"
campaign: sample-campaign
date: ""
title: ""
inputs:
audio_dir: ./audio
# Optional S3 input alternative. Do not configure with audio_dir/audio_files.
# audio_s3:
# prefix: "audio/{{ session_id }}/"
speakers_file: ./speakers.yml
autocorrect_file: ./autocorrect.yml
glossary_file: ./glossary.yml
speakers_file: ./examples/speakers.yml
autocorrect_file: ./examples/autocorrect.yml
glossary_file: ./examples/glossary.yml

2
go.mod
View File

@@ -4,6 +4,7 @@ go 1.25.0
require (
github.com/aws/aws-sdk-go-v2/config v1.32.17
github.com/aws/aws-sdk-go-v2/credentials v1.19.16
github.com/aws/aws-sdk-go-v2/service/s3 v1.101.0
github.com/aws/smithy-go v1.25.1
gopkg.in/yaml.v3 v3.0.1
@@ -12,7 +13,6 @@ require (
require (
github.com/aws/aws-sdk-go-v2 v1.41.7 // indirect
github.com/aws/aws-sdk-go-v2/aws/protocol/eventstream v1.7.10 // indirect
github.com/aws/aws-sdk-go-v2/credentials v1.19.16 // indirect
github.com/aws/aws-sdk-go-v2/feature/ec2/imds v1.18.23 // indirect
github.com/aws/aws-sdk-go-v2/internal/configsources v1.4.23 // indirect
github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.7.23 // indirect

View File

@@ -0,0 +1,66 @@
package app
import (
"fmt"
"sort"
"strings"
"gitea.maximumdirect.net/eric/narratio/internal/config"
)
type artifactSelectionFlag struct {
values []string
}
func (f *artifactSelectionFlag) String() string {
return strings.Join(f.values, ",")
}
func (f *artifactSelectionFlag) Set(value string) error {
f.values = append(f.values, value)
return nil
}
func (f *artifactSelectionFlag) Normalize() ([]string, error) {
if len(f.values) == 0 {
return nil, nil
}
seen := map[string]struct{}{}
out := make([]string, 0, len(f.values))
for _, raw := range f.values {
for _, part := range strings.Split(raw, ",") {
name := strings.TrimSpace(part)
if name == "" {
return nil, fmt.Errorf("artifact names must be non-empty")
}
if _, ok := seen[name]; ok {
continue
}
seen[name] = struct{}{}
out = append(out, name)
}
}
sort.Strings(out)
return out, nil
}
func validateSelectedAnalyzeArtifacts(cfg *config.Config, selected []string) error {
if len(selected) == 0 {
return nil
}
if cfg == nil || cfg.Pipeline == nil || cfg.Pipeline.Scriptorium == nil {
return fmt.Errorf("--artifacts requires pipeline.scriptorium.artifacts to be configured")
}
configured := cfg.Pipeline.Scriptorium.Artifacts
if len(configured) == 0 {
return fmt.Errorf("--artifacts requires at least one configured artifact in pipeline.scriptorium.artifacts")
}
for _, name := range selected {
if _, ok := configured[name]; !ok {
return fmt.Errorf("--artifacts includes unknown artifact %q", name)
}
}
return nil
}

View File

@@ -0,0 +1,140 @@
package app
import (
"bytes"
"context"
"os"
"path/filepath"
"strings"
"testing"
"time"
"gitea.maximumdirect.net/eric/narratio/internal/manifest"
)
func TestExecuteRunStageArtifactsNonAnalyzeFails(t *testing.T) {
workspaceRoot := t.TempDir()
pipelinePath, sessionPath := writeValidConfigFilesWithScriptoriumArtifacts(t, workspaceRoot)
var stdout bytes.Buffer
var stderr bytes.Buffer
code := Execute(
[]string{"run-stage", "--config", pipelinePath, "--session", sessionPath, "--artifacts", "session_recap", "polish"},
&stdout,
&stderr,
)
if code == 0 {
t.Fatal("exit code = 0, want non-zero")
}
if !strings.Contains(stderr.String(), `run-stage: --artifacts is only supported for stage "analyze"`) {
t.Fatalf("stderr = %q, want stage-gating error", stderr.String())
}
}
func TestExecuteUnknownArtifactsFailValidation(t *testing.T) {
workspaceRoot := t.TempDir()
pipelinePath, sessionPath := writeValidConfigFilesWithScriptoriumArtifacts(t, workspaceRoot)
var stdout bytes.Buffer
var stderr bytes.Buffer
code := Execute(
[]string{"run", "--config", pipelinePath, "--session", sessionPath, "--artifacts", "unknown_artifact"},
&stdout,
&stderr,
)
if code == 0 {
t.Fatal("exit code = 0, want non-zero")
}
if !strings.Contains(stderr.String(), `run: --artifacts includes unknown artifact "unknown_artifact"`) {
t.Fatalf("stderr = %q, want unknown-artifact validation error", stderr.String())
}
}
func TestRunStageArtifactsDoesNotImplyForce(t *testing.T) {
workspaceRoot := t.TempDir()
pipelinePath, sessionPath := writeValidConfigFilesWithScriptoriumArtifacts(t, workspaceRoot)
manifestPath := filepath.Join(workspaceRoot, "work", "sample-campaign", "2026-05-03", "manifest.json")
store := &manifest.LocalStore{}
seed := manifest.New("2026-05-03", time.Date(2026, 5, 3, 10, 0, 0, 0, time.UTC))
seed.MarkStageSucceeded("analyze", time.Date(2026, 5, 3, 10, 1, 0, 0, time.UTC), nil)
if err := store.Save(context.Background(), manifestPath, seed); err != nil {
t.Fatalf("save manifest: %v", err)
}
var out bytes.Buffer
err := RunStage(
context.Background(),
[]string{"--config", pipelinePath, "--session", sessionPath, "--artifacts", "session_recap,session_recap", "analyze"},
&out,
)
if err != nil {
t.Fatalf("RunStage() error = %v", err)
}
if !strings.Contains(out.String(), "stage=analyze executed=0 skipped=1 force=false") {
t.Fatalf("output = %q, want analyze skip without force", out.String())
}
}
func TestResumeArtifactsWithSucceededAnalyzeSkipsUnlessForced(t *testing.T) {
workspaceRoot := t.TempDir()
pipelinePath, sessionPath := writeValidConfigFilesWithScriptoriumArtifacts(t, workspaceRoot)
manifestPath := filepath.Join(workspaceRoot, "work", "sample-campaign", "2026-05-03", "manifest.json")
store := &manifest.LocalStore{}
seed := manifest.New("2026-05-03", time.Date(2026, 5, 3, 10, 0, 0, 0, time.UTC))
for _, stageName := range []string{"prepare", "transcribe", "merge", "polish", "normalize", "trim", "analyze", "archive", "notify"} {
seed.MarkStageSucceeded(stageName, time.Date(2026, 5, 3, 10, 1, 0, 0, time.UTC), nil)
}
if err := store.Save(context.Background(), manifestPath, seed); err != nil {
t.Fatalf("save manifest: %v", err)
}
var out bytes.Buffer
err := Resume(
context.Background(),
[]string{"--config", pipelinePath, "--session", sessionPath, "--artifacts", "session_recap"},
&out,
)
if err != nil {
t.Fatalf("Resume() error = %v", err)
}
if !strings.Contains(out.String(), "has no remaining stages") {
t.Fatalf("output = %q, want no remaining stages", out.String())
}
}
func writeValidConfigFilesWithScriptoriumArtifacts(t *testing.T, workspaceRoot string) (string, string) {
t.Helper()
pipelinePath, sessionPath := writeValidConfigFiles(t, workspaceRoot)
f, err := os.OpenFile(pipelinePath, os.O_APPEND|os.O_WRONLY, 0)
if err != nil {
t.Fatalf("open pipeline config for append: %v", err)
}
defer f.Close()
extra := `
scriptorium:
binary: scriptorium
artifacts:
session_recap:
enabled: true
prompt_id: dnd.session_recap
output_path: artifacts/session_recap.md
player_handout:
enabled: true
prompt_id: dnd.player_handout
output_path: artifacts/player_handout.md
depends_on:
- session_recap
inputs:
recap:
source: narratio.artifact.session_recap
required: true
`
if _, err := f.WriteString(extra); err != nil {
t.Fatalf("append scriptorium config: %v", err)
}
return pipelinePath, sessionPath
}

View File

@@ -0,0 +1,132 @@
package app
import (
"testing"
"gitea.maximumdirect.net/eric/narratio/internal/config"
)
func TestArtifactSelectionFlagNormalize(t *testing.T) {
tests := []struct {
name string
inputs []string
want []string
wantErr string
}{
{
name: "single value",
inputs: []string{"session_recap"},
want: []string{"session_recap"},
},
{
name: "repeatable and comma separated values are deduped and sorted",
inputs: []string{"session_recap,player_handout", "session_recap"},
want: []string{"player_handout", "session_recap"},
},
{
name: "empty token fails",
inputs: []string{"session_recap,"},
wantErr: "artifact names must be non-empty",
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
var flag artifactSelectionFlag
for _, in := range tt.inputs {
if err := flag.Set(in); err != nil {
t.Fatalf("Set(%q) error = %v", in, err)
}
}
got, err := flag.Normalize()
if tt.wantErr != "" {
if err == nil {
t.Fatalf("Normalize() error = nil, want %q", tt.wantErr)
}
if err.Error() != tt.wantErr {
t.Fatalf("Normalize() error = %q, want %q", err.Error(), tt.wantErr)
}
return
}
if err != nil {
t.Fatalf("Normalize() error = %v", err)
}
if len(got) != len(tt.want) {
t.Fatalf("Normalize() len = %d, want %d; got=%v", len(got), len(tt.want), got)
}
for i := range got {
if got[i] != tt.want[i] {
t.Fatalf("Normalize()[%d] = %q, want %q", i, got[i], tt.want[i])
}
}
})
}
}
func TestValidateSelectedAnalyzeArtifacts(t *testing.T) {
tests := []struct {
name string
cfg *config.Config
selected []string
wantErr string
}{
{
name: "empty selection is accepted",
cfg: &config.Config{},
selected: nil,
},
{
name: "scriptorium required when selected artifacts present",
cfg: &config.Config{Pipeline: &config.PipelineConfig{}},
selected: []string{"session_recap"},
wantErr: "--artifacts requires pipeline.scriptorium.artifacts to be configured",
},
{
name: "unknown selected artifact fails",
cfg: &config.Config{
Pipeline: &config.PipelineConfig{
Scriptorium: &config.ScriptoriumConfig{
Artifacts: map[string]config.ScriptoriumArtifactConfig{
"session_recap": {Enabled: true, PromptID: "dnd.session_recap", OutputPath: "artifacts/session_recap.md"},
},
},
},
},
selected: []string{"player_handout"},
wantErr: `--artifacts includes unknown artifact "player_handout"`,
},
{
name: "known selected artifacts are accepted",
cfg: &config.Config{
Pipeline: &config.PipelineConfig{
Scriptorium: &config.ScriptoriumConfig{
Artifacts: map[string]config.ScriptoriumArtifactConfig{
"session_recap": {Enabled: true, PromptID: "dnd.session_recap", OutputPath: "artifacts/session_recap.md"},
"player_handout": {Enabled: true, PromptID: "dnd.player_handout", OutputPath: "artifacts/player_handout.md"},
},
},
},
},
selected: []string{"player_handout", "session_recap"},
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
err := validateSelectedAnalyzeArtifacts(tt.cfg, tt.selected)
if tt.wantErr != "" {
if err == nil {
t.Fatalf("error = nil, want %q", tt.wantErr)
}
if err.Error() != tt.wantErr {
t.Fatalf("error = %q, want %q", err.Error(), tt.wantErr)
}
return
}
if err != nil {
t.Fatalf("error = %v, want nil", err)
}
})
}
}

View File

@@ -7,7 +7,7 @@ import (
"strings"
)
var supportedCommands = []string{"run", "plan", "status", "resume", "run-stage"}
var supportedCommands = []string{"run", "plan", "status", "resume", "run-stage", "restore"}
// Execute dispatches CLI commands and returns a process exit code.
func Execute(args []string, stdout, stderr io.Writer) int {
@@ -32,6 +32,8 @@ func Execute(args []string, stdout, stderr io.Writer) int {
err = Resume(ctx, cmdArgs, stdout)
case "run-stage":
err = RunStage(ctx, cmdArgs, stdout)
case "restore":
err = Restore(ctx, cmdArgs, stdout)
default:
fmt.Fprintf(stderr, "unknown command: %q\n\n", cmd)
printUsage(stderr)

View File

@@ -111,7 +111,7 @@ func TestExecuteRunStageUnknownFails(t *testing.T) {
func TestExecuteRunStageNormalizeIsAccepted(t *testing.T) {
workspaceRoot := t.TempDir()
pipelinePath, sessionPath := writeValidConfigFiles(t, workspaceRoot, "https://example.com/transcribe")
workRoot := filepath.Join(workspaceRoot, "work", "2026-05-03")
workRoot := filepath.Join(workspaceRoot, "work", "sample-campaign", "2026-05-03")
mustWriteTestFile(t, filepath.Join(workRoot, "transcripts", "processed.json"), `{"segments":[{"id":1}]}`)
var stdout bytes.Buffer
@@ -156,7 +156,7 @@ func TestExecuteRunStageTranscribeUsesConfiguredWhisperXServer(t *testing.T) {
t.Fatal("expected whisperx server to be called at least once")
}
outPath := filepath.Join(workspaceRoot, "work", "2026-05-03", "transcripts", "raw", "alice.json")
outPath := filepath.Join(workspaceRoot, "work", "sample-campaign", "2026-05-03", "transcripts", "raw", "alice.json")
data, err := os.ReadFile(outPath)
if err != nil {
t.Fatalf("ReadFile(%q): %v", outPath, err)
@@ -233,7 +233,7 @@ inputs:
_ = os.Chdir(originalWD)
})
workRoot := filepath.Join(workspaceRoot, "work", sessionID)
workRoot := filepath.Join(workspaceRoot, "work", "sample-campaign", sessionID)
mustWriteTestFile(t, filepath.Join(workRoot, "transcripts", "merged.json"), `{"schema":"seriatim-intermediate","segments":[]}`)
mustWriteTestFile(t, filepath.Join(workRoot, "inputs", "glossary.yml"), "[]\n")

View File

@@ -57,7 +57,7 @@ func Plan(ctx context.Context, args []string, out io.Writer) error {
}
store := artifacts.NewLocalStore(cfg.Pipeline.Workspace.Root)
paths, err := store.EnsureLayout(cfg.Session.SessionID)
paths, err := store.EnsureLayoutFor(cfg.Session.Campaign, cfg.Session.SessionID)
if err != nil {
return fmt.Errorf("plan: prepare workdir: %w", err)
}

View File

@@ -36,7 +36,7 @@ func TestPlanCreatesAndReusesWorkdir(t *testing.T) {
t.Fatalf("first output = %q, want totals", got)
}
sessionWorkdir := artifacts.SessionWorkDir(workspaceRoot, "2026-05-03")
sessionWorkdir := artifacts.SessionWorkDirForCampaign(workspaceRoot, "sample-campaign", "2026-05-03")
expectedDirs := []string{
sessionWorkdir,
filepath.Join(sessionWorkdir, "inputs"),
@@ -63,7 +63,7 @@ func TestPlanCreatesAndReusesWorkdir(t *testing.T) {
func TestPlanShowsRunAndSkipFromManifest(t *testing.T) {
workspaceRoot := t.TempDir()
pipelinePath, sessionPath := writeValidConfigFiles(t, workspaceRoot)
manifestPath := filepath.Join(workspaceRoot, "work", "2026-05-03", "manifest.json")
manifestPath := filepath.Join(workspaceRoot, "work", "sample-campaign", "2026-05-03", "manifest.json")
store := &manifest.LocalStore{}
m := manifest.New("2026-05-03", time.Date(2026, 5, 3, 10, 0, 0, 0, time.UTC))

View File

@@ -54,7 +54,7 @@ func runPostArchiveCleanup(ctx context.Context, env *Env, manifestPath string, m
}
workDir := strings.TrimSpace(m.LocalWorkDir)
if workDir == "" {
workDir = artifacts.SessionRunWorkDir(
workDir = artifacts.SessionRunRootForCampaign(
env.Config.Pipeline.Workspace.Root,
strings.TrimSpace(env.Config.Session.Campaign),
strings.TrimSpace(env.Config.Session.SessionID),

View File

@@ -195,7 +195,7 @@ func TestPostArchiveCleanupNotRunWhenPromotionIsMissing(t *testing.T) {
cfg.Pipeline.Spool.DeleteAudioAfterArchive = true
cfg.Pipeline.Workspace.CleanupAfterArchive = true
cfg.Pipeline.Archive.PromoteArtifacts = []config.ArchivePromotionRule{
{From: "artifacts/missing.md", To: "artifacts/missing.md", Required: boolPtr(true)},
{Source: "narratio.transcript.merged", Dest: "transcripts/merged.json", Required: boolPtr(true)},
}
archiveStageImpl, err := stage.Select("archive")
@@ -203,14 +203,14 @@ func TestPostArchiveCleanupNotRunWhenPromotionIsMissing(t *testing.T) {
t.Fatalf("Select(archive) error = %v", err)
}
_, err = executeStages(context.Background(), cfg, []stage.Stage{archiveStageImpl}, RunOptions{Env: &Env{ObjectStore: &storage.FakeBackend{}}})
if err == nil || !strings.Contains(err.Error(), "required promotion source missing") {
if err == nil || !strings.Contains(err.Error(), "required promotion source unavailable") {
t.Fatalf("executeStages() error = %v, want promotion-missing failure", err)
}
assertExists(t, seed.spoolAudioDir)
assertExists(t, seed.runWorkDir)
assertExists(t, filepath.Join(seed.runWorkDir, "manifest.json"))
assertExists(t, artifacts.SessionRunWorkDir(cfg.Pipeline.Workspace.Root, cfg.Session.Campaign, cfg.Session.SessionID, runID))
assertExists(t, artifacts.SessionRunRootForCampaign(cfg.Pipeline.Workspace.Root, cfg.Session.Campaign, cfg.Session.SessionID, runID))
}
func TestPostArchiveCleanupNotRunWhenCurrentManifestUploadFails(t *testing.T) {
@@ -271,8 +271,8 @@ func cleanupFixtureConfig(t *testing.T) (*config.Config, cleanupSeed) {
cfg.Pipeline.Spool.Root = filepath.Join(t.TempDir(), "spool")
runID := "20260516T010203Z-1a2b3c4d"
runWorkDir := artifacts.SessionRunWorkDir(cfg.Pipeline.Workspace.Root, cfg.Session.Campaign, cfg.Session.SessionID, runID)
otherRunDir := artifacts.SessionRunWorkDir(cfg.Pipeline.Workspace.Root, cfg.Session.Campaign, cfg.Session.SessionID, "20260516T010204Z-5e6f7a8b")
runWorkDir := artifacts.SessionRunRootForCampaign(cfg.Pipeline.Workspace.Root, cfg.Session.Campaign, cfg.Session.SessionID, runID)
otherRunDir := artifacts.SessionRunRootForCampaign(cfg.Pipeline.Workspace.Root, cfg.Session.Campaign, cfg.Session.SessionID, "20260516T010204Z-5e6f7a8b")
spoolAudioDir := artifacts.SessionSpoolAudioDir(cfg.Pipeline.Spool.Root, cfg.Session.Campaign, cfg.Session.SessionID, runID)
mustWriteFile(t, filepath.Join(runWorkDir, "manifest.json"), "{}\n")
@@ -322,11 +322,22 @@ func archiveStageCleanupFixture(t *testing.T) (*config.Config, cleanupSeed, stri
Enabled: boolPtr(true),
UploadRun: boolPtr(true),
PromoteArtifacts: []config.ArchivePromotionRule{
{From: "transcripts/trimmed.json", To: "transcripts/trimmed.json", Required: boolPtr(true)},
{From: "artifacts/session_recap.md", To: "artifacts/session_recap.md", Required: boolPtr(true)},
{Source: "narratio.transcript.trimmed", Dest: "transcripts/trimmed.json", Required: boolPtr(true)},
{Source: "narratio.artifact.session_recap", Dest: "artifacts/session_recap.md", Required: boolPtr(true)},
},
}
writeArchiveFixtureRunFiles(t, seed.runWorkDir)
cfg.Pipeline.Scriptorium = &config.ScriptoriumConfig{
Artifacts: map[string]config.ScriptoriumArtifactConfig{
"session_recap": {
OutputPath: "artifacts/session_recap.md",
},
},
}
writeArchiveFixtureRunFiles(
t,
seed.runWorkDir,
artifacts.SessionWorkDirForCampaign(cfg.Pipeline.Workspace.Root, cfg.Session.Campaign, cfg.Session.SessionID),
)
store := &manifest.LocalStore{}
seedManifest, err := store.Load(context.Background(), manifestPathFor(cfg))
@@ -345,16 +356,19 @@ func archiveStageCleanupFixture(t *testing.T) (*config.Config, cleanupSeed, stri
return cfg, seed, runID
}
func writeArchiveFixtureRunFiles(t *testing.T, runWorkDir string) {
func writeArchiveFixtureRunFiles(t *testing.T, runWorkDir, sessionRoot string) {
t.Helper()
mustWriteFile(t, filepath.Join(runWorkDir, "inputs", "session.yml"), "session_id: 2026-05-03\n")
mustWriteFile(t, filepath.Join(runWorkDir, "transcripts", "raw", "speaker.json"), "{}\n")
mustWriteFile(t, filepath.Join(runWorkDir, "transcripts", "trimmed.json"), "{}\n")
mustWriteFile(t, filepath.Join(runWorkDir, "artifacts", "session_recap.md"), "# recap\n")
mustWriteFile(t, filepath.Join(runWorkDir, "reports", "audita.report.json"), "{}\n")
mustWriteFile(t, filepath.Join(runWorkDir, "config", "audita.generated.yml"), "key: value\n")
mustWriteFile(t, filepath.Join(runWorkDir, "prepare", "inputs", "session.yml"), "session_id: 2026-05-03\n")
mustWriteFile(t, filepath.Join(runWorkDir, "transcribe", "outputs", "transcripts", "raw", "speaker.json"), "{}\n")
mustWriteFile(t, filepath.Join(runWorkDir, "trim", "outputs", "transcripts", "trimmed.json"), "{\"segments\":[]}\n")
mustWriteFile(t, filepath.Join(runWorkDir, "analyze", "outputs", "artifacts", "session_recap.md"), "# recap\n")
mustWriteFile(t, filepath.Join(runWorkDir, "polish", "reports", "audita.report.json"), "{}\n")
mustWriteFile(t, filepath.Join(runWorkDir, "merge", "config", "seriatim.generated.yml"), "key: value\n")
mustWriteFile(t, filepath.Join(runWorkDir, "logs", "audita.stderr.log"), "stderr\n")
mustWriteFile(t, filepath.Join(runWorkDir, "manifest.json"), "{}\n")
mustWriteFile(t, filepath.Join(sessionRoot, "transcripts", "trimmed.json"), "{\"segments\":[]}\n")
mustWriteFile(t, filepath.Join(sessionRoot, "artifacts", "session_recap.md"), "# recap\n")
}
type failKeyStore struct {

152
internal/app/restore.go Normal file
View File

@@ -0,0 +1,152 @@
package app
import (
"context"
"errors"
"flag"
"fmt"
"io"
"log/slog"
"os"
"gitea.maximumdirect.net/eric/narratio/internal/adapters/storage"
"gitea.maximumdirect.net/eric/narratio/internal/artifacts"
"gitea.maximumdirect.net/eric/narratio/internal/config"
"gitea.maximumdirect.net/eric/narratio/internal/logging"
)
var newObjectStoreFromConfigFn = storage.NewObjectStoreFromConfig
var discoverRemoteCurrentStateFn = discoverRemoteCurrentState
var buildRestorePlanFn = buildRestorePlan
var executeRestorePlanFn = executeRestorePlan
// Restore validates restore CLI/config inputs and storage preflight for future restore phases.
func Restore(ctx context.Context, args []string, out io.Writer) error {
fs := flag.NewFlagSet("restore", flag.ContinueOnError)
fs.SetOutput(out)
var pipelinePath string
var sessionPath string
var sessionID string
var dryRun bool
var force bool
var includeAudio bool
fs.StringVar(&pipelinePath, "config", "", "path to pipeline.yml (optional; defaults searched)")
fs.StringVar(&sessionPath, "session", "", "path to session.yml")
fs.StringVar(&sessionID, "session-id", "", "session identifier for session.yml templates")
fs.BoolVar(&dryRun, "dry-run", false, "plan restore actions without writing local files")
fs.BoolVar(&force, "force", false, "overwrite local conflicts with remote state")
fs.BoolVar(&includeAudio, "include-audio", false, "include archived session-level audio objects")
fs.Usage = func() {
_, _ = fmt.Fprintln(out, "Usage: narratio restore [--config <path>] [--session <path>] [--session-id <value>] [--dry-run] [--force] [--include-audio]")
_, _ = fmt.Fprintln(out)
_, _ = fmt.Fprintln(out, "Flags:")
fs.PrintDefaults()
}
if err := fs.Parse(args); err != nil {
if errors.Is(err, flag.ErrHelp) {
return nil
}
return fmt.Errorf("restore: invalid flags: %w", err)
}
if fs.NArg() != 0 {
return fmt.Errorf("restore: unexpected positional arguments")
}
resolvedPipelinePath, err := resolvePipelineConfigPath(pipelinePath)
if err != nil {
return fmt.Errorf("restore: %w", err)
}
resolvedSessionPath, err := resolveSessionConfigPath(sessionPath)
if err != nil {
return fmt.Errorf("restore: %w", err)
}
cfg, err := config.LoadWithSessionOptions(resolvedPipelinePath, resolvedSessionPath, config.SessionLoadOptions{
SessionID: sessionID,
})
if err != nil {
return fmt.Errorf("restore: %w", err)
}
if err := config.Validate(cfg); err != nil {
return fmt.Errorf("restore: %w", err)
}
if _, err := loadSecretsFromConfig(cfg, logging.NewLogger(os.Stderr, slog.LevelInfo)); err != nil {
return fmt.Errorf("restore: %w", err)
}
objectStore, err := newObjectStoreFromConfigFn(ctx, cfg)
if err != nil {
return fmt.Errorf("restore: %w", err)
}
current, err := discoverRemoteCurrentStateFn(ctx, cfg, objectStore)
if err != nil {
return fmt.Errorf("restore: %w", err)
}
plan, err := buildRestorePlanFn(ctx, cfg, current, objectStore, RestorePlanOptions{
IncludeAudio: includeAudio,
Force: force,
DryRun: dryRun,
})
if err != nil {
return fmt.Errorf("restore: %w", err)
}
report, err := newRestoreReport(current, plan, RestorePlanOptions{
IncludeAudio: includeAudio,
Force: force,
DryRun: dryRun,
})
if err != nil {
return fmt.Errorf("restore: %w", err)
}
if dryRun {
if err := writeRestoreDryRunSummary(out, report); err != nil {
return fmt.Errorf("restore: write plan output: %w", err)
}
return nil
}
artifactStore := artifacts.NewLocalStore(cfg.Pipeline.Workspace.Root)
if _, err := artifactStore.EnsureLayoutFor(cfg.Session.Campaign, cfg.Session.SessionID); err != nil {
return fmt.Errorf("restore: prepare workdir: %w", err)
}
lock, err := artifactStore.AcquireSessionLockFor(cfg.Session.Campaign, cfg.Session.SessionID)
if err != nil {
return fmt.Errorf("restore: acquire session lock: %w", err)
}
defer func() {
_ = artifactStore.ReleaseSessionLock(lock)
}()
if plan.ConflictCount > 0 && !force {
report.setFailed(fmt.Errorf("conflict: %d conflicting path(s)", plan.ConflictCount))
if _, reportErr := persistRestoreReport(artifactStore, cfg, report); reportErr != nil {
return fmt.Errorf("restore: report failure: %w", reportErr)
}
return fmt.Errorf(
"restore conflict: %d conflicting path(s); rerun with --force to overwrite (download=%d skip_same=%d conflicts=%d)",
plan.ConflictCount,
plan.DownloadCount,
plan.SkipSameCount,
plan.ConflictCount,
)
}
result, err := executeRestorePlanFn(ctx, cfg, current, plan, report, objectStore)
if err != nil {
report.setFailed(err)
if _, reportErr := persistRestoreReport(artifactStore, cfg, report); reportErr != nil {
return fmt.Errorf("restore: execute plan failed (%v) and report write failed (%v)", err, reportErr)
}
return fmt.Errorf("restore: execute plan: %w", err)
}
report.Execution.Downloaded = result.DownloadedCount
report.setSucceeded()
if _, err := persistRestoreReport(artifactStore, cfg, report); err != nil {
return fmt.Errorf("restore: write report: %w", err)
}
if err := writeRestoreSuccessSummary(out, report); err != nil {
return fmt.Errorf("restore: write summary: %w", err)
}
return nil
}

View File

@@ -0,0 +1,139 @@
package app
import (
"context"
"fmt"
"os"
"strings"
"gitea.maximumdirect.net/eric/narratio/internal/adapters/storage"
"gitea.maximumdirect.net/eric/narratio/internal/artifacts"
"gitea.maximumdirect.net/eric/narratio/internal/config"
"gitea.maximumdirect.net/eric/narratio/internal/manifest"
)
// RemoteCurrentState captures discovered committed remote archive state for one session.
type RemoteCurrentState struct {
Bucket string
SessionPrefix string
CurrentRunIDKey string
CurrentManifestKey string
RunID string
SessionID string
Campaign string
Manifest *manifest.Manifest
}
func discoverRemoteCurrentState(ctx context.Context, cfg *config.Config, store storage.ObjectStore) (*RemoteCurrentState, error) {
if cfg == nil || cfg.Pipeline == nil || cfg.Session == nil {
return nil, fmt.Errorf("resolved config with pipeline/session is required")
}
if store == nil {
return nil, fmt.Errorf("remote object store is required")
}
bucket := artifacts.ResolveArchiveBucket(cfg, nil)
if strings.TrimSpace(bucket) == "" {
return nil, fmt.Errorf("archive bucket is required")
}
sessionPrefix, err := artifacts.ResolveArchiveSessionPrefix(cfg, nil)
if err != nil {
return nil, fmt.Errorf("resolve archive session prefix: %w", err)
}
currentManifestKey, currentRunIDKey := artifacts.ResolveArchiveCurrentStateKeys(sessionPrefix)
exists, err := store.Exists(ctx, currentRunIDKey)
if err != nil {
return nil, fmt.Errorf("check remote current run pointer %q: %w", currentRunIDKey, err)
}
if !exists {
return nil, fmt.Errorf("remote current run pointer missing: %q", currentRunIDKey)
}
runIDPath, err := downloadObjectToTemp(ctx, store, currentRunIDKey, "narratio-restore-current-run-id-*.txt")
if err != nil {
return nil, fmt.Errorf("download remote current run pointer %q: %w", currentRunIDKey, err)
}
defer func() { _ = os.Remove(runIDPath) }()
runIDData, err := os.ReadFile(runIDPath)
if err != nil {
return nil, fmt.Errorf("read downloaded run pointer %q: %w", currentRunIDKey, err)
}
runID := strings.TrimSpace(string(runIDData))
if runID == "" {
return nil, fmt.Errorf("remote current run pointer %q is empty", currentRunIDKey)
}
exists, err = store.Exists(ctx, currentManifestKey)
if err != nil {
return nil, fmt.Errorf("check remote current manifest %q: %w", currentManifestKey, err)
}
if !exists {
return nil, fmt.Errorf("remote current manifest missing: %q", currentManifestKey)
}
manifestPath, err := downloadObjectToTemp(ctx, store, currentManifestKey, "narratio-restore-current-manifest-*.json")
if err != nil {
return nil, fmt.Errorf("download remote current manifest %q: %w", currentManifestKey, err)
}
defer func() { _ = os.Remove(manifestPath) }()
manifestStore := &manifest.LocalStore{}
remoteManifest, err := manifestStore.Load(ctx, manifestPath)
if err != nil {
return nil, fmt.Errorf("remote current manifest decode failed: %w", err)
}
requestedSession := strings.TrimSpace(cfg.Session.SessionID)
requestedCampaign := strings.TrimSpace(cfg.Session.Campaign)
manifestSession := strings.TrimSpace(remoteManifest.SessionID)
manifestCampaign := strings.TrimSpace(remoteManifest.Campaign)
if manifestSession != requestedSession {
return nil, fmt.Errorf(
"remote current manifest session_id %q does not match requested session_id %q",
manifestSession,
requestedSession,
)
}
if manifestCampaign == "" {
return nil, fmt.Errorf("remote current manifest campaign is required")
}
if manifestCampaign != requestedCampaign {
return nil, fmt.Errorf(
"remote current manifest campaign %q does not match requested campaign %q",
manifestCampaign,
requestedCampaign,
)
}
return &RemoteCurrentState{
Bucket: bucket,
SessionPrefix: sessionPrefix,
CurrentRunIDKey: currentRunIDKey,
CurrentManifestKey: currentManifestKey,
RunID: runID,
SessionID: manifestSession,
Campaign: manifestCampaign,
Manifest: remoteManifest,
}, nil
}
func downloadObjectToTemp(ctx context.Context, store storage.ObjectStore, key, pattern string) (string, error) {
tmp, err := os.CreateTemp("", pattern)
if err != nil {
return "", fmt.Errorf("create temp file: %w", err)
}
path := tmp.Name()
if err := tmp.Close(); err != nil {
_ = os.Remove(path)
return "", fmt.Errorf("close temp file: %w", err)
}
if err := store.Download(ctx, key, path); err != nil {
_ = os.Remove(path)
return "", err
}
return path, nil
}

View File

@@ -0,0 +1,239 @@
package app
import (
"context"
"encoding/json"
"fmt"
"strings"
"testing"
"time"
"gitea.maximumdirect.net/eric/narratio/internal/adapters/storage"
"gitea.maximumdirect.net/eric/narratio/internal/artifacts"
"gitea.maximumdirect.net/eric/narratio/internal/config"
)
func TestDiscoverRemoteCurrentStateSuccess(t *testing.T) {
cfg := restoreDiscoveryConfig()
store := &storage.FakeBackend{}
sessionPrefix, manifestKey, runIDKey := restoreDiscoveryKeys(cfg)
store.SeedObject(storage.FakeObject{Key: runIDKey, Data: []byte("20260519T010203Z-a1b2c3d4\n")})
store.SeedObject(storage.FakeObject{Key: manifestKey, Data: restoreManifestJSON(t, cfg.Session.SessionID, cfg.Session.Campaign)})
state, err := discoverRemoteCurrentState(context.Background(), cfg, store)
if err != nil {
t.Fatalf("discoverRemoteCurrentState() error = %v", err)
}
if state.RunID != "20260519T010203Z-a1b2c3d4" {
t.Fatalf("run id = %q, want 20260519T010203Z-a1b2c3d4", state.RunID)
}
if state.SessionPrefix != sessionPrefix {
t.Fatalf("session prefix = %q, want %q", state.SessionPrefix, sessionPrefix)
}
if state.CurrentRunIDKey != runIDKey {
t.Fatalf("current run id key = %q, want %q", state.CurrentRunIDKey, runIDKey)
}
if state.CurrentManifestKey != manifestKey {
t.Fatalf("current manifest key = %q, want %q", state.CurrentManifestKey, manifestKey)
}
if state.Manifest == nil {
t.Fatal("manifest is nil")
}
}
func TestDiscoverRemoteCurrentStateMissingRunPointerFails(t *testing.T) {
cfg := restoreDiscoveryConfig()
store := &storage.FakeBackend{}
_, err := discoverRemoteCurrentState(context.Background(), cfg, store)
if err == nil || !strings.Contains(err.Error(), "remote current run pointer missing") {
t.Fatalf("error = %v, want missing run pointer failure", err)
}
}
func TestDiscoverRemoteCurrentStateEmptyRunPointerFails(t *testing.T) {
cfg := restoreDiscoveryConfig()
store := &storage.FakeBackend{}
_, manifestKey, runIDKey := restoreDiscoveryKeys(cfg)
store.SeedObject(storage.FakeObject{Key: runIDKey, Data: []byte(" \n\t")})
store.SeedObject(storage.FakeObject{Key: manifestKey, Data: restoreManifestJSON(t, cfg.Session.SessionID, cfg.Session.Campaign)})
_, err := discoverRemoteCurrentState(context.Background(), cfg, store)
if err == nil || !strings.Contains(err.Error(), "is empty") {
t.Fatalf("error = %v, want empty run pointer failure", err)
}
}
func TestDiscoverRemoteCurrentStateMissingManifestFails(t *testing.T) {
cfg := restoreDiscoveryConfig()
store := &storage.FakeBackend{}
_, _, runIDKey := restoreDiscoveryKeys(cfg)
store.SeedObject(storage.FakeObject{Key: runIDKey, Data: []byte("20260519T010203Z-a1b2c3d4\n")})
_, err := discoverRemoteCurrentState(context.Background(), cfg, store)
if err == nil || !strings.Contains(err.Error(), "remote current manifest missing") {
t.Fatalf("error = %v, want missing manifest failure", err)
}
}
func TestDiscoverRemoteCurrentStateInvalidManifestFails(t *testing.T) {
cfg := restoreDiscoveryConfig()
store := &storage.FakeBackend{}
_, manifestKey, runIDKey := restoreDiscoveryKeys(cfg)
store.SeedObject(storage.FakeObject{Key: runIDKey, Data: []byte("20260519T010203Z-a1b2c3d4\n")})
store.SeedObject(storage.FakeObject{Key: manifestKey, Data: []byte("{invalid json")})
_, err := discoverRemoteCurrentState(context.Background(), cfg, store)
if err == nil || !strings.Contains(err.Error(), "remote current manifest decode failed") {
t.Fatalf("error = %v, want manifest decode failure", err)
}
}
func TestDiscoverRemoteCurrentStateSessionMismatchFails(t *testing.T) {
cfg := restoreDiscoveryConfig()
store := &storage.FakeBackend{}
_, manifestKey, runIDKey := restoreDiscoveryKeys(cfg)
store.SeedObject(storage.FakeObject{Key: runIDKey, Data: []byte("20260519T010203Z-a1b2c3d4\n")})
store.SeedObject(storage.FakeObject{Key: manifestKey, Data: restoreManifestJSON(t, "wrong-session", cfg.Session.Campaign)})
_, err := discoverRemoteCurrentState(context.Background(), cfg, store)
if err == nil || !strings.Contains(err.Error(), "does not match requested session_id") {
t.Fatalf("error = %v, want session mismatch failure", err)
}
}
func TestDiscoverRemoteCurrentStateCampaignMismatchFails(t *testing.T) {
cfg := restoreDiscoveryConfig()
store := &storage.FakeBackend{}
_, manifestKey, runIDKey := restoreDiscoveryKeys(cfg)
store.SeedObject(storage.FakeObject{Key: runIDKey, Data: []byte("20260519T010203Z-a1b2c3d4\n")})
store.SeedObject(storage.FakeObject{Key: manifestKey, Data: restoreManifestJSON(t, cfg.Session.SessionID, "wrong-campaign")})
_, err := discoverRemoteCurrentState(context.Background(), cfg, store)
if err == nil || !strings.Contains(err.Error(), "does not match requested campaign") {
t.Fatalf("error = %v, want campaign mismatch failure", err)
}
}
func TestDiscoverRemoteCurrentStateEmptyCampaignFails(t *testing.T) {
cfg := restoreDiscoveryConfig()
store := &storage.FakeBackend{}
_, manifestKey, runIDKey := restoreDiscoveryKeys(cfg)
store.SeedObject(storage.FakeObject{Key: runIDKey, Data: []byte("20260519T010203Z-a1b2c3d4\n")})
store.SeedObject(storage.FakeObject{Key: manifestKey, Data: restoreManifestJSON(t, cfg.Session.SessionID, "")})
_, err := discoverRemoteCurrentState(context.Background(), cfg, store)
if err == nil || !strings.Contains(err.Error(), "campaign is required") {
t.Fatalf("error = %v, want empty campaign failure", err)
}
}
func TestDiscoverRemoteCurrentStateUsesCurrentKeysUnderSessionPrefix(t *testing.T) {
cfg := restoreDiscoveryConfig()
sessionPrefix, manifestKey, runIDKey := restoreDiscoveryKeys(cfg)
base := &storage.FakeBackend{}
store := &captureObjectStore{delegate: base}
base.SeedObject(storage.FakeObject{Key: runIDKey, Data: []byte("20260519T010203Z-a1b2c3d4\n")})
base.SeedObject(storage.FakeObject{Key: manifestKey, Data: restoreManifestJSON(t, cfg.Session.SessionID, cfg.Session.Campaign)})
_, err := discoverRemoteCurrentState(context.Background(), cfg, store)
if err != nil {
t.Fatalf("discoverRemoteCurrentState() error = %v", err)
}
expectedRunKey := fmt.Sprintf("%scurrent/run_id.txt", sessionPrefix)
expectedManifestKey := fmt.Sprintf("%scurrent/manifest.json", sessionPrefix)
if !containsString(store.existsKeys, expectedRunKey) {
t.Fatalf("exists keys = %#v, want run pointer key %q", store.existsKeys, expectedRunKey)
}
if !containsString(store.existsKeys, expectedManifestKey) {
t.Fatalf("exists keys = %#v, want manifest key %q", store.existsKeys, expectedManifestKey)
}
if !containsString(store.downloadKeys, expectedRunKey) {
t.Fatalf("download keys = %#v, want run pointer key %q", store.downloadKeys, expectedRunKey)
}
if !containsString(store.downloadKeys, expectedManifestKey) {
t.Fatalf("download keys = %#v, want manifest key %q", store.downloadKeys, expectedManifestKey)
}
}
type captureObjectStore struct {
delegate storage.ObjectStore
existsKeys []string
downloadKeys []string
}
func (s *captureObjectStore) List(ctx context.Context, prefix string) ([]storage.ObjectInfo, error) {
return s.delegate.List(ctx, prefix)
}
func (s *captureObjectStore) Download(ctx context.Context, key, localPath string) error {
s.downloadKeys = append(s.downloadKeys, key)
return s.delegate.Download(ctx, key, localPath)
}
func (s *captureObjectStore) Upload(ctx context.Context, localPath, key string, opts storage.UploadOptions) (storage.ObjectInfo, error) {
return s.delegate.Upload(ctx, localPath, key, opts)
}
func (s *captureObjectStore) Exists(ctx context.Context, key string) (bool, error) {
s.existsKeys = append(s.existsKeys, key)
return s.delegate.Exists(ctx, key)
}
func restoreDiscoveryConfig() *config.Config {
return &config.Config{
Pipeline: &config.PipelineConfig{
Storage: config.StorageConfig{
S3: &config.StorageS3Config{
Bucket: "my-dnd-archive",
RootPrefix: "dnd",
},
},
},
Session: &config.SessionConfig{
SessionID: "2026-05-03",
Campaign: "sample-campaign",
},
}
}
func restoreDiscoveryKeys(cfg *config.Config) (sessionPrefix, manifestKey, runIDKey string) {
sessionPrefix = artifacts.S3SessionPrefix(cfg.Pipeline.Storage.S3.RootPrefix, cfg.Session.Campaign, cfg.Session.SessionID)
manifestKey, runIDKey = artifacts.ResolveArchiveCurrentStateKeys(sessionPrefix)
return sessionPrefix, manifestKey, runIDKey
}
func restoreManifestJSON(t *testing.T, sessionID, campaign string) []byte {
t.Helper()
now := time.Date(2026, 5, 19, 23, 0, 0, 0, time.UTC).Format(time.RFC3339Nano)
payload := map[string]any{
"session_id": sessionID,
"campaign": campaign,
"created_at": now,
"updated_at": now,
"stages": map[string]any{},
}
data, err := json.Marshal(payload)
if err != nil {
t.Fatalf("marshal manifest payload: %v", err)
}
return append(data, '\n')
}
func containsString(values []string, target string) bool {
for _, value := range values {
if value == target {
return true
}
}
return false
}

View File

@@ -0,0 +1,180 @@
package app
import (
"context"
"fmt"
"os"
"path/filepath"
"strings"
"gitea.maximumdirect.net/eric/narratio/internal/adapters/storage"
"gitea.maximumdirect.net/eric/narratio/internal/artifacts"
"gitea.maximumdirect.net/eric/narratio/internal/config"
"gitea.maximumdirect.net/eric/narratio/internal/manifest"
)
// RestoreExecutionResult captures concrete file-install results for one restore execution.
type RestoreExecutionResult struct {
DownloadedCount int
}
func executeRestorePlan(
ctx context.Context,
cfg *config.Config,
current *RemoteCurrentState,
plan *RestorePlan,
report *RestoreReport,
store storage.ObjectStore,
) (*RestoreExecutionResult, error) {
if cfg == nil || cfg.Pipeline == nil || cfg.Session == nil {
return nil, fmt.Errorf("resolved config with pipeline/session is required")
}
if current == nil {
return nil, fmt.Errorf("remote current state is required")
}
if plan == nil {
return nil, fmt.Errorf("restore plan is required")
}
if store == nil {
return nil, fmt.Errorf("remote object store is required")
}
sessionRoot := artifacts.SessionWorkDirForCampaign(cfg.Pipeline.Workspace.Root, cfg.Session.Campaign, cfg.Session.SessionID)
manifestActions := make([]RestoreAction, 0, 1)
actions := make([]RestoreAction, 0, len(plan.Actions))
for _, action := range plan.Actions {
if action.Kind != RestoreActionDownload {
continue
}
if action.LocalRelativePath == config.PathManifestFile {
manifestActions = append(manifestActions, action)
continue
}
actions = append(actions, action)
}
if len(manifestActions) > 1 {
return nil, fmt.Errorf("restore plan includes multiple manifest download actions")
}
if len(manifestActions) == 1 {
actions = append(actions, manifestActions[0])
}
result := &RestoreExecutionResult{}
for _, action := range actions {
if err := executeRestoreDownloadAction(ctx, cfg, sessionRoot, current, action, store); err != nil {
if report != nil {
report.markFailed(action, err)
}
return nil, fmt.Errorf("install %q from %q: %w", action.LocalRelativePath, action.RemoteKey, err)
}
if report != nil {
report.markDownloaded(action)
}
result.DownloadedCount++
}
return result, nil
}
func executeRestoreDownloadAction(
ctx context.Context,
cfg *config.Config,
sessionRoot string,
current *RemoteCurrentState,
action RestoreAction,
store storage.ObjectStore,
) error {
safeLocalPath, err := joinWithinSessionRoot(sessionRoot, action.LocalRelativePath)
if err != nil {
return fmt.Errorf("resolve safe local path: %w", err)
}
if strings.TrimSpace(action.LocalPath) != "" && filepath.Clean(action.LocalPath) != safeLocalPath {
return fmt.Errorf("restore plan local path mismatch for %q", action.LocalRelativePath)
}
tmpPath, err := downloadObjectToSiblingTemp(ctx, store, action.RemoteKey, safeLocalPath)
if err != nil {
return fmt.Errorf("download to temp file: %w", err)
}
removeTmp := true
defer func() {
if removeTmp {
_ = os.Remove(tmpPath)
}
}()
if action.LocalRelativePath == config.PathManifestFile {
if err := validateRestoredManifest(ctx, cfg, current, tmpPath); err != nil {
return err
}
}
if err := os.Chmod(tmpPath, 0o644); err != nil {
return fmt.Errorf("set file permissions: %w", err)
}
if err := os.Rename(tmpPath, safeLocalPath); err != nil {
return fmt.Errorf("install file atomically: %w", err)
}
removeTmp = false
return nil
}
func downloadObjectToSiblingTemp(ctx context.Context, store storage.ObjectStore, remoteKey, destPath string) (string, error) {
if strings.TrimSpace(destPath) == "" {
return "", fmt.Errorf("destination path is required")
}
dir := filepath.Dir(destPath)
if err := os.MkdirAll(dir, 0o755); err != nil {
return "", fmt.Errorf("create destination directory: %w", err)
}
base := filepath.Base(destPath)
tmp, err := os.CreateTemp(dir, "."+base+".restore-*.tmp")
if err != nil {
return "", fmt.Errorf("create temp file: %w", err)
}
tmpPath := tmp.Name()
if err := tmp.Close(); err != nil {
_ = os.Remove(tmpPath)
return "", fmt.Errorf("close temp file: %w", err)
}
if err := store.Download(ctx, remoteKey, tmpPath); err != nil {
_ = os.Remove(tmpPath)
return "", err
}
return tmpPath, nil
}
func validateRestoredManifest(ctx context.Context, cfg *config.Config, current *RemoteCurrentState, path string) error {
manifestStore := &manifest.LocalStore{}
m, err := manifestStore.Load(ctx, path)
if err != nil {
return fmt.Errorf("validate manifest decode: %w", err)
}
requestedSession := strings.TrimSpace(cfg.Session.SessionID)
requestedCampaign := strings.TrimSpace(cfg.Session.Campaign)
manifestSession := strings.TrimSpace(m.SessionID)
manifestCampaign := strings.TrimSpace(m.Campaign)
if manifestSession != requestedSession {
return fmt.Errorf("manifest session_id %q does not match requested session_id %q", manifestSession, requestedSession)
}
if manifestCampaign == "" {
return fmt.Errorf("manifest campaign is required")
}
if manifestCampaign != requestedCampaign {
return fmt.Errorf("manifest campaign %q does not match requested campaign %q", manifestCampaign, requestedCampaign)
}
if current != nil {
if expected := strings.TrimSpace(current.SessionID); expected != "" && manifestSession != expected {
return fmt.Errorf("manifest session_id %q does not match discovered session_id %q", manifestSession, expected)
}
if expected := strings.TrimSpace(current.Campaign); expected != "" && manifestCampaign != expected {
return fmt.Errorf("manifest campaign %q does not match discovered campaign %q", manifestCampaign, expected)
}
}
return nil
}

View File

@@ -0,0 +1,364 @@
package app
import (
"bytes"
"context"
"encoding/json"
"fmt"
"os"
"path/filepath"
"strings"
"testing"
"gitea.maximumdirect.net/eric/narratio/internal/adapters/storage"
"gitea.maximumdirect.net/eric/narratio/internal/artifacts"
"gitea.maximumdirect.net/eric/narratio/internal/config"
"gitea.maximumdirect.net/eric/narratio/internal/manifest"
)
func TestExecuteRestoreNonDryRunRestoresDurableFiles(t *testing.T) {
workspaceRoot := t.TempDir()
pipelinePath, sessionPath := writeValidConfigFiles(t, workspaceRoot)
fake := &storage.FakeBackend{}
cfg, sessionPrefix, manifestKey, runIDKey := seedRestoreCommittedState(t, fake, pipelinePath, sessionPath)
seedRestoreObject(fake, sessionPrefix+"transcripts/full.json", []byte(`{"segments":[1,2,3]}`))
seedRestoreObject(fake, sessionPrefix+"artifacts/session_recap.md", []byte("# recap\n"))
seedRestoreObject(fake, sessionPrefix+"audio/alice.flac", []byte("remote-audio"))
seedRestoreObject(fake, runIDKey, []byte("20260519T010203Z-a1b2c3d4\n"))
seedRestoreObject(fake, manifestKey, restoreManifestJSON(t, cfg.Session.SessionID, cfg.Session.Campaign))
restoreWithStoreAndRealPhases(t, fake)
var stdout bytes.Buffer
var stderr bytes.Buffer
code := Execute([]string{"restore", "--config", pipelinePath, "--session", sessionPath}, &stdout, &stderr)
if code != 0 {
t.Fatalf("exit code = %d, want 0; stderr=%q", code, stderr.String())
}
if stderr.Len() != 0 {
t.Fatalf("stderr = %q, want empty", stderr.String())
}
if !strings.Contains(stdout.String(), "Restored session archive for sample-campaign/2026-05-03") {
t.Fatalf("stdout = %q, want completion summary", stdout.String())
}
sessionRoot := artifacts.SessionWorkDirForCampaign(workspaceRoot, cfg.Session.Campaign, cfg.Session.SessionID)
mustReadEquals(t, filepath.Join(sessionRoot, "transcripts", "full.json"), `{"segments":[1,2,3]}`)
mustReadEquals(t, filepath.Join(sessionRoot, "artifacts", "session_recap.md"), "# recap\n")
reportPath := filepath.Join(sessionRoot, "reports", "restore-latest.json")
report := mustReadRestoreReport(t, reportPath)
if report.Status != "succeeded" {
t.Fatalf("report status = %q, want succeeded", report.Status)
}
if report.Execution.Downloaded != 3 {
t.Fatalf("report execution.downloaded = %d, want 3", report.Execution.Downloaded)
}
if len(report.Actions) == 0 {
t.Fatal("report actions is empty")
}
if _, err := os.Stat(filepath.Join(sessionRoot, "audio", "alice.flac")); !os.IsNotExist(err) {
t.Fatalf("audio should not be restored by default; stat err=%v", err)
}
}
func TestExecuteRestoreIncludeAudioRestoresAudio(t *testing.T) {
workspaceRoot := t.TempDir()
pipelinePath, sessionPath := writeValidConfigFiles(t, workspaceRoot)
fake := &storage.FakeBackend{}
cfg, sessionPrefix, _, _ := seedRestoreCommittedState(t, fake, pipelinePath, sessionPath)
seedRestoreObject(fake, sessionPrefix+"audio/alice.flac", []byte("remote-audio"))
restoreWithStoreAndRealPhases(t, fake)
var stdout bytes.Buffer
var stderr bytes.Buffer
code := Execute([]string{"restore", "--config", pipelinePath, "--session", sessionPath, "--include-audio"}, &stdout, &stderr)
if code != 0 {
t.Fatalf("exit code = %d, want 0; stderr=%q", code, stderr.String())
}
sessionRoot := artifacts.SessionWorkDirForCampaign(workspaceRoot, cfg.Session.Campaign, cfg.Session.SessionID)
mustReadEquals(t, filepath.Join(sessionRoot, "audio", "alice.flac"), "remote-audio")
report := mustReadRestoreReport(t, filepath.Join(sessionRoot, "reports", "restore-latest.json"))
if !report.IncludeAudio {
t.Fatalf("report include_audio = %v, want true", report.IncludeAudio)
}
}
func TestExecuteRestoreConflictWithoutForceDoesNotOverwrite(t *testing.T) {
workspaceRoot := t.TempDir()
pipelinePath, sessionPath := writeValidConfigFiles(t, workspaceRoot)
fake := &storage.FakeBackend{}
cfg, sessionPrefix, _, _ := seedRestoreCommittedState(t, fake, pipelinePath, sessionPath)
seedRestoreObject(fake, sessionPrefix+"transcripts/full.json", []byte("remote-transcript"))
sessionRoot := artifacts.SessionWorkDirForCampaign(workspaceRoot, cfg.Session.Campaign, cfg.Session.SessionID)
mustWriteTestFile(t, filepath.Join(sessionRoot, "transcripts", "full.json"), "local-transcript")
restoreWithStoreAndRealPhases(t, fake)
var stdout bytes.Buffer
var stderr bytes.Buffer
code := Execute([]string{"restore", "--config", pipelinePath, "--session", sessionPath}, &stdout, &stderr)
if code == 0 {
t.Fatal("exit code = 0, want non-zero")
}
if !strings.Contains(stderr.String(), "conflicting path") {
t.Fatalf("stderr = %q, want conflict failure", stderr.String())
}
mustReadEquals(t, filepath.Join(sessionRoot, "transcripts", "full.json"), "local-transcript")
report := mustReadRestoreReport(t, filepath.Join(sessionRoot, "reports", "restore-latest.json"))
if report.Status != "failed" {
t.Fatalf("report status = %q, want failed", report.Status)
}
if report.Plan.Conflicts != 1 {
t.Fatalf("report plan.conflicts = %d, want 1", report.Plan.Conflicts)
}
}
func TestExecuteRestoreForceOverwritesDifferingFile(t *testing.T) {
workspaceRoot := t.TempDir()
pipelinePath, sessionPath := writeValidConfigFiles(t, workspaceRoot)
fake := &storage.FakeBackend{}
cfg, sessionPrefix, _, _ := seedRestoreCommittedState(t, fake, pipelinePath, sessionPath)
seedRestoreObject(fake, sessionPrefix+"transcripts/full.json", []byte("remote-transcript"))
sessionRoot := artifacts.SessionWorkDirForCampaign(workspaceRoot, cfg.Session.Campaign, cfg.Session.SessionID)
mustWriteTestFile(t, filepath.Join(sessionRoot, "transcripts", "full.json"), "local-transcript")
restoreWithStoreAndRealPhases(t, fake)
var stdout bytes.Buffer
var stderr bytes.Buffer
code := Execute([]string{"restore", "--config", pipelinePath, "--session", sessionPath, "--force"}, &stdout, &stderr)
if code != 0 {
t.Fatalf("exit code = %d, want 0; stderr=%q", code, stderr.String())
}
mustReadEquals(t, filepath.Join(sessionRoot, "transcripts", "full.json"), "remote-transcript")
report := mustReadRestoreReport(t, filepath.Join(sessionRoot, "reports", "restore-latest.json"))
if !report.Force {
t.Fatalf("report force = %v, want true", report.Force)
}
}
func TestExecuteRestoreLockConflictFailsAndWritesNothing(t *testing.T) {
workspaceRoot := t.TempDir()
pipelinePath, sessionPath := writeValidConfigFiles(t, workspaceRoot)
fake := &storage.FakeBackend{}
cfg, sessionPrefix, _, _ := seedRestoreCommittedState(t, fake, pipelinePath, sessionPath)
seedRestoreObject(fake, sessionPrefix+"transcripts/full.json", []byte("remote-transcript"))
store := artifacts.NewLocalStore(workspaceRoot)
lock, err := store.AcquireSessionLockFor(cfg.Session.Campaign, cfg.Session.SessionID)
if err != nil {
t.Fatalf("AcquireSessionLockFor() error = %v", err)
}
defer func() { _ = store.ReleaseSessionLock(lock) }()
restoreWithStoreAndRealPhases(t, fake)
var stdout bytes.Buffer
var stderr bytes.Buffer
code := Execute([]string{"restore", "--config", pipelinePath, "--session", sessionPath}, &stdout, &stderr)
if code == 0 {
t.Fatal("exit code = 0, want non-zero")
}
if !strings.Contains(stderr.String(), "acquire session lock") {
t.Fatalf("stderr = %q, want lock failure", stderr.String())
}
sessionRoot := artifacts.SessionWorkDirForCampaign(workspaceRoot, cfg.Session.Campaign, cfg.Session.SessionID)
if _, err := os.Stat(filepath.Join(sessionRoot, "transcripts", "full.json")); !os.IsNotExist(err) {
t.Fatalf("transcript should not be restored when lock acquisition fails; stat err=%v", err)
}
}
func TestExecuteRestoreInvalidManifestDoesNotCorruptExistingManifest(t *testing.T) {
workspaceRoot := t.TempDir()
pipelinePath, sessionPath := writeValidConfigFiles(t, workspaceRoot)
base := &storage.FakeBackend{}
cfg, sessionPrefix, manifestKey, _ := seedRestoreCommittedState(t, base, pipelinePath, sessionPath)
seedRestoreObject(base, sessionPrefix+"transcripts/full.json", []byte("remote-transcript"))
toggled := &stagedManifestDownloadStore{
delegate: base,
manifestKey: manifestKey,
firstManifest: restoreManifestJSON(t, cfg.Session.SessionID, cfg.Session.Campaign),
secondManifest: []byte("{invalid json"),
manifestReads: 0,
}
sessionRoot := artifacts.SessionWorkDirForCampaign(workspaceRoot, cfg.Session.Campaign, cfg.Session.SessionID)
existing := manifest.New(cfg.Session.SessionID, nowUTC())
existing.Campaign = cfg.Session.Campaign
existingPath := filepath.Join(sessionRoot, "manifest.json")
manifestStore := &manifest.LocalStore{}
if err := manifestStore.Save(context.Background(), existingPath, existing); err != nil {
t.Fatalf("save existing local manifest: %v", err)
}
existingData, err := os.ReadFile(existingPath)
if err != nil {
t.Fatalf("read existing local manifest: %v", err)
}
restoreWithStoreAndRealPhases(t, toggled)
var stdout bytes.Buffer
var stderr bytes.Buffer
code := Execute([]string{"restore", "--config", pipelinePath, "--session", sessionPath, "--force"}, &stdout, &stderr)
if code == 0 {
t.Fatal("exit code = 0, want non-zero")
}
if !strings.Contains(stderr.String(), "validate manifest decode") {
t.Fatalf("stderr = %q, want manifest validation failure", stderr.String())
}
mustReadEquals(t, filepath.Join(sessionRoot, "transcripts", "full.json"), "remote-transcript")
report := mustReadRestoreReport(t, filepath.Join(sessionRoot, "reports", "restore-latest.json"))
if report.Status != "failed" {
t.Fatalf("report status = %q, want failed", report.Status)
}
if strings.TrimSpace(report.Error) == "" {
t.Fatal("report error is empty, want failure context")
}
afterData, err := os.ReadFile(existingPath)
if err != nil {
t.Fatalf("read local manifest after failure: %v", err)
}
if string(afterData) != string(existingData) {
t.Fatalf("local manifest changed after failed restore; before=%q after=%q", string(existingData), string(afterData))
}
}
func TestExecuteRestorePlanPathMismatchFails(t *testing.T) {
cfg := restorePlanConfig(t)
current := restorePlanCurrentState(t, cfg)
store := &storage.FakeBackend{}
seedRestoreObject(store, current.SessionPrefix+"transcripts/full.json", []byte("remote-transcript"))
plan := &RestorePlan{Actions: []RestoreAction{{
Kind: RestoreActionDownload,
RemoteKey: current.SessionPrefix + "transcripts/full.json",
LocalRelativePath: "transcripts/full.json",
LocalPath: "/tmp/escape.txt",
}}}
report, err := newRestoreReport(current, plan, RestorePlanOptions{})
if err != nil {
t.Fatalf("newRestoreReport() error = %v", err)
}
_, err = executeRestorePlan(context.Background(), cfg, current, plan, report, store)
if err == nil {
t.Fatal("expected error, got nil")
}
if !strings.Contains(err.Error(), "local path mismatch") {
t.Fatalf("error = %v, want local path mismatch", err)
}
}
func mustReadRestoreReport(t *testing.T, path string) *RestoreReport {
t.Helper()
data, err := os.ReadFile(path)
if err != nil {
t.Fatalf("ReadFile(%q): %v", path, err)
}
var report RestoreReport
if err := json.Unmarshal(data, &report); err != nil {
t.Fatalf("Unmarshal restore report %q: %v", path, err)
}
return &report
}
func restoreWithStoreAndRealPhases(t *testing.T, objectStore storage.ObjectStore) {
t.Helper()
origStoreFn := newObjectStoreFromConfigFn
origDiscoverFn := discoverRemoteCurrentStateFn
origPlanFn := buildRestorePlanFn
origExecuteFn := executeRestorePlanFn
t.Cleanup(func() {
newObjectStoreFromConfigFn = origStoreFn
discoverRemoteCurrentStateFn = origDiscoverFn
buildRestorePlanFn = origPlanFn
executeRestorePlanFn = origExecuteFn
})
newObjectStoreFromConfigFn = func(context.Context, *config.Config) (storage.ObjectStore, error) {
return objectStore, nil
}
discoverRemoteCurrentStateFn = discoverRemoteCurrentState
buildRestorePlanFn = buildRestorePlan
executeRestorePlanFn = executeRestorePlan
}
func seedRestoreCommittedState(t *testing.T, fake *storage.FakeBackend, pipelinePath, sessionPath string) (*config.Config, string, string, string) {
t.Helper()
cfg, err := config.LoadWithSessionOptions(pipelinePath, sessionPath, config.SessionLoadOptions{})
if err != nil {
t.Fatalf("LoadWithSessionOptions() error = %v", err)
}
if err := config.Validate(cfg); err != nil {
t.Fatalf("Validate() error = %v", err)
}
sessionPrefix := artifacts.S3SessionPrefix(cfg.Pipeline.Storage.S3.RootPrefix, cfg.Session.Campaign, cfg.Session.SessionID)
manifestKey, runIDKey := artifacts.ResolveArchiveCurrentStateKeys(sessionPrefix)
seedRestoreObject(fake, runIDKey, []byte("20260519T010203Z-a1b2c3d4\n"))
seedRestoreObject(fake, manifestKey, restoreManifestJSON(t, cfg.Session.SessionID, cfg.Session.Campaign))
return cfg, sessionPrefix, manifestKey, runIDKey
}
func mustReadEquals(t *testing.T, path, want string) {
t.Helper()
data, err := os.ReadFile(path)
if err != nil {
t.Fatalf("ReadFile(%q): %v", path, err)
}
if string(data) != want {
t.Fatalf("file %q = %q, want %q", path, string(data), want)
}
}
type stagedManifestDownloadStore struct {
delegate *storage.FakeBackend
manifestKey string
firstManifest []byte
secondManifest []byte
manifestReads int
}
func (s *stagedManifestDownloadStore) List(ctx context.Context, prefix string) ([]storage.ObjectInfo, error) {
return s.delegate.List(ctx, prefix)
}
func (s *stagedManifestDownloadStore) Download(ctx context.Context, key, localPath string) error {
if strings.TrimSpace(key) == strings.TrimSpace(s.manifestKey) {
s.manifestReads++
payload := s.secondManifest
if s.manifestReads <= 1 {
payload = s.firstManifest
}
if err := os.MkdirAll(filepath.Dir(localPath), 0o755); err != nil {
return fmt.Errorf("download staged manifest: create parent: %w", err)
}
if err := os.WriteFile(localPath, payload, 0o644); err != nil {
return fmt.Errorf("download staged manifest: write local file: %w", err)
}
return nil
}
return s.delegate.Download(ctx, key, localPath)
}
func (s *stagedManifestDownloadStore) Upload(ctx context.Context, localPath, key string, opts storage.UploadOptions) (storage.ObjectInfo, error) {
return s.delegate.Upload(ctx, localPath, key, opts)
}
func (s *stagedManifestDownloadStore) Exists(ctx context.Context, key string) (bool, error) {
return s.delegate.Exists(ctx, key)
}

View File

@@ -0,0 +1,344 @@
package app
import (
"context"
"fmt"
"io"
"os"
"path"
"path/filepath"
"sort"
"strings"
"gitea.maximumdirect.net/eric/narratio/internal/adapters/storage"
"gitea.maximumdirect.net/eric/narratio/internal/artifacts"
"gitea.maximumdirect.net/eric/narratio/internal/config"
)
// RestoreActionKind identifies one restore planner action.
type RestoreActionKind string
const (
RestoreActionDownload RestoreActionKind = "download"
RestoreActionSkipSame RestoreActionKind = "skip_same"
RestoreActionConflict RestoreActionKind = "conflict"
)
// RestoreAction is one deterministic planner action.
type RestoreAction struct {
Kind RestoreActionKind
RemoteKey string
LocalRelativePath string
LocalPath string
Size int64
ETag string
ExistsLocal bool
SameLocal bool
Conflict bool
Reason string
}
// RestorePlan is the deterministic output of restore planning.
type RestorePlan struct {
Actions []RestoreAction
DownloadCount int
SkipSameCount int
ConflictCount int
}
// RestorePlanOptions control restore planning scope and classification.
type RestorePlanOptions struct {
IncludeAudio bool
Force bool
DryRun bool
}
func buildRestorePlan(ctx context.Context, cfg *config.Config, current *RemoteCurrentState, store storage.ObjectStore, opts RestorePlanOptions) (*RestorePlan, error) {
if cfg == nil || cfg.Pipeline == nil || cfg.Session == nil {
return nil, fmt.Errorf("resolved config with pipeline/session is required")
}
if current == nil {
return nil, fmt.Errorf("remote current state is required")
}
if store == nil {
return nil, fmt.Errorf("remote object store is required")
}
prefix := normalizeRemoteKey(current.SessionPrefix)
if strings.TrimSpace(prefix) == "" {
return nil, fmt.Errorf("remote session prefix is required")
}
if !strings.HasSuffix(prefix, "/") {
prefix += "/"
}
sessionPaths := artifacts.NewLocalStore(cfg.Pipeline.Workspace.Root).SessionPathsFor(cfg.Session.Campaign, cfg.Session.SessionID)
objects, err := store.List(ctx, prefix)
if err != nil {
return nil, fmt.Errorf("list remote session objects under %q: %w", prefix, err)
}
candidates := make(map[string]storage.ObjectInfo, len(objects)+1)
for _, obj := range objects {
key := normalizeRemoteKey(obj.Key)
if key == "" {
continue
}
obj.Key = key
candidates[key] = obj
}
if strings.TrimSpace(current.CurrentManifestKey) != "" {
key := normalizeRemoteKey(current.CurrentManifestKey)
if _, ok := candidates[key]; !ok {
candidates[key] = storage.ObjectInfo{Key: key}
}
}
actions := make([]RestoreAction, 0, len(candidates))
for key, obj := range candidates {
rel, include, err := restoreLocalRelativePathForKey(prefix, normalizeRemoteKey(current.CurrentManifestKey), key, opts.IncludeAudio)
if err != nil {
return nil, fmt.Errorf("map remote key %q: %w", key, err)
}
if !include {
continue
}
localPath, err := joinWithinSessionRoot(sessionPaths.Root, rel)
if err != nil {
return nil, fmt.Errorf("map remote key %q: %w", key, err)
}
action, err := classifyRestoreAction(ctx, store, obj, rel, localPath, opts.Force)
if err != nil {
return nil, fmt.Errorf("classify remote key %q: %w", key, err)
}
actions = append(actions, action)
}
sort.Slice(actions, func(i, j int) bool {
if actions[i].LocalRelativePath == actions[j].LocalRelativePath {
return actions[i].RemoteKey < actions[j].RemoteKey
}
return actions[i].LocalRelativePath < actions[j].LocalRelativePath
})
plan := &RestorePlan{Actions: actions}
for _, action := range actions {
switch action.Kind {
case RestoreActionDownload:
plan.DownloadCount++
case RestoreActionSkipSame:
plan.SkipSameCount++
case RestoreActionConflict:
plan.ConflictCount++
}
}
_ = opts.DryRun
return plan, nil
}
func normalizeRemoteKey(v string) string {
return strings.Trim(strings.ReplaceAll(strings.TrimSpace(v), "\\", "/"), "/")
}
func restoreLocalRelativePathForKey(sessionPrefix, currentManifestKey, key string, includeAudio bool) (string, bool, error) {
if key == "" {
return "", false, nil
}
if key == currentManifestKey {
return config.PathManifestFile, true, nil
}
if !strings.HasPrefix(key, sessionPrefix) {
return "", false, fmt.Errorf("key is outside resolved session prefix %q", sessionPrefix)
}
rel := strings.TrimPrefix(key, sessionPrefix)
rel = strings.TrimSpace(rel)
if rel == "" {
return "", false, nil
}
cleanRel := path.Clean(rel)
if cleanRel == "." || cleanRel == "" {
return "", false, nil
}
if cleanRel == ".." || strings.HasPrefix(cleanRel, "../") || strings.HasPrefix(cleanRel, "/") {
return "", false, fmt.Errorf("key relative path %q escapes session scope", rel)
}
if cleanRel == config.PathManifestFile {
return config.PathManifestFile, true, nil
}
if strings.HasPrefix(cleanRel, config.S3CurrentSegment+"/") {
return "", false, nil
}
if strings.HasPrefix(cleanRel, config.S3RunsSegment+"/") {
return "", false, nil
}
excludedRoots := []string{
config.PathLogsDirSegment,
config.PathReportsDirSegment,
config.PathConfigDirSegment,
config.PathInputsDirSegment,
}
for _, root := range excludedRoots {
if cleanRel == root || strings.HasPrefix(cleanRel, root+"/") {
return "", false, nil
}
}
if cleanRel == config.PathTranscriptsSegment || strings.HasPrefix(cleanRel, config.PathTranscriptsSegment+"/") {
return cleanRel, true, nil
}
if cleanRel == config.PathArtifactsDirSegment || strings.HasPrefix(cleanRel, config.PathArtifactsDirSegment+"/") {
return cleanRel, true, nil
}
if includeAudio && (cleanRel == config.PathAudioDirSegment || strings.HasPrefix(cleanRel, config.PathAudioDirSegment+"/")) {
return cleanRel, true, nil
}
return "", false, nil
}
func joinWithinSessionRoot(sessionRoot, relative string) (string, error) {
if strings.TrimSpace(sessionRoot) == "" {
return "", fmt.Errorf("session root is required")
}
cleanRel := path.Clean(strings.TrimSpace(relative))
if cleanRel == "." || cleanRel == "" {
return "", fmt.Errorf("relative path is required")
}
if cleanRel == ".." || strings.HasPrefix(cleanRel, "../") || strings.HasPrefix(cleanRel, "/") {
return "", fmt.Errorf("relative path escapes session root")
}
abs := filepath.Clean(filepath.Join(sessionRoot, filepath.FromSlash(cleanRel)))
root := filepath.Clean(sessionRoot)
if abs != root && !strings.HasPrefix(abs, root+string(filepath.Separator)) {
return "", fmt.Errorf("resolved local path escapes session root")
}
return abs, nil
}
func classifyRestoreAction(
ctx context.Context,
store storage.ObjectStore,
object storage.ObjectInfo,
localRelPath string,
localPath string,
force bool,
) (RestoreAction, error) {
action := RestoreAction{
RemoteKey: normalizeRemoteKey(object.Key),
LocalRelativePath: localRelPath,
LocalPath: localPath,
Size: object.Size,
ETag: object.ETag,
}
info, err := os.Stat(localPath)
if err != nil {
if os.IsNotExist(err) {
action.Kind = RestoreActionDownload
action.Reason = "local file missing"
return action, nil
}
return RestoreAction{}, fmt.Errorf("stat local file: %w", err)
}
action.ExistsLocal = true
if info.IsDir() {
action.Kind = RestoreActionConflict
action.Conflict = true
action.Reason = "local path is a directory"
return action, nil
}
if object.Size > 0 && info.Size() != object.Size {
if force {
action.Kind = RestoreActionDownload
action.Reason = "local file differs (size mismatch); overwrite with --force"
return action, nil
}
action.Kind = RestoreActionConflict
action.Conflict = true
action.Reason = "local file differs (size mismatch)"
return action, nil
}
localDigest, err := artifacts.SHA256File(localPath)
if err != nil {
return RestoreAction{}, fmt.Errorf("checksum local file: %w", err)
}
remotePath, err := downloadObjectToTemp(ctx, store, action.RemoteKey, "narratio-restore-plan-remote-*.tmp")
if err != nil {
return RestoreAction{}, fmt.Errorf("download remote object: %w", err)
}
defer func() { _ = os.Remove(remotePath) }()
remoteDigest, err := artifacts.SHA256File(remotePath)
if err != nil {
return RestoreAction{}, fmt.Errorf("checksum remote object: %w", err)
}
if remoteDigest == localDigest {
action.Kind = RestoreActionSkipSame
action.SameLocal = true
action.Reason = "local file matches remote content"
return action, nil
}
if force {
action.Kind = RestoreActionDownload
action.Reason = "local file differs; overwrite with --force"
return action, nil
}
action.Kind = RestoreActionConflict
action.Conflict = true
action.Reason = "local file differs"
return action, nil
}
func writeRestorePlan(out io.Writer, current *RemoteCurrentState, plan *RestorePlan, opts RestorePlanOptions) error {
if out == nil {
return fmt.Errorf("output writer is required")
}
if current == nil {
return fmt.Errorf("remote current state is required")
}
if plan == nil {
return fmt.Errorf("restore plan is required")
}
if _, err := fmt.Fprintf(
out,
"restore plan: session %s/%s run=%s actions=%d download=%d skip_same=%d conflict=%d dry_run=%t force=%t include_audio=%t\n",
current.Campaign,
current.SessionID,
current.RunID,
len(plan.Actions),
plan.DownloadCount,
plan.SkipSameCount,
plan.ConflictCount,
opts.DryRun,
opts.Force,
opts.IncludeAudio,
); err != nil {
return err
}
for _, action := range plan.Actions {
if _, err := fmt.Fprintf(out, "%s %s <- %s", action.Kind, action.LocalRelativePath, action.RemoteKey); err != nil {
return err
}
if strings.TrimSpace(action.Reason) != "" {
if _, err := fmt.Fprintf(out, " (%s)", action.Reason); err != nil {
return err
}
}
if _, err := fmt.Fprintln(out); err != nil {
return err
}
}
return nil
}

View File

@@ -0,0 +1,199 @@
package app
import (
"context"
"path/filepath"
"reflect"
"strings"
"testing"
"gitea.maximumdirect.net/eric/narratio/internal/adapters/storage"
"gitea.maximumdirect.net/eric/narratio/internal/artifacts"
"gitea.maximumdirect.net/eric/narratio/internal/config"
)
func TestRestorePlanDefaultScope(t *testing.T) {
cfg := restorePlanConfig(t)
current := restorePlanCurrentState(t, cfg)
store := &storage.FakeBackend{}
seedRestoreObject(store, current.CurrentManifestKey, []byte(`{"session_id":"2026-05-03"}`))
seedRestoreObject(store, current.SessionPrefix+"transcripts/full.json", []byte(`{"segments":[1]}`))
seedRestoreObject(store, current.SessionPrefix+"artifacts/session_recap.md", []byte("# recap\n"))
seedRestoreObject(store, current.SessionPrefix+"audio/alice.flac", []byte("audio"))
seedRestoreObject(store, current.SessionPrefix+"runs/20260519T010203Z-a1b2/manifest.json", []byte("{}"))
seedRestoreObject(store, current.SessionPrefix+"logs/archive.log", []byte("log"))
plan, err := buildRestorePlan(context.Background(), cfg, current, store, RestorePlanOptions{})
if err != nil {
t.Fatalf("buildRestorePlan() error = %v", err)
}
got := actionRelPaths(plan.Actions)
want := []string{"artifacts/session_recap.md", "manifest.json", "transcripts/full.json"}
if !reflect.DeepEqual(got, want) {
t.Fatalf("action local paths = %#v, want %#v", got, want)
}
if plan.DownloadCount != 3 || plan.SkipSameCount != 0 || plan.ConflictCount != 0 {
t.Fatalf("counts = download=%d skip_same=%d conflict=%d, want 3/0/0", plan.DownloadCount, plan.SkipSameCount, plan.ConflictCount)
}
}
func TestRestorePlanIncludeAudio(t *testing.T) {
cfg := restorePlanConfig(t)
current := restorePlanCurrentState(t, cfg)
store := &storage.FakeBackend{}
seedRestoreObject(store, current.CurrentManifestKey, []byte(`{"session_id":"2026-05-03"}`))
seedRestoreObject(store, current.SessionPrefix+"audio/alice.flac", []byte("audio"))
plan, err := buildRestorePlan(context.Background(), cfg, current, store, RestorePlanOptions{IncludeAudio: true})
if err != nil {
t.Fatalf("buildRestorePlan() error = %v", err)
}
got := actionRelPaths(plan.Actions)
want := []string{"audio/alice.flac", "manifest.json"}
if !reflect.DeepEqual(got, want) {
t.Fatalf("action local paths = %#v, want %#v", got, want)
}
}
func TestRestorePlanClassifiesSameAndConflict(t *testing.T) {
cfg := restorePlanConfig(t)
current := restorePlanCurrentState(t, cfg)
store := &storage.FakeBackend{}
seedRestoreObject(store, current.CurrentManifestKey, []byte(`{"session_id":"2026-05-03"}`))
seedRestoreObject(store, current.SessionPrefix+"transcripts/full.json", []byte(`{"segments":[1]}`))
seedRestoreObject(store, current.SessionPrefix+"artifacts/session_recap.md", []byte("remote-content\n"))
sessionRoot := artifacts.SessionWorkDirForCampaign(cfg.Pipeline.Workspace.Root, cfg.Session.Campaign, cfg.Session.SessionID)
mustWriteTestFile(t, filepath.Join(sessionRoot, "transcripts", "full.json"), `{"segments":[1]}`)
mustWriteTestFile(t, filepath.Join(sessionRoot, "artifacts", "session_recap.md"), "different\n")
plan, err := buildRestorePlan(context.Background(), cfg, current, store, RestorePlanOptions{})
if err != nil {
t.Fatalf("buildRestorePlan() error = %v", err)
}
if plan.SkipSameCount != 1 {
t.Fatalf("SkipSameCount = %d, want 1", plan.SkipSameCount)
}
if plan.ConflictCount != 1 {
t.Fatalf("ConflictCount = %d, want 1", plan.ConflictCount)
}
actionByRel := map[string]RestoreAction{}
for _, action := range plan.Actions {
actionByRel[action.LocalRelativePath] = action
}
if actionByRel["transcripts/full.json"].Kind != RestoreActionSkipSame {
t.Fatalf("transcripts/full.json kind = %q, want %q", actionByRel["transcripts/full.json"].Kind, RestoreActionSkipSame)
}
if actionByRel["artifacts/session_recap.md"].Kind != RestoreActionConflict {
t.Fatalf("artifacts/session_recap.md kind = %q, want %q", actionByRel["artifacts/session_recap.md"].Kind, RestoreActionConflict)
}
}
func TestRestorePlanForceTurnsConflictsIntoDownloads(t *testing.T) {
cfg := restorePlanConfig(t)
current := restorePlanCurrentState(t, cfg)
store := &storage.FakeBackend{}
seedRestoreObject(store, current.CurrentManifestKey, []byte(`{"session_id":"2026-05-03"}`))
seedRestoreObject(store, current.SessionPrefix+"artifacts/session_recap.md", []byte("remote-content\n"))
sessionRoot := artifacts.SessionWorkDirForCampaign(cfg.Pipeline.Workspace.Root, cfg.Session.Campaign, cfg.Session.SessionID)
mustWriteTestFile(t, filepath.Join(sessionRoot, "artifacts", "session_recap.md"), "different\n")
plan, err := buildRestorePlan(context.Background(), cfg, current, store, RestorePlanOptions{Force: true})
if err != nil {
t.Fatalf("buildRestorePlan() error = %v", err)
}
actionByRel := map[string]RestoreAction{}
for _, action := range plan.Actions {
actionByRel[action.LocalRelativePath] = action
}
recap := actionByRel["artifacts/session_recap.md"]
if recap.Kind != RestoreActionDownload {
t.Fatalf("artifacts/session_recap.md kind = %q, want %q", recap.Kind, RestoreActionDownload)
}
if plan.ConflictCount != 0 {
t.Fatalf("ConflictCount = %d, want 0", plan.ConflictCount)
}
}
func TestRestorePlanTraversalUnsafeKeyFails(t *testing.T) {
cfg := restorePlanConfig(t)
current := restorePlanCurrentState(t, cfg)
store := &storage.FakeBackend{}
seedRestoreObject(store, current.CurrentManifestKey, []byte(`{"session_id":"2026-05-03"}`))
seedRestoreObject(store, current.SessionPrefix+"artifacts/../../escape.txt", []byte("bad"))
_, err := buildRestorePlan(context.Background(), cfg, current, store, RestorePlanOptions{})
if err == nil {
t.Fatal("expected error, got nil")
}
if !strings.Contains(err.Error(), "escapes session scope") {
t.Fatalf("error = %v, want traversal safety failure", err)
}
}
func seedRestoreObject(store *storage.FakeBackend, key string, data []byte) {
store.SeedObject(storage.FakeObject{Key: key, Data: data})
}
func actionRelPaths(actions []RestoreAction) []string {
out := make([]string, 0, len(actions))
for _, action := range actions {
out = append(out, action.LocalRelativePath)
}
return out
}
func restorePlanConfig(t *testing.T) *config.Config {
t.Helper()
workspaceRoot := t.TempDir()
return &config.Config{
Pipeline: &config.PipelineConfig{
Workspace: config.WorkspaceConfig{Root: workspaceRoot},
},
Session: &config.SessionConfig{
SessionID: "2026-05-03",
Campaign: "sample-campaign",
},
}
}
func restorePlanCurrentState(t *testing.T, cfg *config.Config) *RemoteCurrentState {
t.Helper()
sessionPrefix := artifacts.S3SessionPrefix("dnd", cfg.Session.Campaign, cfg.Session.SessionID)
manifestKey, runIDKey := artifacts.ResolveArchiveCurrentStateKeys(sessionPrefix)
return &RemoteCurrentState{
Bucket: "test-bucket",
SessionPrefix: sessionPrefix,
CurrentManifestKey: manifestKey,
CurrentRunIDKey: runIDKey,
RunID: "20260519T010203Z-a1b2c3d4",
SessionID: cfg.Session.SessionID,
Campaign: cfg.Session.Campaign,
}
}
func TestWriteRestorePlan(t *testing.T) {
current := &RemoteCurrentState{Campaign: "sample-campaign", SessionID: "2026-05-03", RunID: "r-1"}
plan := &RestorePlan{Actions: []RestoreAction{{Kind: RestoreActionDownload, LocalRelativePath: "manifest.json", RemoteKey: "k", Reason: "local file missing"}}, DownloadCount: 1}
var out strings.Builder
if err := writeRestorePlan(&out, current, plan, RestorePlanOptions{DryRun: true}); err != nil {
t.Fatalf("writeRestorePlan() error = %v", err)
}
text := out.String()
if !strings.Contains(text, "restore plan: session sample-campaign/2026-05-03 run=r-1") {
t.Fatalf("output = %q, want plan summary", text)
}
if !strings.Contains(text, "download manifest.json <- k") {
t.Fatalf("output = %q, want action line", text)
}
}

View File

@@ -0,0 +1,243 @@
package app
import (
"encoding/json"
"fmt"
"io"
"path/filepath"
"strings"
"gitea.maximumdirect.net/eric/narratio/internal/artifacts"
"gitea.maximumdirect.net/eric/narratio/internal/config"
)
// RestoreReport is the durable restore diagnostic model.
type RestoreReport struct {
GeneratedAt string `json:"generated_at"`
SessionID string `json:"session_id"`
Campaign string `json:"campaign"`
RunID string `json:"run_id"`
DryRun bool `json:"dry_run"`
Force bool `json:"force"`
IncludeAudio bool `json:"include_audio"`
Status string `json:"status"`
Error string `json:"error,omitempty"`
Plan RestorePlanSummary `json:"plan"`
Execution RestoreExecutionStats `json:"execution"`
Actions []RestoreReportAction `json:"actions"`
reportPathRel string
}
type RestorePlanSummary struct {
Actions int `json:"actions"`
Download int `json:"download"`
SkipSame int `json:"skip_same"`
Conflicts int `json:"conflicts"`
}
type RestoreExecutionStats struct {
Downloaded int `json:"downloaded"`
Failed int `json:"failed"`
}
type RestoreReportAction struct {
Kind string `json:"kind"`
LocalRelativePath string `json:"local_relative_path"`
RemoteKey string `json:"remote_key"`
Reason string `json:"reason,omitempty"`
Status string `json:"status"`
Error string `json:"error,omitempty"`
}
func newRestoreReport(current *RemoteCurrentState, plan *RestorePlan, opts RestorePlanOptions) (*RestoreReport, error) {
if current == nil {
return nil, fmt.Errorf("remote current state is required")
}
if plan == nil {
return nil, fmt.Errorf("restore plan is required")
}
r := &RestoreReport{
GeneratedAt: nowUTC().Format("2006-01-02T15:04:05.999999999Z07:00"),
SessionID: current.SessionID,
Campaign: current.Campaign,
RunID: current.RunID,
DryRun: opts.DryRun,
Force: opts.Force,
IncludeAudio: opts.IncludeAudio,
Status: "planned",
Plan: RestorePlanSummary{
Actions: len(plan.Actions),
Download: plan.DownloadCount,
SkipSame: plan.SkipSameCount,
Conflicts: plan.ConflictCount,
},
Actions: make([]RestoreReportAction, 0, len(plan.Actions)),
reportPathRel: filepath.ToSlash(filepath.Join(config.PathReportsDirSegment, "restore-latest.json")),
}
for _, action := range plan.Actions {
r.Actions = append(r.Actions, RestoreReportAction{
Kind: string(action.Kind),
LocalRelativePath: action.LocalRelativePath,
RemoteKey: action.RemoteKey,
Reason: action.Reason,
Status: initialRestoreActionStatus(action.Kind),
})
}
return r, nil
}
func initialRestoreActionStatus(kind RestoreActionKind) string {
switch kind {
case RestoreActionDownload:
return "planned_download"
case RestoreActionSkipSame:
return "skipped_same"
case RestoreActionConflict:
return "conflict"
default:
return "planned"
}
}
func (r *RestoreReport) markDownloaded(action RestoreAction) {
if r == nil {
return
}
if idx := r.findAction(action); idx >= 0 {
r.Actions[idx].Status = "downloaded"
r.Actions[idx].Error = ""
}
r.Execution.Downloaded++
}
func (r *RestoreReport) markFailed(action RestoreAction, err error) {
if r == nil {
return
}
if idx := r.findAction(action); idx >= 0 {
r.Actions[idx].Status = "failed"
if err != nil {
r.Actions[idx].Error = err.Error()
}
}
r.Execution.Failed++
}
func (r *RestoreReport) setFailed(err error) {
if r == nil {
return
}
r.Status = "failed"
if err != nil {
r.Error = err.Error()
}
}
func (r *RestoreReport) setSucceeded() {
if r == nil {
return
}
r.Status = "succeeded"
r.Error = ""
}
func (r *RestoreReport) findAction(action RestoreAction) int {
if r == nil {
return -1
}
for i := range r.Actions {
if r.Actions[i].LocalRelativePath == action.LocalRelativePath && r.Actions[i].RemoteKey == action.RemoteKey {
return i
}
}
return -1
}
func writeRestoreDryRunSummary(out io.Writer, report *RestoreReport) error {
if out == nil {
return fmt.Errorf("output writer is required")
}
if report == nil {
return fmt.Errorf("restore report is required")
}
if _, err := fmt.Fprintf(out, "Restore plan for %s/%s\n", report.Campaign, report.SessionID); err != nil {
return err
}
if _, err := fmt.Fprintf(out, "Remote run: %s\n", report.RunID); err != nil {
return err
}
if _, err := fmt.Fprintf(out, "Would download: %d\n", report.Plan.Download); err != nil {
return err
}
if _, err := fmt.Fprintf(out, "Would skip unchanged: %d\n", report.Plan.SkipSame); err != nil {
return err
}
if _, err := fmt.Fprintf(out, "Conflicts: %d\n", report.Plan.Conflicts); err != nil {
return err
}
for _, action := range report.Actions {
line := ""
switch action.Status {
case "planned_download":
line = "Would download: " + action.LocalRelativePath
case "skipped_same":
line = "Would skip unchanged: " + action.LocalRelativePath
case "conflict":
line = "Conflict: " + action.LocalRelativePath
default:
line = strings.TrimSpace(action.Kind) + ": " + action.LocalRelativePath
}
if _, err := fmt.Fprintln(out, line); err != nil {
return err
}
}
return nil
}
func writeRestoreSuccessSummary(out io.Writer, report *RestoreReport) error {
if out == nil {
return fmt.Errorf("output writer is required")
}
if report == nil {
return fmt.Errorf("restore report is required")
}
if _, err := fmt.Fprintf(out, "Restored session archive for %s/%s\n", report.Campaign, report.SessionID); err != nil {
return err
}
if _, err := fmt.Fprintf(out, "Remote run: %s\n", report.RunID); err != nil {
return err
}
if _, err := fmt.Fprintf(out, "Downloaded: %d\n", report.Execution.Downloaded); err != nil {
return err
}
if _, err := fmt.Fprintf(out, "Skipped unchanged: %d\n", report.Plan.SkipSame); err != nil {
return err
}
if _, err := fmt.Fprintf(out, "Conflicts: %d\n", report.Plan.Conflicts); err != nil {
return err
}
return nil
}
func persistRestoreReport(store artifacts.Store, cfg *config.Config, report *RestoreReport) (string, error) {
if store == nil {
return "", fmt.Errorf("artifact store is required")
}
if cfg == nil || cfg.Pipeline == nil || cfg.Session == nil {
return "", fmt.Errorf("resolved config with pipeline/session is required")
}
if report == nil {
return "", fmt.Errorf("restore report is required")
}
sessionRoot := artifacts.SessionWorkDirForCampaign(cfg.Pipeline.Workspace.Root, cfg.Session.Campaign, cfg.Session.SessionID)
reportPath := filepath.Join(sessionRoot, filepath.FromSlash(report.reportPathRel))
payload, err := json.MarshalIndent(report, "", " ")
if err != nil {
return "", fmt.Errorf("marshal restore report: %w", err)
}
payload = append(payload, '\n')
if err := store.WriteFileAtomic(reportPath, payload, 0o644); err != nil {
return "", fmt.Errorf("write restore report %q: %w", reportPath, err)
}
return reportPath, nil
}

View File

@@ -0,0 +1,409 @@
package app
import (
"bytes"
"context"
"fmt"
"os"
"path/filepath"
"strings"
"testing"
"gitea.maximumdirect.net/eric/narratio/internal/adapters/storage"
"gitea.maximumdirect.net/eric/narratio/internal/artifacts"
"gitea.maximumdirect.net/eric/narratio/internal/config"
)
func TestExecuteRestoreHelp(t *testing.T) {
var stdout bytes.Buffer
var stderr bytes.Buffer
code := Execute([]string{"restore", "--help"}, &stdout, &stderr)
if code != 0 {
t.Fatalf("exit code = %d, want 0", code)
}
if stderr.Len() != 0 {
t.Fatalf("stderr = %q, want empty", stderr.String())
}
out := stdout.String()
if !strings.Contains(out, "Usage: narratio restore") {
t.Fatalf("stdout = %q, want restore usage", out)
}
if !strings.Contains(out, "--include-audio") {
t.Fatalf("stdout = %q, want --include-audio flag", out)
}
}
func TestExecuteRestoreRecognizedAndReturnsNYI(t *testing.T) {
origStoreFn := newObjectStoreFromConfigFn
origDiscoverFn := discoverRemoteCurrentStateFn
origPlanFn := buildRestorePlanFn
origExecuteFn := executeRestorePlanFn
t.Cleanup(func() {
newObjectStoreFromConfigFn = origStoreFn
discoverRemoteCurrentStateFn = origDiscoverFn
buildRestorePlanFn = origPlanFn
executeRestorePlanFn = origExecuteFn
})
newObjectStoreFromConfigFn = func(context.Context, *config.Config) (storage.ObjectStore, error) {
return &storage.FakeBackend{}, nil
}
discoverRemoteCurrentStateFn = func(context.Context, *config.Config, storage.ObjectStore) (*RemoteCurrentState, error) {
return &RemoteCurrentState{
SessionID: "2026-05-03",
Campaign: "sample-campaign",
RunID: "20260519T010203Z-a1b2c3d4",
}, nil
}
buildRestorePlanFn = func(context.Context, *config.Config, *RemoteCurrentState, storage.ObjectStore, RestorePlanOptions) (*RestorePlan, error) {
return &RestorePlan{
Actions: []RestoreAction{
{
Kind: RestoreActionDownload,
LocalRelativePath: "manifest.json",
RemoteKey: "dnd/campaigns/sample-campaign/sessions/2026-05-03/current/manifest.json",
Reason: "local file missing",
},
},
DownloadCount: 1,
}, nil
}
workspaceRoot := t.TempDir()
pipelinePath, sessionPath := writeValidConfigFiles(t, workspaceRoot)
var stdout bytes.Buffer
var stderr bytes.Buffer
code := Execute(
[]string{
"restore",
"--config", pipelinePath,
"--session", sessionPath,
"--session-id", "2026-05-03",
"--dry-run",
"--force",
"--include-audio",
},
&stdout,
&stderr,
)
if code != 0 {
t.Fatalf("exit code = %d, want 0 for --dry-run restore planning; stderr=%q", code, stderr.String())
}
if stderr.Len() != 0 {
t.Fatalf("stderr = %q, want empty", stderr.String())
}
outText := stdout.String()
if !strings.Contains(outText, "Restore plan for sample-campaign/2026-05-03") {
t.Fatalf("stdout = %q, want restore plan summary", outText)
}
if !strings.Contains(outText, "Would download: 1") {
t.Fatalf("stdout = %q, want plan count output", outText)
}
if !strings.Contains(outText, "Would download: manifest.json") {
t.Fatalf("stdout = %q, want action output", outText)
}
manifestPath := artifacts.SessionManifestPathForCampaign(workspaceRoot, "sample-campaign", "2026-05-03")
if _, err := os.Stat(manifestPath); !os.IsNotExist(err) {
t.Fatalf("manifest should not be created during phase-4 restore planning; stat err=%v", err)
}
reportPath := filepath.Join(workspaceRoot, "work", "sample-campaign", "2026-05-03", "reports", "restore-latest.json")
if _, err := os.Stat(reportPath); !os.IsNotExist(err) {
t.Fatalf("restore report should not be written during dry-run; stat err=%v", err)
}
}
func TestExecuteRestoreRejectsUnexpectedPositionalArguments(t *testing.T) {
workspaceRoot := t.TempDir()
pipelinePath, sessionPath := writeValidConfigFiles(t, workspaceRoot)
var stdout bytes.Buffer
var stderr bytes.Buffer
code := Execute([]string{"restore", "--config", pipelinePath, "--session", sessionPath, "extra"}, &stdout, &stderr)
if code == 0 {
t.Fatal("exit code = 0, want non-zero")
}
if !strings.Contains(stderr.String(), "restore: unexpected positional arguments") {
t.Fatalf("stderr = %q, want positional-args failure", stderr.String())
}
}
func TestExecuteRestoreFailsWhenStorageBackendNotConfigured(t *testing.T) {
origStoreFn := newObjectStoreFromConfigFn
origDiscoverFn := discoverRemoteCurrentStateFn
t.Cleanup(func() {
newObjectStoreFromConfigFn = origStoreFn
discoverRemoteCurrentStateFn = origDiscoverFn
})
workspaceRoot := t.TempDir()
pipelinePath, sessionPath := writeRestoreConfigWithoutStorage(t, workspaceRoot)
var stdout bytes.Buffer
var stderr bytes.Buffer
code := Execute([]string{"restore", "--config", pipelinePath, "--session", sessionPath}, &stdout, &stderr)
if code == 0 {
t.Fatal("exit code = 0, want non-zero")
}
if !strings.Contains(stderr.String(), "no remote object store backend is configured") {
t.Fatalf("stderr = %q, want storage backend preflight failure", stderr.String())
}
}
func TestExecuteRestoreDiscoveryErrorSurfaced(t *testing.T) {
origStoreFn := newObjectStoreFromConfigFn
origDiscoverFn := discoverRemoteCurrentStateFn
t.Cleanup(func() {
newObjectStoreFromConfigFn = origStoreFn
discoverRemoteCurrentStateFn = origDiscoverFn
})
newObjectStoreFromConfigFn = func(context.Context, *config.Config) (storage.ObjectStore, error) {
return &storage.FakeBackend{}, nil
}
discoverRemoteCurrentStateFn = func(context.Context, *config.Config, storage.ObjectStore) (*RemoteCurrentState, error) {
return nil, fmt.Errorf("remote current run pointer missing: %q", "dnd/campaigns/sample-campaign/sessions/2026-05-03/current/run_id.txt")
}
workspaceRoot := t.TempDir()
pipelinePath, sessionPath := writeValidConfigFiles(t, workspaceRoot)
var stdout bytes.Buffer
var stderr bytes.Buffer
code := Execute([]string{"restore", "--config", pipelinePath, "--session", sessionPath}, &stdout, &stderr)
if code == 0 {
t.Fatal("exit code = 0, want non-zero")
}
if !strings.Contains(stderr.String(), "remote current run pointer missing") {
t.Fatalf("stderr = %q, want discovery error context", stderr.String())
}
}
func TestExecuteRestoreLoadsSecretsBeforeObjectStoreInit(t *testing.T) {
origStoreFn := newObjectStoreFromConfigFn
origDiscoverFn := discoverRemoteCurrentStateFn
origPlanFn := buildRestorePlanFn
origExecuteFn := executeRestorePlanFn
t.Cleanup(func() {
newObjectStoreFromConfigFn = origStoreFn
discoverRemoteCurrentStateFn = origDiscoverFn
buildRestorePlanFn = origPlanFn
executeRestorePlanFn = origExecuteFn
})
const accessKeyEnv = "OBJECT_STORAGE_KEY_ID"
const secretKeyEnv = "OBJECT_STORAGE_KEY"
restoreEnv := func(name string) {
value, exists := os.LookupEnv(name)
_ = os.Unsetenv(name)
t.Cleanup(func() {
if exists {
_ = os.Setenv(name, value)
return
}
_ = os.Unsetenv(name)
})
}
restoreEnv(accessKeyEnv)
restoreEnv(secretKeyEnv)
workspaceRoot := t.TempDir()
pipelinePath, sessionPath := writeValidConfigFiles(t, workspaceRoot)
secretsDir := filepath.Join(t.TempDir(), "secrets")
mustWriteTestFile(t, filepath.Join(secretsDir, accessKeyEnv), "test-access-key-id\n")
mustWriteTestFile(t, filepath.Join(secretsDir, secretKeyEnv), "test-secret-key\n")
f, err := os.OpenFile(pipelinePath, os.O_APPEND|os.O_WRONLY, 0)
if err != nil {
t.Fatalf("open pipeline config for append: %v", err)
}
defer f.Close()
if _, err := f.WriteString("\nsecrets:\n env_dir: " + secretsDir + "\n"); err != nil {
t.Fatalf("append secrets config: %v", err)
}
storeInitCalled := false
newObjectStoreFromConfigFn = func(context.Context, *config.Config) (storage.ObjectStore, error) {
storeInitCalled = true
gotID, okID := os.LookupEnv(accessKeyEnv)
if !okID || gotID != "test-access-key-id" {
return nil, fmt.Errorf("missing or unexpected %s: %q (set=%t)", accessKeyEnv, gotID, okID)
}
gotSecret, okSecret := os.LookupEnv(secretKeyEnv)
if !okSecret || gotSecret != "test-secret-key" {
return nil, fmt.Errorf("missing or unexpected %s: %q (set=%t)", secretKeyEnv, gotSecret, okSecret)
}
return &storage.FakeBackend{}, nil
}
discoverRemoteCurrentStateFn = func(context.Context, *config.Config, storage.ObjectStore) (*RemoteCurrentState, error) {
return &RemoteCurrentState{
SessionID: "2026-05-03",
Campaign: "sample-campaign",
RunID: "20260519T010203Z-a1b2c3d4",
}, nil
}
buildRestorePlanFn = func(context.Context, *config.Config, *RemoteCurrentState, storage.ObjectStore, RestorePlanOptions) (*RestorePlan, error) {
return &RestorePlan{
Actions: []RestoreAction{
{
Kind: RestoreActionDownload,
LocalRelativePath: "manifest.json",
RemoteKey: "dnd/campaigns/sample-campaign/sessions/2026-05-03/current/manifest.json",
Reason: "local file missing",
},
},
DownloadCount: 1,
}, nil
}
var stdout bytes.Buffer
var stderr bytes.Buffer
code := Execute(
[]string{
"restore",
"--config", pipelinePath,
"--session", sessionPath,
"--session-id", "2026-05-03",
"--dry-run",
},
&stdout,
&stderr,
)
if code != 0 {
t.Fatalf("exit code = %d, want 0; stderr=%q", code, stderr.String())
}
if !storeInitCalled {
t.Fatal("expected object store initialization to be called")
}
}
func TestExecuteRestoreNonDryRunConflictFailsBeforeNYI(t *testing.T) {
origStoreFn := newObjectStoreFromConfigFn
origDiscoverFn := discoverRemoteCurrentStateFn
origPlanFn := buildRestorePlanFn
origExecuteFn := executeRestorePlanFn
t.Cleanup(func() {
newObjectStoreFromConfigFn = origStoreFn
discoverRemoteCurrentStateFn = origDiscoverFn
buildRestorePlanFn = origPlanFn
executeRestorePlanFn = origExecuteFn
})
newObjectStoreFromConfigFn = func(context.Context, *config.Config) (storage.ObjectStore, error) {
return &storage.FakeBackend{}, nil
}
discoverRemoteCurrentStateFn = func(context.Context, *config.Config, storage.ObjectStore) (*RemoteCurrentState, error) {
return &RemoteCurrentState{
SessionID: "2026-05-03",
Campaign: "sample-campaign",
RunID: "20260519T010203Z-a1b2c3d4",
}, nil
}
buildRestorePlanFn = func(context.Context, *config.Config, *RemoteCurrentState, storage.ObjectStore, RestorePlanOptions) (*RestorePlan, error) {
return &RestorePlan{
Actions: []RestoreAction{
{Kind: RestoreActionConflict, LocalRelativePath: "transcripts/full.json", RemoteKey: "k", Reason: "local file differs"},
},
ConflictCount: 1,
}, nil
}
workspaceRoot := t.TempDir()
pipelinePath, sessionPath := writeValidConfigFiles(t, workspaceRoot)
var stdout bytes.Buffer
var stderr bytes.Buffer
code := Execute([]string{"restore", "--config", pipelinePath, "--session", sessionPath}, &stdout, &stderr)
if code == 0 {
t.Fatal("exit code = 0, want non-zero")
}
if stdout.Len() != 0 {
t.Fatalf("stdout = %q, want empty on conflict failure", stdout.String())
}
if !strings.Contains(stderr.String(), "restore conflict: 1 conflicting path(s); rerun with --force to overwrite") {
t.Fatalf("stderr = %q, want conflict failure", stderr.String())
}
if strings.Contains(stderr.String(), "phase 4: restore execution") {
t.Fatalf("stderr = %q, should fail before phase-4 NYI boundary", stderr.String())
}
}
func TestExecuteRestoreNonDryRunForceExecutesPlan(t *testing.T) {
origStoreFn := newObjectStoreFromConfigFn
origDiscoverFn := discoverRemoteCurrentStateFn
origPlanFn := buildRestorePlanFn
origExecuteFn := executeRestorePlanFn
t.Cleanup(func() {
newObjectStoreFromConfigFn = origStoreFn
discoverRemoteCurrentStateFn = origDiscoverFn
buildRestorePlanFn = origPlanFn
executeRestorePlanFn = origExecuteFn
})
newObjectStoreFromConfigFn = func(context.Context, *config.Config) (storage.ObjectStore, error) {
return &storage.FakeBackend{}, nil
}
discoverRemoteCurrentStateFn = func(context.Context, *config.Config, storage.ObjectStore) (*RemoteCurrentState, error) {
return &RemoteCurrentState{
SessionID: "2026-05-03",
Campaign: "sample-campaign",
RunID: "20260519T010203Z-a1b2c3d4",
}, nil
}
buildRestorePlanFn = func(context.Context, *config.Config, *RemoteCurrentState, storage.ObjectStore, RestorePlanOptions) (*RestorePlan, error) {
return &RestorePlan{
Actions: []RestoreAction{
{Kind: RestoreActionDownload, LocalRelativePath: "transcripts/full.json", RemoteKey: "k", Reason: "local file differs; overwrite with --force"},
},
DownloadCount: 1,
}, nil
}
executeRestorePlanFn = func(context.Context, *config.Config, *RemoteCurrentState, *RestorePlan, *RestoreReport, storage.ObjectStore) (*RestoreExecutionResult, error) {
return &RestoreExecutionResult{DownloadedCount: 1}, nil
}
workspaceRoot := t.TempDir()
pipelinePath, sessionPath := writeValidConfigFiles(t, workspaceRoot)
var stdout bytes.Buffer
var stderr bytes.Buffer
code := Execute([]string{"restore", "--config", pipelinePath, "--session", sessionPath, "--force"}, &stdout, &stderr)
if code != 0 {
t.Fatalf("exit code = %d, want 0; stderr=%q", code, stderr.String())
}
if !strings.Contains(stdout.String(), "Restored session archive for sample-campaign/2026-05-03") {
t.Fatalf("stdout = %q, want completion summary", stdout.String())
}
if stderr.Len() != 0 {
t.Fatalf("stderr = %q, want empty", stderr.String())
}
}
func writeRestoreConfigWithoutStorage(t *testing.T, workspaceRoot string) (string, string) {
t.Helper()
dir := t.TempDir()
pipelinePath := filepath.Join(dir, "pipeline.yml")
sessionPath := filepath.Join(dir, "session.yml")
pipelineYAML := `workspace:
root: ` + workspaceRoot + `
whisperx:
transcribe_url: https://example.com/transcribe
`
sessionYAML := `session_id: 2026-05-03
campaign: sample-campaign
inputs:
audio_dir: ./audio
speakers_file: ./speakers.yml
autocorrect_file: ./autocorrect.yml
glossary_file: ./glossary.yml
`
if err := os.WriteFile(pipelinePath, []byte(pipelineYAML), 0o644); err != nil {
t.Fatalf("write pipeline config: %v", err)
}
if err := os.WriteFile(sessionPath, []byte(sessionYAML), 0o644); err != nil {
t.Fatalf("write session config: %v", err)
}
mustWriteTestFile(t, filepath.Join(dir, "speakers.yml"), "alice: alice.flac\n")
mustWriteTestFile(t, filepath.Join(dir, "autocorrect.yml"), "[]\n")
mustWriteTestFile(t, filepath.Join(dir, "glossary.yml"), "[]\n")
mustWriteTestFile(t, filepath.Join(dir, "audio", "alice.flac"), "audio-bytes")
return pipelinePath, sessionPath
}

View File

@@ -0,0 +1,178 @@
package app
import (
"bytes"
"context"
"os"
"path/filepath"
"strings"
"testing"
"time"
"gitea.maximumdirect.net/eric/narratio/internal/adapters/scriptorium"
"gitea.maximumdirect.net/eric/narratio/internal/adapters/storage"
"gitea.maximumdirect.net/eric/narratio/internal/artifacts"
"gitea.maximumdirect.net/eric/narratio/internal/config"
"gitea.maximumdirect.net/eric/narratio/internal/manifest"
"gitea.maximumdirect.net/eric/narratio/internal/stage"
)
func TestRestoreThenRunStageForceAnalyzeUsesRestoredDurableState(t *testing.T) {
workspaceRoot := t.TempDir()
pipelinePath, sessionPath := writeValidConfigFilesWithScriptoriumArtifacts(t, workspaceRoot)
fake := &storage.FakeBackend{}
cfg, sessionPrefix, manifestKey, runIDKey := seedRestoreCommittedState(t, fake, pipelinePath, sessionPath)
seedRestoreObject(fake, runIDKey, []byte("20260519T010203Z-a1b2c3d4\n"))
seedRestoreObject(fake, manifestKey, restoreWorkflowManifestJSON(t, cfg.Session.SessionID, cfg.Session.Campaign))
seedRestoreObject(fake, sessionPrefix+"transcripts/full.json", []byte(`{"segments":[1,2,3]}`+"\n"))
seedRestoreObject(fake, sessionPrefix+"artifacts/session_recap.md", []byte("# restored recap\n"))
restoreWithStoreAndRealPhases(t, fake)
origExecuteStagesFn := executeStagesFn
t.Cleanup(func() {
executeStagesFn = origExecuteStagesFn
})
executeStagesFn = func(ctx context.Context, cfg *config.Config, stages []stage.Stage, opts RunOptions) (*RunSummary, error) {
if opts.Env == nil {
opts.Env = &Env{}
}
opts.Env.Scriptorium = &scriptorium.NoopRunner{}
return executeStages(ctx, cfg, stages, opts)
}
var stdout bytes.Buffer
var stderr bytes.Buffer
restoreCode := Execute(
[]string{
"restore",
"--config", pipelinePath,
"--session", sessionPath,
"--session-id", cfg.Session.SessionID,
},
&stdout,
&stderr,
)
if restoreCode != 0 {
t.Fatalf("restore exit code = %d, want 0; stderr=%q", restoreCode, stderr.String())
}
if stderr.Len() != 0 {
t.Fatalf("restore stderr = %q, want empty", stderr.String())
}
sessionRoot := artifacts.SessionWorkDirForCampaign(workspaceRoot, cfg.Session.Campaign, cfg.Session.SessionID)
mustReadEquals(t, filepath.Join(sessionRoot, "transcripts", "full.json"), `{"segments":[1,2,3]}`+"\n")
mustReadEquals(t, filepath.Join(sessionRoot, "artifacts", "session_recap.md"), "# restored recap\n")
manifestStore := &manifest.LocalStore{}
sessionManifestPath := artifacts.SessionManifestPathForCampaign(workspaceRoot, cfg.Session.Campaign, cfg.Session.SessionID)
beforeAnalyze, err := manifestStore.Load(context.Background(), sessionManifestPath)
if err != nil {
t.Fatalf("load restored session manifest: %v", err)
}
upstreamCompletedAt := map[string]time.Time{}
for _, stageName := range []string{"prepare", "transcribe", "merge", "polish", "normalize", "trim"} {
rec := beforeAnalyze.Stages[stageName]
if rec == nil || rec.Status != manifest.StatusSucceeded || rec.CompletedAt == nil {
t.Fatalf("restored manifest stage %q = %#v, want succeeded with completion timestamp", stageName, rec)
}
upstreamCompletedAt[stageName] = *rec.CompletedAt
}
stdout.Reset()
stderr.Reset()
runStageCode := Execute(
[]string{
"run-stage",
"--config", pipelinePath,
"--session", sessionPath,
"--session-id", cfg.Session.SessionID,
"--force",
"--artifacts", "player_handout",
"analyze",
},
&stdout,
&stderr,
)
if runStageCode != 0 {
t.Fatalf("run-stage exit code = %d, want 0; stderr=%q", runStageCode, stderr.String())
}
if stderr.Len() != 0 {
t.Fatalf("run-stage stderr = %q, want empty", stderr.String())
}
if !strings.Contains(stdout.String(), "stage=analyze executed=1 skipped=0 force=true") {
t.Fatalf("run-stage stdout = %q, want analyze execution summary", stdout.String())
}
playerHandoutPath := filepath.Join(sessionRoot, "artifacts", "player_handout.md")
if _, err := os.Stat(playerHandoutPath); err != nil {
t.Fatalf("restored analyze output %q missing: %v", playerHandoutPath, err)
}
afterAnalyze, err := manifestStore.Load(context.Background(), sessionManifestPath)
if err != nil {
t.Fatalf("load session manifest after run-stage analyze: %v", err)
}
for _, stageName := range []string{"prepare", "transcribe", "merge", "polish", "normalize", "trim"} {
rec := afterAnalyze.Stages[stageName]
if rec == nil || rec.Status != manifest.StatusSucceeded || rec.CompletedAt == nil {
t.Fatalf("post-analyze manifest stage %q = %#v, want succeeded with completion timestamp", stageName, rec)
}
if !rec.CompletedAt.Equal(upstreamCompletedAt[stageName]) {
t.Fatalf(
"stage %q completion changed: before=%s after=%s",
stageName,
upstreamCompletedAt[stageName].Format(time.RFC3339Nano),
rec.CompletedAt.Format(time.RFC3339Nano),
)
}
}
analyzeRec := afterAnalyze.Stages["analyze"]
if analyzeRec == nil || analyzeRec.Status != manifest.StatusSucceeded {
t.Fatalf("post-analyze stage record = %#v, want succeeded", analyzeRec)
}
runManifestPaths, err := filepath.Glob(filepath.Join(sessionRoot, "runs", "*", "manifest.json"))
if err != nil {
t.Fatalf("glob run manifests: %v", err)
}
if len(runManifestPaths) != 1 {
t.Fatalf("run manifest count = %d, want 1; paths=%v", len(runManifestPaths), runManifestPaths)
}
runManifest, err := manifestStore.LoadRun(context.Background(), runManifestPaths[0])
if err != nil {
t.Fatalf("load run manifest %q: %v", runManifestPaths[0], err)
}
if len(runManifest.RequestedStages) != 1 || runManifest.RequestedStages[0] != "analyze" {
t.Fatalf("run manifest requested_stages = %#v, want [analyze]", runManifest.RequestedStages)
}
if runManifest.Stages["analyze"] == nil || runManifest.Stages["analyze"].Status != manifest.StatusSucceeded {
t.Fatalf("run manifest analyze stage = %#v, want succeeded", runManifest.Stages["analyze"])
}
if runManifest.Stages["prepare"] != nil {
t.Fatalf("run manifest should not include upstream prepare stage, got %#v", runManifest.Stages["prepare"])
}
}
func restoreWorkflowManifestJSON(t *testing.T, sessionID, campaign string) []byte {
t.Helper()
store := &manifest.LocalStore{}
now := time.Date(2026, 5, 19, 23, 0, 0, 0, time.UTC)
m := manifest.New(sessionID, now)
m.Campaign = campaign
m.RunID = "20260519T010203Z-a1b2c3d4"
stages := []string{"prepare", "transcribe", "merge", "polish", "normalize", "trim"}
for i, stageName := range stages {
m.MarkStageSucceeded(stageName, now.Add(time.Duration(i+1)*time.Minute), nil)
}
path := filepath.Join(t.TempDir(), "manifest.json")
if err := store.Save(context.Background(), path, m); err != nil {
t.Fatalf("save workflow manifest fixture: %v", err)
}
data, err := os.ReadFile(path)
if err != nil {
t.Fatalf("read workflow manifest fixture: %v", err)
}
return data
}

View File

@@ -6,6 +6,7 @@ import (
"fmt"
"io"
"gitea.maximumdirect.net/eric/narratio/internal/artifacts"
"gitea.maximumdirect.net/eric/narratio/internal/config"
"gitea.maximumdirect.net/eric/narratio/internal/manifest"
)
@@ -19,10 +20,12 @@ func Resume(ctx context.Context, args []string, out io.Writer) error {
var sessionPath string
var sessionID string
var force bool
var selectedArtifacts artifactSelectionFlag
fs.StringVar(&pipelinePath, "config", "", "path to pipeline.yml (optional; defaults searched)")
fs.StringVar(&sessionPath, "session", "", "path to session.yml")
fs.StringVar(&sessionID, "session-id", "", "session identifier for session.yml templates")
fs.BoolVar(&force, "force", false, "force stage execution")
fs.Var(&selectedArtifacts, "artifacts", "artifact names to execute during analyze (comma-separated or repeatable)")
if err := fs.Parse(args); err != nil {
return fmt.Errorf("resume: invalid flags: %w", err)
@@ -48,6 +51,13 @@ func Resume(ctx context.Context, args []string, out io.Writer) error {
if err := config.Validate(cfg); err != nil {
return fmt.Errorf("resume: %w", err)
}
normalizedArtifacts, err := selectedArtifacts.Normalize()
if err != nil {
return fmt.Errorf("resume: invalid --artifacts: %w", err)
}
if err := validateSelectedAnalyzeArtifacts(cfg, normalizedArtifacts); err != nil {
return fmt.Errorf("resume: %w", err)
}
full := BuildFullPlan()
selected := full
@@ -66,7 +76,10 @@ func Resume(ctx context.Context, args []string, out io.Writer) error {
}
}
summary, err := executeStages(ctx, cfg, selected, RunOptions{Force: force})
summary, err := executeStagesFn(ctx, cfg, selected, RunOptions{
Force: force,
SelectedArtifacts: normalizedArtifacts,
})
if err != nil {
return fmt.Errorf("resume: %w", err)
}
@@ -83,7 +96,11 @@ func Resume(ctx context.Context, args []string, out io.Writer) error {
}
func loadManifestIfPresent(ctx context.Context, cfg *config.Config) (*manifest.Manifest, error) {
path := manifestPathFor(cfg)
path := artifacts.SessionManifestPathForCampaign(
cfg.Pipeline.Workspace.Root,
cfg.Session.Campaign,
cfg.Session.SessionID,
)
exists, err := fileExists(path)
if err != nil {
return nil, fmt.Errorf("check manifest %q: %w", path, err)

View File

@@ -16,7 +16,7 @@ import (
func TestResumeStartsAfterCompletedStages(t *testing.T) {
workspaceRoot := t.TempDir()
pipelinePath, sessionPath := writeValidConfigFiles(t, workspaceRoot)
manifestPath := filepath.Join(workspaceRoot, "work", "2026-05-03", "manifest.json")
manifestPath := filepath.Join(workspaceRoot, "work", "sample-campaign", "2026-05-03", "manifest.json")
store := &manifest.LocalStore{}
m := manifest.New("2026-05-03", time.Date(2026, 5, 3, 10, 0, 0, 0, time.UTC))
@@ -25,7 +25,7 @@ func TestResumeStartsAfterCompletedStages(t *testing.T) {
if err := store.Save(context.Background(), manifestPath, m); err != nil {
t.Fatalf("save manifest: %v", err)
}
workRoot := filepath.Join(workspaceRoot, "work", "2026-05-03")
workRoot := filepath.Join(workspaceRoot, "work", "sample-campaign", "2026-05-03")
mustWriteTestFile(t, filepath.Join(workRoot, "transcripts", "raw", "alice.json"), `{"segments":[]}`)
mustWriteTestFile(t, filepath.Join(workRoot, "inputs", "speakers.yml"), "match:\n - speaker: Alice\n match: [\"alice\"]\n")
mustWriteTestFile(t, filepath.Join(workRoot, "inputs", "autocorrect.yml"), "[]\n")
@@ -52,7 +52,7 @@ func TestResumeStartsAfterCompletedStages(t *testing.T) {
func TestResumeNoRemainingStages(t *testing.T) {
workspaceRoot := t.TempDir()
pipelinePath, sessionPath := writeValidConfigFiles(t, workspaceRoot)
manifestPath := filepath.Join(workspaceRoot, "work", "2026-05-03", "manifest.json")
manifestPath := filepath.Join(workspaceRoot, "work", "sample-campaign", "2026-05-03", "manifest.json")
store := &manifest.LocalStore{}
m := manifest.New("2026-05-03", time.Date(2026, 5, 3, 10, 0, 0, 0, time.UTC))
@@ -81,7 +81,7 @@ func TestResumeForceRerunsSucceeded(t *testing.T) {
}))
defer srv.Close()
pipelinePath, sessionPath := writeValidConfigFiles(t, workspaceRoot, srv.URL)
manifestPath := filepath.Join(workspaceRoot, "work", "2026-05-03", "manifest.json")
manifestPath := filepath.Join(workspaceRoot, "work", "sample-campaign", "2026-05-03", "manifest.json")
store := &manifest.LocalStore{}
m := manifest.New("2026-05-03", time.Date(2026, 5, 3, 10, 0, 0, 0, time.UTC))
@@ -105,8 +105,8 @@ func TestResumeForceRerunsSucceeded(t *testing.T) {
func TestRunStageExecutesOnlySelectedStage(t *testing.T) {
workspaceRoot := t.TempDir()
pipelinePath, sessionPath := writeValidConfigFiles(t, workspaceRoot)
manifestPath := filepath.Join(workspaceRoot, "work", "2026-05-03", "manifest.json")
workRoot := filepath.Join(workspaceRoot, "work", "2026-05-03")
manifestPath := filepath.Join(workspaceRoot, "work", "sample-campaign", "2026-05-03", "manifest.json")
workRoot := filepath.Join(workspaceRoot, "work", "sample-campaign", "2026-05-03")
mustWriteTestFile(t, filepath.Join(workRoot, "transcripts", "merged.json"), `{"segments":[]}`)
mustWriteTestFile(t, filepath.Join(workRoot, "inputs", "glossary.yml"), "terms: []\n")
@@ -135,8 +135,8 @@ func TestRunStageExecutesOnlySelectedStage(t *testing.T) {
func TestRunStageSkipAndForce(t *testing.T) {
workspaceRoot := t.TempDir()
pipelinePath, sessionPath := writeValidConfigFiles(t, workspaceRoot)
manifestPath := filepath.Join(workspaceRoot, "work", "2026-05-03", "manifest.json")
workRoot := filepath.Join(workspaceRoot, "work", "2026-05-03")
manifestPath := filepath.Join(workspaceRoot, "work", "sample-campaign", "2026-05-03", "manifest.json")
workRoot := filepath.Join(workspaceRoot, "work", "sample-campaign", "2026-05-03")
mustWriteTestFile(t, filepath.Join(workRoot, "transcripts", "merged.json"), `{"segments":[]}`)
mustWriteTestFile(t, filepath.Join(workRoot, "inputs", "glossary.yml"), "terms: []\n")
@@ -166,11 +166,57 @@ func TestRunStageSkipAndForce(t *testing.T) {
}
}
func TestRunStageForceMarksDownstreamStaleAndResumeContinuesFromStale(t *testing.T) {
workspaceRoot := t.TempDir()
pipelinePath, sessionPath := writeValidConfigFiles(t, workspaceRoot)
manifestPath := filepath.Join(workspaceRoot, "work", "sample-campaign", "2026-05-03", "manifest.json")
workRoot := filepath.Join(workspaceRoot, "work", "sample-campaign", "2026-05-03")
mustWriteTestFile(t, filepath.Join(workRoot, "transcripts", "merged.json"), `{"segments":[]}`)
mustWriteTestFile(t, filepath.Join(workRoot, "inputs", "glossary.yml"), "terms: []\n")
store := &manifest.LocalStore{}
seed := manifest.New("2026-05-03", time.Date(2026, 5, 3, 10, 0, 0, 0, time.UTC))
for _, name := range []string{"prepare", "transcribe", "merge", "polish", "normalize", "trim", "analyze", "archive", "notify"} {
seed.MarkStageSucceeded(name, time.Date(2026, 5, 3, 10, 1, 0, 0, time.UTC), nil)
}
if err := store.Save(context.Background(), manifestPath, seed); err != nil {
t.Fatalf("save manifest: %v", err)
}
var out bytes.Buffer
err := RunStage(context.Background(), []string{"--config", pipelinePath, "--session", sessionPath, "--force", "polish"}, &out)
if err != nil {
t.Fatalf("RunStage(force) error = %v", err)
}
if !strings.Contains(out.String(), "stage=polish executed=1 skipped=0 force=true") {
t.Fatalf("output = %q, want forced polish rerun", out.String())
}
afterForce, err := store.Load(context.Background(), manifestPath)
if err != nil {
t.Fatalf("load manifest after force: %v", err)
}
for _, name := range []string{"normalize", "trim", "analyze", "archive", "notify"} {
if afterForce.Stages[name] == nil || afterForce.Stages[name].Status != manifest.StatusStale {
t.Fatalf("stage %q = %#v, want stale", name, afterForce.Stages[name])
}
}
out.Reset()
err = Resume(context.Background(), []string{"--config", pipelinePath, "--session", sessionPath}, &out)
if err != nil {
t.Fatalf("Resume() error = %v", err)
}
if !strings.Contains(out.String(), "executed=5 skipped=0") {
t.Fatalf("output = %q, want resume to execute normalize..notify", out.String())
}
}
func TestRunStageTrimExecutes(t *testing.T) {
workspaceRoot := t.TempDir()
pipelinePath, sessionPath := writeValidConfigFiles(t, workspaceRoot)
manifestPath := filepath.Join(workspaceRoot, "work", "2026-05-03", "manifest.json")
workRoot := filepath.Join(workspaceRoot, "work", "2026-05-03")
manifestPath := filepath.Join(workspaceRoot, "work", "sample-campaign", "2026-05-03", "manifest.json")
workRoot := filepath.Join(workspaceRoot, "work", "sample-campaign", "2026-05-03")
mustWriteTestFile(t, filepath.Join(workRoot, "transcripts", "normalized.json"), `{"segments":[{"id":1},{"id":2}]}`)
var out bytes.Buffer
@@ -198,8 +244,8 @@ func TestRunStageTrimExecutes(t *testing.T) {
func TestRunStageNormalizeExecutes(t *testing.T) {
workspaceRoot := t.TempDir()
pipelinePath, sessionPath := writeValidConfigFiles(t, workspaceRoot)
manifestPath := filepath.Join(workspaceRoot, "work", "2026-05-03", "manifest.json")
workRoot := filepath.Join(workspaceRoot, "work", "2026-05-03")
manifestPath := filepath.Join(workspaceRoot, "work", "sample-campaign", "2026-05-03", "manifest.json")
workRoot := filepath.Join(workspaceRoot, "work", "sample-campaign", "2026-05-03")
mustWriteTestFile(t, filepath.Join(workRoot, "transcripts", "processed.json"), `{"segments":[{"id":1},{"id":2}]}`)
var out bytes.Buffer

View File

@@ -18,10 +18,12 @@ func Run(ctx context.Context, args []string, out io.Writer) error {
var sessionPath string
var sessionID string
var force bool
var selectedArtifacts artifactSelectionFlag
fs.StringVar(&pipelinePath, "config", "", "path to pipeline.yml (optional; defaults searched)")
fs.StringVar(&sessionPath, "session", "", "path to session.yml")
fs.StringVar(&sessionID, "session-id", "", "session identifier for session.yml templates")
fs.BoolVar(&force, "force", false, "force stage execution (reserved for future behavior)")
fs.Var(&selectedArtifacts, "artifacts", "artifact names to execute during analyze (comma-separated or repeatable)")
if err := fs.Parse(args); err != nil {
return fmt.Errorf("run: invalid flags: %w", err)
@@ -47,9 +49,19 @@ func Run(ctx context.Context, args []string, out io.Writer) error {
if err := config.Validate(cfg); err != nil {
return fmt.Errorf("run: %w", err)
}
normalizedArtifacts, err := selectedArtifacts.Normalize()
if err != nil {
return fmt.Errorf("run: invalid --artifacts: %w", err)
}
if err := validateSelectedAnalyzeArtifacts(cfg, normalizedArtifacts); err != nil {
return fmt.Errorf("run: %w", err)
}
stages := BuildFullPlan()
summary, err := executeStages(ctx, cfg, stages, RunOptions{Force: force})
summary, err := executeStagesFn(ctx, cfg, stages, RunOptions{
Force: force,
SelectedArtifacts: normalizedArtifacts,
})
if err != nil {
return fmt.Errorf("run: %w", err)
}

View File

@@ -1,6 +1,8 @@
package app
import (
"time"
"gitea.maximumdirect.net/eric/narratio/internal/manifest"
"gitea.maximumdirect.net/eric/narratio/internal/stage"
)
@@ -49,3 +51,43 @@ func firstNonSucceededIndex(stages []stage.Stage, m *manifest.Manifest) int {
}
return len(stages)
}
func canonicalStageNames() []string {
all := stage.All()
out := make([]string, 0, len(all))
for _, s := range all {
if s == nil {
continue
}
out = append(out, s.Name())
}
return out
}
func downstreamStageNames(stageName string) []string {
names := canonicalStageNames()
for i, name := range names {
if name != stageName {
continue
}
return append([]string(nil), names[i+1:]...)
}
return nil
}
func invalidateDownstreamSucceededStages(m *manifest.Manifest, upstreamStage string, at time.Time) []string {
if m == nil || m.Stages == nil {
return nil
}
invalidated := make([]string, 0)
for _, downstream := range downstreamStageNames(upstreamStage) {
sr := m.Stages[downstream]
if sr == nil || sr.Status != manifest.StatusSucceeded {
continue
}
m.MarkStageStale(downstream, at, "upstream stage rerun with force")
invalidated = append(invalidated, downstream)
}
return invalidated
}

View File

@@ -1,6 +1,7 @@
package app
import (
"reflect"
"testing"
"time"
@@ -40,3 +41,48 @@ func TestDecideStageActions(t *testing.T) {
t.Fatalf("forced prepare action = %q, want %q", forced[0].Action, stageActionRun)
}
}
func TestDownstreamStageNames(t *testing.T) {
got := downstreamStageNames("polish")
want := []string{"normalize", "trim", "analyze", "archive", "notify"}
if !reflect.DeepEqual(got, want) {
t.Fatalf("downstreamStageNames(polish) = %#v, want %#v", got, want)
}
missing := downstreamStageNames("unknown")
if len(missing) != 0 {
t.Fatalf("downstreamStageNames(unknown) = %#v, want empty", missing)
}
}
func TestInvalidateDownstreamSucceededStages(t *testing.T) {
now := time.Now().UTC()
m := manifest.New("2026-05-03", now)
m.MarkStageSucceeded("prepare", now, nil)
m.MarkStageSucceeded("transcribe", now, nil)
m.MarkStageSucceeded("merge", now, nil)
m.MarkStageSucceeded("polish", now, nil)
m.MarkStageSucceeded("normalize", now, nil)
m.MarkStageSucceeded("trim", now, nil)
m.MarkStageFailed("analyze", now, "analysis failed")
m.MarkStageSucceeded("archive", now, nil)
m.MarkStageSucceeded("notify", now, nil)
got := invalidateDownstreamSucceededStages(m, "polish", now.Add(1*time.Second))
want := []string{"normalize", "trim", "archive", "notify"}
if !reflect.DeepEqual(got, want) {
t.Fatalf("invalidateDownstreamSucceededStages() = %#v, want %#v", got, want)
}
for _, stageName := range want {
if m.Stages[stageName].Status != manifest.StatusStale {
t.Fatalf("%s status = %q, want stale", stageName, m.Stages[stageName].Status)
}
}
if m.Stages["analyze"].Status != manifest.StatusFailed {
t.Fatalf("analyze status = %q, want failed", m.Stages["analyze"].Status)
}
if m.Stages["prepare"].Status != manifest.StatusSucceeded {
t.Fatalf("prepare status = %q, want succeeded", m.Stages["prepare"].Status)
}
}

View File

@@ -18,10 +18,12 @@ func RunStage(ctx context.Context, args []string, out io.Writer) error {
var sessionPath string
var sessionID string
var force bool
var selectedArtifacts artifactSelectionFlag
fs.StringVar(&pipelinePath, "config", "", "path to pipeline.yml (optional; defaults searched)")
fs.StringVar(&sessionPath, "session", "", "path to session.yml")
fs.StringVar(&sessionID, "session-id", "", "session identifier for session.yml templates")
fs.BoolVar(&force, "force", false, "force stage execution (reserved for future behavior)")
fs.Var(&selectedArtifacts, "artifacts", "artifact names to execute during analyze (comma-separated or repeatable)")
if err := fs.Parse(args); err != nil {
return fmt.Errorf("run-stage: invalid flags: %w", err)
@@ -30,6 +32,13 @@ func RunStage(ctx context.Context, args []string, out io.Writer) error {
return fmt.Errorf("run-stage: expected exactly one stage name")
}
stageName := fs.Arg(0)
normalizedArtifacts, err := selectedArtifacts.Normalize()
if err != nil {
return fmt.Errorf("run-stage: invalid --artifacts: %w", err)
}
if len(normalizedArtifacts) > 0 && stageName != "analyze" {
return fmt.Errorf("run-stage: --artifacts is only supported for stage \"analyze\"")
}
stages, err := BuildSingleStagePlan(stageName)
if err != nil {
return fmt.Errorf("run-stage: %w", err)
@@ -53,8 +62,14 @@ func RunStage(ctx context.Context, args []string, out io.Writer) error {
if err := config.Validate(cfg); err != nil {
return fmt.Errorf("run-stage: %w", err)
}
if err := validateSelectedAnalyzeArtifacts(cfg, normalizedArtifacts); err != nil {
return fmt.Errorf("run-stage: %w", err)
}
summary, err := executeStages(ctx, cfg, stages, RunOptions{Force: force})
summary, err := executeStagesFn(ctx, cfg, stages, RunOptions{
Force: force,
SelectedArtifacts: normalizedArtifacts,
})
if err != nil {
return fmt.Errorf("run-stage: %w", err)
}

View File

@@ -5,7 +5,6 @@ import (
"fmt"
"log/slog"
"os"
"path/filepath"
"strings"
"gitea.maximumdirect.net/eric/narratio/internal/adapters/audita"
@@ -22,18 +21,23 @@ import (
)
type RunOptions struct {
Force bool
Env *Env
Force bool
SelectedArtifacts []string
Env *Env
}
type RunSummary struct {
SessionID string
ManifestPath string
StageNames []string
Executed []string
Skipped []string
SessionID string
RunID string
ManifestPath string
RunManifestPath string
StageNames []string
Executed []string
Skipped []string
}
var executeStagesFn = executeStages
func executeStages(ctx context.Context, cfg *config.Config, stages []stage.Stage, opts RunOptions) (*RunSummary, error) {
env := opts.Env
if env == nil {
@@ -42,6 +46,7 @@ func executeStages(ctx context.Context, cfg *config.Config, stages []stage.Stage
if env.Config == nil {
env.Config = cfg
}
env.SelectedAnalyzeArtifacts = append([]string(nil), opts.SelectedArtifacts...)
if env.ArtifactStore == nil {
env.ArtifactStore = artifacts.NewLocalStore(cfg.Pipeline.Workspace.Root)
}
@@ -93,12 +98,12 @@ func executeStages(ctx context.Context, cfg *config.Config, stages []stage.Stage
}
artifactStore := env.ArtifactStore
paths, err := artifactStore.EnsureLayout(cfg.Session.SessionID)
paths, err := artifactStore.EnsureLayoutFor(cfg.Session.Campaign, cfg.Session.SessionID)
if err != nil {
return nil, fmt.Errorf("prepare workdir: %w", err)
}
lock, err := artifactStore.AcquireSessionLock(cfg.Session.SessionID)
lock, err := artifactStore.AcquireSessionLockFor(cfg.Session.Campaign, cfg.Session.SessionID)
if err != nil {
return nil, fmt.Errorf("acquire session lock: %w", err)
}
@@ -111,7 +116,11 @@ func executeStages(ctx context.Context, cfg *config.Config, stages []stage.Stage
if err != nil {
return nil, err
}
identityChanged, err := ensureManifestIdentity(cfg, m)
runID, err := artifacts.NewRunID()
if err != nil {
return nil, fmt.Errorf("generate run id: %w", err)
}
identityChanged, err := ensureManifestIdentity(cfg, m, runID)
if err != nil {
return nil, fmt.Errorf("initialize manifest identity: %w", err)
}
@@ -120,6 +129,29 @@ func executeStages(ctx context.Context, cfg *config.Config, stages []stage.Stage
return nil, fmt.Errorf("save manifest identity %q: %w", manifestPath, err)
}
}
runManifestPath := artifacts.SessionRunManifestPathForCampaign(
cfg.Pipeline.Workspace.Root,
cfg.Session.Campaign,
cfg.Session.SessionID,
runID,
)
runManifestStore := &manifest.LocalStore{}
runManifest, err := runManifestStore.CreateRun(
ctx,
cfg.Session.SessionID,
cfg.Session.Campaign,
runID,
opts.Force,
requestedStageNames(stages),
)
if err != nil {
return nil, fmt.Errorf("create run manifest: %w", err)
}
runManifest.SessionManifestPath = manifestPath
syncRunManifestIdentityFromSession(m, runManifest)
if err := runManifestStore.SaveRun(ctx, runManifestPath, runManifest); err != nil {
return nil, fmt.Errorf("save initial run manifest %q: %w", runManifestPath, err)
}
stageEnv := env
@@ -134,12 +166,23 @@ func executeStages(ctx context.Context, cfg *config.Config, stages []stage.Stage
if d.Action == stageActionSkip {
skipped = append(skipped, s.Name())
skipAt := nowUTC()
runManifest.SetStageAction(s.Name(), manifest.RunStageActionSkip, skipAt)
runManifest.MarkStageSkipped(s.Name(), skipAt, "already_succeeded")
if err := runManifestStore.SaveRun(ctx, runManifestPath, runManifest); err != nil {
return nil, fmt.Errorf("save run manifest after skip %q: %w", s.Name(), err)
}
env.Logger.Info("skipping stage", "stage", s.Name(), "reason", "already_succeeded", "force", opts.Force)
continue
}
executed = append(executed, s.Name())
now := nowUTC()
runManifest.SetStageAction(s.Name(), manifest.RunStageActionRun, now)
runManifest.MarkStageRunning(s.Name(), now)
if err := runManifestStore.SaveRun(ctx, runManifestPath, runManifest); err != nil {
return nil, fmt.Errorf("save run manifest before stage %q: %w", s.Name(), err)
}
m.MarkStageRunning(s.Name(), now)
env.Logger.Info("starting stage", "stage", s.Name())
if err := env.ManifestStore.Save(ctx, manifestPath, m); err != nil {
@@ -149,35 +192,65 @@ func executeStages(ctx context.Context, cfg *config.Config, stages []stage.Stage
result, err := s.Run(ctx, stageEnv, m)
if err != nil {
m.MarkStageFailed(s.Name(), nowUTC(), err.Error())
failedAt := nowUTC()
m.MarkStageFailed(s.Name(), failedAt, err.Error())
if saveErr := env.ManifestStore.Save(ctx, manifestPath, m); saveErr != nil {
return nil, fmt.Errorf("stage %q failed (%v) and manifest save failed (%v)", s.Name(), err, saveErr)
}
runManifest.MarkStageFailed(s.Name(), failedAt, err.Error())
syncRunManifestIdentityFromSession(m, runManifest)
if saveErr := runManifestStore.SaveRun(ctx, runManifestPath, runManifest); saveErr != nil {
return nil, fmt.Errorf("stage %q failed (%v) and run-manifest save failed (%v)", s.Name(), err, saveErr)
}
env.Logger.Info("stage failed", "stage", s.Name(), "error", err)
return nil, fmt.Errorf("stage %q failed: %w", s.Name(), err)
}
outputs := mapResultOutputs(result)
m.MarkStageSucceeded(s.Name(), nowUTC(), outputs)
outputs := mapResultOutputs(s.Name(), result, runID)
succeededAt := nowUTC()
m.MarkStageSucceeded(s.Name(), succeededAt, outputs)
applyStageResultToManifest(m, s.Name(), result)
if opts.Force {
invalidateDownstreamSucceededStages(m, s.Name(), succeededAt)
}
if err := env.ManifestStore.Save(ctx, manifestPath, m); err != nil {
return nil, fmt.Errorf("save manifest after stage %q: %w", s.Name(), err)
}
runManifest.MarkStageSucceeded(s.Name(), succeededAt, outputs)
applyStageResultToRunManifest(runManifest, s.Name(), result)
syncRunManifestIdentityFromSession(m, runManifest)
if err := runManifestStore.SaveRun(ctx, runManifestPath, runManifest); err != nil {
return nil, fmt.Errorf("save run manifest after stage %q: %w", s.Name(), err)
}
env.Logger.Debug("manifest saved", "stage", s.Name(), "transition", "succeeded", "path", manifestPath)
env.Logger.Info("stage succeeded", "stage", s.Name())
}
if err := runPostArchiveCleanup(ctx, env, manifestPath, m, executed); err != nil {
failedAt := nowUTC()
runManifest.MarkFailed(failedAt, err.Error())
syncRunManifestIdentityFromSession(m, runManifest)
if saveErr := runManifestStore.SaveRun(ctx, runManifestPath, runManifest); saveErr != nil {
return nil, fmt.Errorf("post-archive cleanup failed (%v) and run-manifest save failed (%v)", err, saveErr)
}
return nil, fmt.Errorf("post-archive cleanup: %w", err)
}
completedAt := nowUTC()
runManifest.MarkSucceeded(completedAt)
syncRunManifestIdentityFromSession(m, runManifest)
if err := runManifestStore.SaveRun(ctx, runManifestPath, runManifest); err != nil {
return nil, fmt.Errorf("save final run manifest %q: %w", runManifestPath, err)
}
return &RunSummary{
SessionID: cfg.Session.SessionID,
ManifestPath: manifestPath,
StageNames: runNames,
Executed: executed,
Skipped: skipped,
SessionID: cfg.Session.SessionID,
RunID: runID,
ManifestPath: manifestPath,
RunManifestPath: runManifestPath,
StageNames: runNames,
Executed: executed,
Skipped: skipped,
}, nil
}
@@ -317,22 +390,31 @@ func fileExists(path string) (bool, error) {
return false, err
}
func mapResultOutputs(result *stage.StageResult) []manifest.ArtifactRecord {
func mapResultOutputs(stageName string, result *stage.StageResult, runID string) []manifest.ArtifactRecord {
if result == nil || len(result.Outputs) == 0 {
return nil
}
runID = strings.TrimSpace(runID)
out := make([]manifest.ArtifactRecord, 0, len(result.Outputs))
for _, ref := range result.Outputs {
localPath := ref.AbsolutePath
if localPath == "" {
localPath = ref.RelativePath
}
kind := ref.Kind
sourceID := ""
if stageName == "analyze" {
sourceID = artifacts.ConfiguredArtifactSourceID(ref.Kind)
kind = "scriptorium_artifact"
}
out = append(out, manifest.ArtifactRecord{
Kind: ref.Kind,
LocalPath: localPath,
RemoteKey: ref.RemoteKey,
Checksum: ref.Checksum,
Kind: kind,
SourceID: sourceID,
LocalPath: localPath,
ProducerRunID: runID,
RemoteKey: ref.RemoteKey,
Checksum: ref.Checksum,
})
}
@@ -358,7 +440,7 @@ func applyStageResultToManifest(m *manifest.Manifest, stageName string, result *
}
}
func ensureManifestIdentity(cfg *config.Config, m *manifest.Manifest) (bool, error) {
func ensureManifestIdentity(cfg *config.Config, m *manifest.Manifest, runID string) (bool, error) {
if cfg == nil || cfg.Pipeline == nil || cfg.Session == nil || m == nil {
return false, nil
}
@@ -374,16 +456,13 @@ func ensureManifestIdentity(cfg *config.Config, m *manifest.Manifest) (bool, err
m.Campaign = campaign
changed = true
}
if m.RunID == "" {
runID, err := artifacts.NewRunID()
if err != nil {
return false, err
}
runID = strings.TrimSpace(runID)
if runID != "" && m.RunID != runID {
m.RunID = runID
changed = true
}
if m.LocalWorkDir == "" && campaign != "" && sessionID != "" && m.RunID != "" {
m.LocalWorkDir = artifacts.SessionRunWorkDir(cfg.Pipeline.Workspace.Root, campaign, sessionID, m.RunID)
m.LocalWorkDir = artifacts.SessionRunRootForCampaign(cfg.Pipeline.Workspace.Root, campaign, sessionID, m.RunID)
changed = true
}
if m.LocalSpoolDir == "" && campaign != "" && sessionID != "" && m.RunID != "" && strings.TrimSpace(cfg.Pipeline.Spool.Root) != "" {
@@ -410,8 +489,54 @@ func ensureManifestIdentity(cfg *config.Config, m *manifest.Manifest) (bool, err
return changed, nil
}
func requestedStageNames(stages []stage.Stage) []string {
out := make([]string, 0, len(stages))
for _, s := range stages {
if s == nil {
continue
}
out = append(out, s.Name())
}
return out
}
func applyStageResultToRunManifest(m *manifest.RunManifest, stageName string, result *stage.StageResult) {
if m == nil || result == nil {
return
}
sr := m.Stages[stageName]
if sr == nil {
return
}
if len(result.Logs) > 0 {
sr.Logs = append([]string(nil), result.Logs...)
}
if len(result.GeneratedConfigs) > 0 {
sr.GeneratedConfigs = append([]string(nil), result.GeneratedConfigs...)
}
if len(result.Metadata) > 0 {
sr.Metadata = result.Metadata
}
}
func syncRunManifestIdentityFromSession(session *manifest.Manifest, run *manifest.RunManifest) {
if session == nil || run == nil {
return
}
run.Campaign = session.Campaign
run.LocalWorkDir = session.LocalWorkDir
run.LocalSpoolDir = session.LocalSpoolDir
run.S3Bucket = session.S3Bucket
run.S3SessionPrefix = session.S3SessionPrefix
run.S3RunPrefix = session.S3RunPrefix
}
func manifestPathFor(cfg *config.Config) string {
return filepath.Join(cfg.Pipeline.Workspace.Root, "work", cfg.Session.SessionID, "manifest.json")
return artifacts.SessionManifestPathForCampaign(
cfg.Pipeline.Workspace.Root,
cfg.Session.Campaign,
cfg.Session.SessionID,
)
}
func needsObjectStoreForRun(cfg *config.Config, stages []stage.Stage) bool {

View File

@@ -3,6 +3,7 @@ package app
import (
"context"
"errors"
"fmt"
"os"
"path/filepath"
"strings"
@@ -44,6 +45,220 @@ func (s countingStage) Run(_ context.Context, _ *stage.Env, _ *manifest.Manifest
return &stage.StageResult{Metadata: map[string]any{"counting": true}}, nil
}
type captureSelectedArtifactsStage struct {
name string
captured *[]string
}
func (s captureSelectedArtifactsStage) Name() string { return s.name }
func (s captureSelectedArtifactsStage) Declares() stage.IODecl { return stage.IODecl{} }
func (s captureSelectedArtifactsStage) Run(_ context.Context, env *stage.Env, _ *manifest.Manifest) (*stage.StageResult, error) {
if s.captured != nil {
*s.captured = append((*s.captured)[:0], env.SelectedAnalyzeArtifacts...)
}
return &stage.StageResult{Metadata: map[string]any{"captured": true}}, nil
}
type analyzeOutputStage struct {
output artifacts.Ref
}
func (s analyzeOutputStage) Name() string { return "analyze" }
func (s analyzeOutputStage) Declares() stage.IODecl { return stage.IODecl{} }
func (s analyzeOutputStage) Run(_ context.Context, _ *stage.Env, _ *manifest.Manifest) (*stage.StageResult, error) {
return &stage.StageResult{
Outputs: []artifacts.Ref{s.output},
}, nil
}
type selectedAnalyzeArtifactStage struct {
expected []string
}
func (s selectedAnalyzeArtifactStage) Name() string { return "analyze" }
func (s selectedAnalyzeArtifactStage) Declares() stage.IODecl { return stage.IODecl{} }
func (s selectedAnalyzeArtifactStage) Run(_ context.Context, env *stage.Env, m *manifest.Manifest) (*stage.StageResult, error) {
if len(env.SelectedAnalyzeArtifacts) != len(s.expected) {
return nil, fmt.Errorf("selected artifacts len = %d, want %d", len(env.SelectedAnalyzeArtifacts), len(s.expected))
}
for i := range s.expected {
if env.SelectedAnalyzeArtifacts[i] != s.expected[i] {
return nil, fmt.Errorf("selected artifacts[%d] = %q, want %q", i, env.SelectedAnalyzeArtifacts[i], s.expected[i])
}
}
outputPath := filepath.Join(
artifacts.SessionWorkDirForCampaign(env.Config.Pipeline.Workspace.Root, env.Config.Session.Campaign, m.SessionID),
"artifacts",
"player_handout.md",
)
if err := os.MkdirAll(filepath.Dir(outputPath), 0o755); err != nil {
return nil, fmt.Errorf("mkdir artifact dir: %w", err)
}
if err := os.WriteFile(outputPath, []byte("player handout\n"), 0o644); err != nil {
return nil, fmt.Errorf("write player handout: %w", err)
}
return &stage.StageResult{
Outputs: []artifacts.Ref{
{
Kind: "player_handout",
Category: "artifacts",
RelativePath: "artifacts/player_handout.md",
AbsolutePath: outputPath,
},
},
Metadata: map[string]any{
"stage": "analyze",
},
}, nil
}
func TestExecuteStagesPropagatesSelectedArtifactsToEnv(t *testing.T) {
cfg := testConfig(t)
captured := []string{}
stageToRun := captureSelectedArtifactsStage{name: "analyze", captured: &captured}
_, err := executeStages(context.Background(), cfg, []stage.Stage{stageToRun}, RunOptions{
SelectedArtifacts: []string{"player_handout", "session_recap"},
})
if err != nil {
t.Fatalf("executeStages() error = %v", err)
}
if len(captured) != 2 {
t.Fatalf("captured len = %d, want 2 (%v)", len(captured), captured)
}
if captured[0] != "player_handout" || captured[1] != "session_recap" {
t.Fatalf("captured = %v, want [player_handout session_recap]", captured)
}
}
func TestExecuteStagesAnalyzeOutputsPersistAsScriptoriumArtifacts(t *testing.T) {
cfg := testConfig(t)
storeForPaths := artifacts.NewLocalStore(cfg.Pipeline.Workspace.Root)
sessionPaths := storeForPaths.SessionPathsFor(cfg.Session.Campaign, cfg.Session.SessionID)
outputPath := filepath.Join(sessionPaths.ArtifactsDir, "session_recap.md")
stageToRun := analyzeOutputStage{
output: artifacts.Ref{
Kind: "session_recap",
Category: "artifacts",
RelativePath: "artifacts/session_recap.md",
AbsolutePath: outputPath,
},
}
summary, err := executeStages(context.Background(), cfg, []stage.Stage{stageToRun}, RunOptions{})
if err != nil {
t.Fatalf("executeStages() error = %v", err)
}
store := &manifest.LocalStore{}
sessionManifest, err := store.Load(context.Background(), summary.ManifestPath)
if err != nil {
t.Fatalf("load session manifest: %v", err)
}
sessionStage := sessionManifest.Stages["analyze"]
if sessionStage == nil {
t.Fatal("session manifest analyze stage missing")
}
if len(sessionStage.Outputs) != 1 {
t.Fatalf("session analyze outputs len = %d, want 1", len(sessionStage.Outputs))
}
sessionOutput := sessionStage.Outputs[0]
if sessionOutput.Kind != "scriptorium_artifact" {
t.Fatalf("session output kind = %q, want scriptorium_artifact", sessionOutput.Kind)
}
if sessionOutput.SourceID != "narratio.artifact.session_recap" {
t.Fatalf("session output source_id = %q, want narratio.artifact.session_recap", sessionOutput.SourceID)
}
if sessionOutput.LocalPath != outputPath {
t.Fatalf("session output local_path = %q, want %q", sessionOutput.LocalPath, outputPath)
}
runManifest, err := store.LoadRun(context.Background(), summary.RunManifestPath)
if err != nil {
t.Fatalf("load run manifest: %v", err)
}
runStage := runManifest.Stages["analyze"]
if runStage == nil {
t.Fatal("run manifest analyze stage missing")
}
if len(runStage.Outputs) != 1 {
t.Fatalf("run analyze outputs len = %d, want 1", len(runStage.Outputs))
}
runOutput := runStage.Outputs[0]
if runOutput.Kind != "scriptorium_artifact" {
t.Fatalf("run output kind = %q, want scriptorium_artifact", runOutput.Kind)
}
if runOutput.SourceID != "narratio.artifact.session_recap" {
t.Fatalf("run output source_id = %q, want narratio.artifact.session_recap", runOutput.SourceID)
}
if runOutput.LocalPath != outputPath {
t.Fatalf("run output local_path = %q, want %q", runOutput.LocalPath, outputPath)
}
}
func TestExecuteStagesArchiveFailsWhenRequiredRecapPromotionMissingForSelectedArtifacts(t *testing.T) {
cfg := testConfig(t)
cfg.Pipeline.Storage.S3 = &config.StorageS3Config{
Bucket: "my-dnd-archive",
RootPrefix: "dnd",
}
cfg.Pipeline.Archive = &config.ArchiveConfig{
Enabled: boolPtr(true),
UploadRun: boolPtr(true),
PromoteArtifacts: []config.ArchivePromotionRule{
{Source: "narratio.artifact.session_recap", Dest: "artifacts/session_recap.md", Required: boolPtr(true)},
},
}
cfg.Pipeline.Scriptorium = &config.ScriptoriumConfig{
Artifacts: map[string]config.ScriptoriumArtifactConfig{
"session_recap": {
OutputPath: "artifacts/session_recap.md",
},
},
}
store := &manifest.LocalStore{}
manifestPath := manifestPathFor(cfg)
seed := manifest.New(cfg.Session.SessionID, time.Now().UTC())
seed.Campaign = cfg.Session.Campaign
for _, stageName := range []string{"prepare", "transcribe", "merge", "polish", "normalize", "trim"} {
seed.MarkStageSucceeded(stageName, time.Now().UTC(), nil)
}
if err := os.MkdirAll(filepath.Dir(manifestPath), 0o755); err != nil {
t.Fatalf("MkdirAll() error = %v", err)
}
if err := store.Save(context.Background(), manifestPath, seed); err != nil {
t.Fatalf("Save manifest error = %v", err)
}
archiveStageImpl, err := stage.Select("archive")
if err != nil {
t.Fatalf("Select(archive) error = %v", err)
}
_, err = executeStages(
context.Background(),
cfg,
[]stage.Stage{
selectedAnalyzeArtifactStage{expected: []string{"player_handout"}},
archiveStageImpl,
},
RunOptions{
SelectedArtifacts: []string{"player_handout"},
Env: &Env{ObjectStore: &storage.FakeBackend{}},
},
)
if err == nil {
t.Fatal("expected archive promotion failure, got nil")
}
if !strings.Contains(err.Error(), "required promotion source unavailable") {
t.Fatalf("error = %q, want required promotion source unavailable", err.Error())
}
}
func TestExecuteStagesPlaceholderSuccessUpdatesManifest(t *testing.T) {
cfg := testConfig(t)
@@ -235,6 +450,56 @@ func TestExecuteStagesForceRerunsSucceeded(t *testing.T) {
}
}
func TestExecuteStagesForceSuccessInvalidatesDownstreamSucceededStages(t *testing.T) {
cfg := testConfig(t)
manifestPath := manifestPathFor(cfg)
store := &manifest.LocalStore{}
existing := manifest.New(cfg.Session.SessionID, time.Date(2026, 5, 3, 1, 0, 0, 0, time.UTC))
for _, stageName := range []string{"prepare", "transcribe", "merge", "polish", "normalize", "trim", "archive", "notify"} {
existing.MarkStageSucceeded(stageName, time.Date(2026, 5, 3, 1, 1, 0, 0, time.UTC), nil)
}
existing.MarkStageFailed("analyze", time.Date(2026, 5, 3, 1, 1, 0, 0, time.UTC), "previous analyze failure")
if err := os.MkdirAll(filepath.Dir(manifestPath), 0o755); err != nil {
t.Fatalf("MkdirAll() error = %v", err)
}
if err := store.Save(context.Background(), manifestPath, existing); err != nil {
t.Fatalf("Save manifest error = %v", err)
}
runs := 0
stageToRun := countingStage{name: "polish", runs: &runs}
summary, err := executeStages(context.Background(), cfg, []stage.Stage{stageToRun}, RunOptions{Force: true})
if err != nil {
t.Fatalf("executeStages() error = %v", err)
}
if runs != 1 {
t.Fatalf("runs = %d, want 1 with force", runs)
}
if len(summary.Executed) != 1 || summary.Executed[0] != "polish" || len(summary.Skipped) != 0 {
t.Fatalf("summary = %#v, want executed polish", summary)
}
loaded, err := store.Load(context.Background(), manifestPath)
if err != nil {
t.Fatalf("Load manifest error = %v", err)
}
if loaded.Stages["polish"] == nil || loaded.Stages["polish"].Status != manifest.StatusSucceeded {
t.Fatalf("polish status = %#v, want succeeded", loaded.Stages["polish"])
}
for _, stageName := range []string{"normalize", "trim", "archive", "notify"} {
if loaded.Stages[stageName] == nil || loaded.Stages[stageName].Status != manifest.StatusStale {
t.Fatalf("%s status = %#v, want stale", stageName, loaded.Stages[stageName])
}
}
if loaded.Stages["analyze"] == nil || loaded.Stages["analyze"].Status != manifest.StatusFailed {
t.Fatalf("analyze status = %#v, want preserved failed", loaded.Stages["analyze"])
}
if loaded.Stages["transcribe"] == nil || loaded.Stages["transcribe"].Status != manifest.StatusSucceeded {
t.Fatalf("transcribe status = %#v, want preserved succeeded", loaded.Stages["transcribe"])
}
}
func TestExecuteStagesFailureUpdatesManifest(t *testing.T) {
cfg := testConfig(t)
@@ -280,7 +545,14 @@ func TestExecuteStagesLoadsExistingManifest(t *testing.T) {
existing := manifest.New(cfg.Session.SessionID, time.Date(2026, 5, 3, 1, 0, 0, 0, time.UTC))
existing.MarkStageSucceeded("prepare", time.Date(2026, 5, 3, 1, 1, 0, 0, time.UTC), nil)
audioPath := filepath.Join(cfg.Pipeline.Workspace.Root, "work", cfg.Session.SessionID, "audio", "alice.flac")
audioPath := filepath.Join(
cfg.Pipeline.Workspace.Root,
"work",
cfg.Session.Campaign,
cfg.Session.SessionID,
"audio",
"alice.flac",
)
if err := os.MkdirAll(filepath.Dir(audioPath), 0o755); err != nil {
t.Fatalf("MkdirAll() error = %v", err)
}
@@ -315,6 +587,208 @@ func TestExecuteStagesLoadsExistingManifest(t *testing.T) {
}
}
func TestExecuteStagesCreatesRunManifestPerInvocation(t *testing.T) {
cfg := testConfig(t)
run1, err := executeStages(context.Background(), cfg, []stage.Stage{BuildFullPlan()[0]}, RunOptions{})
if err != nil {
t.Fatalf("first executeStages() error = %v", err)
}
run2, err := executeStages(context.Background(), cfg, []stage.Stage{BuildFullPlan()[0]}, RunOptions{Force: true})
if err != nil {
t.Fatalf("second executeStages() error = %v", err)
}
if run1.RunID == "" || run2.RunID == "" {
t.Fatalf("run ids must be set, got %q and %q", run1.RunID, run2.RunID)
}
if run1.RunID == run2.RunID {
t.Fatalf("expected distinct run ids, got %q", run1.RunID)
}
if run1.RunManifestPath == "" || run2.RunManifestPath == "" {
t.Fatalf("run manifest paths must be set, got %q and %q", run1.RunManifestPath, run2.RunManifestPath)
}
if run1.RunManifestPath == run2.RunManifestPath {
t.Fatalf("expected distinct run manifest paths, got %q", run1.RunManifestPath)
}
for _, path := range []string{run1.RunManifestPath, run2.RunManifestPath} {
if _, statErr := os.Stat(path); statErr != nil {
t.Fatalf("run manifest missing at %q: %v", path, statErr)
}
}
store := &manifest.LocalStore{}
sessionManifest, err := store.Load(context.Background(), run2.ManifestPath)
if err != nil {
t.Fatalf("Load session manifest error = %v", err)
}
if sessionManifest.RunID != run2.RunID {
t.Fatalf("session manifest run_id = %q, want latest run id %q", sessionManifest.RunID, run2.RunID)
}
}
func TestExecuteStagesRunManifestRecordsSkippedStage(t *testing.T) {
cfg := testConfig(t)
store := &manifest.LocalStore{}
manifestPath := manifestPathFor(cfg)
existing := manifest.New(cfg.Session.SessionID, time.Date(2026, 5, 3, 1, 0, 0, 0, time.UTC))
existing.MarkStageSucceeded("transcribe", time.Date(2026, 5, 3, 1, 1, 0, 0, time.UTC), nil)
if err := os.MkdirAll(filepath.Dir(manifestPath), 0o755); err != nil {
t.Fatalf("MkdirAll() error = %v", err)
}
if err := store.Save(context.Background(), manifestPath, existing); err != nil {
t.Fatalf("Save manifest error = %v", err)
}
summary, err := executeStages(context.Background(), cfg, []stage.Stage{BuildFullPlan()[1]}, RunOptions{})
if err != nil {
t.Fatalf("executeStages() error = %v", err)
}
if len(summary.Skipped) != 1 || summary.Skipped[0] != "transcribe" {
t.Fatalf("summary = %#v, want skipped transcribe", summary)
}
runManifest, err := store.LoadRun(context.Background(), summary.RunManifestPath)
if err != nil {
t.Fatalf("LoadRun() error = %v", err)
}
sr := runManifest.Stages["transcribe"]
if sr == nil {
t.Fatal("run manifest transcribe stage missing")
}
if sr.Action != manifest.RunStageActionSkip {
t.Fatalf("action = %q, want %q", sr.Action, manifest.RunStageActionSkip)
}
if sr.Status != manifest.StatusSkipped {
t.Fatalf("status = %q, want %q", sr.Status, manifest.StatusSkipped)
}
}
func TestExecuteStagesRunLocalArtifactsAndCanonicalPromotion(t *testing.T) {
cfg := testConfig(t)
stages := []stage.Stage{
BuildFullPlan()[0], // prepare
BuildFullPlan()[1], // transcribe
BuildFullPlan()[2], // merge
BuildFullPlan()[3], // polish
BuildFullPlan()[4], // normalize
BuildFullPlan()[5], // trim
}
summary, err := executeStages(context.Background(), cfg, stages, RunOptions{})
if err != nil {
t.Fatalf("executeStages() error = %v", err)
}
if summary.RunID == "" {
t.Fatal("run id must be set")
}
runRoot := artifacts.SessionRunRootForCampaign(
cfg.Pipeline.Workspace.Root,
cfg.Session.Campaign,
cfg.Session.SessionID,
summary.RunID,
)
paths := artifacts.NewLocalStore(cfg.Pipeline.Workspace.Root).SessionPathsFor(cfg.Session.Campaign, cfg.Session.SessionID)
runLocalChecks := []string{
filepath.Join(runRoot, "transcribe", "outputs", "transcripts", "raw", "alice.json"),
filepath.Join(runRoot, "merge", "logs", "seriatim.stdout.log"),
filepath.Join(runRoot, "polish", "config", "audita.generated.yml"),
filepath.Join(runRoot, "normalize", "logs", "seriatim.normalize.stdout.log"),
filepath.Join(runRoot, "trim", "outputs", "transcripts", "trimmed.json"),
}
for _, p := range runLocalChecks {
if _, statErr := os.Stat(p); statErr != nil {
t.Fatalf("run-local artifact missing at %q: %v", p, statErr)
}
}
canonicalChecks := []string{
filepath.Join(paths.TranscriptsRawDir, "alice.json"),
filepath.Join(paths.TranscriptsDir, "merged.json"),
filepath.Join(paths.TranscriptsDir, "processed.json"),
filepath.Join(paths.TranscriptsDir, "normalized.json"),
filepath.Join(paths.TranscriptsDir, "trimmed.json"),
}
for _, p := range canonicalChecks {
if _, statErr := os.Stat(p); statErr != nil {
t.Fatalf("canonical promoted artifact missing at %q: %v", p, statErr)
}
}
store := &manifest.LocalStore{}
sessionManifest, err := store.Load(context.Background(), summary.ManifestPath)
if err != nil {
t.Fatalf("Load manifest error = %v", err)
}
if got := sessionManifest.Stages["trim"]; got == nil || len(got.Outputs) == 0 {
t.Fatalf("trim stage outputs missing in session manifest: %#v", got)
}
for _, out := range sessionManifest.Stages["trim"].Outputs {
if strings.Contains(out.LocalPath, string(filepath.Separator)+"runs"+string(filepath.Separator)) {
t.Fatalf("session manifest output should be canonical, got run-local path %q", out.LocalPath)
}
if out.ProducerRunID != summary.RunID {
t.Fatalf("producer_run_id = %q, want %q", out.ProducerRunID, summary.RunID)
}
}
runManifest, err := store.LoadRun(context.Background(), summary.RunManifestPath)
if err != nil {
t.Fatalf("LoadRun() error = %v", err)
}
mergeStage := runManifest.Stages["merge"]
if mergeStage == nil || len(mergeStage.Logs) == 0 {
t.Fatalf("merge logs missing in run manifest: %#v", mergeStage)
}
for _, logPath := range mergeStage.Logs {
if !strings.Contains(logPath, filepath.Join("runs", summary.RunID, "merge", "logs")) {
t.Fatalf("run manifest merge log path = %q, want run-local merge logs path", logPath)
}
}
}
func TestExecuteStagesSkippedStagePreservesExistingOutputsProvenance(t *testing.T) {
cfg := testConfig(t)
store := &manifest.LocalStore{}
manifestPath := manifestPathFor(cfg)
existing := manifest.New(cfg.Session.SessionID, time.Date(2026, 5, 3, 1, 0, 0, 0, time.UTC))
existing.MarkStageSucceeded("transcribe", time.Date(2026, 5, 3, 1, 1, 0, 0, time.UTC), []manifest.ArtifactRecord{
{
Kind: "transcript_raw",
LocalPath: "transcripts/raw/alice.json",
ProducerRunID: "20260501T000000Z-deadbeef",
},
})
if err := os.MkdirAll(filepath.Dir(manifestPath), 0o755); err != nil {
t.Fatalf("MkdirAll() error = %v", err)
}
if err := store.Save(context.Background(), manifestPath, existing); err != nil {
t.Fatalf("Save manifest error = %v", err)
}
summary, err := executeStages(context.Background(), cfg, []stage.Stage{BuildFullPlan()[1]}, RunOptions{})
if err != nil {
t.Fatalf("executeStages() error = %v", err)
}
if len(summary.Skipped) != 1 || summary.Skipped[0] != "transcribe" {
t.Fatalf("summary = %#v, want skipped transcribe", summary)
}
loaded, err := store.Load(context.Background(), manifestPath)
if err != nil {
t.Fatalf("Load manifest error = %v", err)
}
got := loaded.Stages["transcribe"]
if got == nil || len(got.Outputs) != 1 {
t.Fatalf("transcribe outputs = %#v, want one preserved output", got)
}
if got.Outputs[0].ProducerRunID != "20260501T000000Z-deadbeef" {
t.Fatalf("producer_run_id = %q, want preserved value", got.Outputs[0].ProducerRunID)
}
}
func TestAdapterBackedStageFailureMarksManifestFailed(t *testing.T) {
cases := []struct {
name string
@@ -341,7 +815,7 @@ func TestAdapterBackedStageFailureMarksManifestFailed(t *testing.T) {
tc.env.ArtifactStore = artifactStore
tc.env.ManifestStore = &manifest.LocalStore{}
if tc.name == "transcribe" {
paths, ensureErr := artifactStore.EnsureLayout(cfg.Session.SessionID)
paths, ensureErr := artifactStore.EnsureLayoutFor(cfg.Session.Campaign, cfg.Session.SessionID)
if ensureErr != nil {
t.Fatalf("EnsureLayout() error = %v", ensureErr)
}
@@ -356,7 +830,7 @@ func TestAdapterBackedStageFailureMarksManifestFailed(t *testing.T) {
}
}
if tc.name == "merge" {
paths, ensureErr := artifactStore.EnsureLayout(cfg.Session.SessionID)
paths, ensureErr := artifactStore.EnsureLayoutFor(cfg.Session.Campaign, cfg.Session.SessionID)
if ensureErr != nil {
t.Fatalf("EnsureLayout() error = %v", ensureErr)
}
@@ -375,7 +849,7 @@ func TestAdapterBackedStageFailureMarksManifestFailed(t *testing.T) {
}
}
if tc.name == "polish" {
paths, ensureErr := artifactStore.EnsureLayout(cfg.Session.SessionID)
paths, ensureErr := artifactStore.EnsureLayoutFor(cfg.Session.Campaign, cfg.Session.SessionID)
if ensureErr != nil {
t.Fatalf("EnsureLayout() error = %v", ensureErr)
}
@@ -387,7 +861,7 @@ func TestAdapterBackedStageFailureMarksManifestFailed(t *testing.T) {
}
}
if tc.name == "analyze" {
paths, ensureErr := artifactStore.EnsureLayout(cfg.Session.SessionID)
paths, ensureErr := artifactStore.EnsureLayoutFor(cfg.Session.Campaign, cfg.Session.SessionID)
if ensureErr != nil {
t.Fatalf("EnsureLayout() error = %v", ensureErr)
}
@@ -403,7 +877,7 @@ func TestAdapterBackedStageFailureMarksManifestFailed(t *testing.T) {
PromptID: "dnd.session_recap",
OutputPath: "artifacts/session_recap.md",
Inputs: map[string]config.ScriptoriumInputConfig{
"transcript": {Source: "processed_transcript", Required: true},
"transcript": {Source: "narratio.transcript.polished", Required: true},
},
},
},

View File

@@ -63,7 +63,7 @@ func TestExecuteStagesDefaultWiringUsesWhisperXHTTPClient(t *testing.T) {
t.Fatal("audio file payload was empty")
}
outPath := filepath.Join(cfg.Pipeline.Workspace.Root, "work", cfg.Session.SessionID, "transcripts", "raw", "alice.json")
outPath := filepath.Join(cfg.Pipeline.Workspace.Root, "work", cfg.Session.Campaign, cfg.Session.SessionID, "transcripts", "raw", "alice.json")
data, err := os.ReadFile(outPath)
if err != nil {
t.Fatalf("ReadFile(%q) error = %v", outPath, err)

View File

@@ -0,0 +1,77 @@
package artifacts
import (
"fmt"
"strings"
"gitea.maximumdirect.net/eric/narratio/internal/config"
"gitea.maximumdirect.net/eric/narratio/internal/manifest"
)
// ResolveArchiveBucket resolves archive bucket identity with manifest-first precedence.
func ResolveArchiveBucket(cfg *config.Config, m *manifest.Manifest) string {
if m != nil && strings.TrimSpace(m.S3Bucket) != "" {
return strings.TrimSpace(m.S3Bucket)
}
if cfg == nil || cfg.Pipeline == nil || cfg.Pipeline.Storage.S3 == nil {
return ""
}
return strings.TrimSpace(cfg.Pipeline.Storage.S3.Bucket)
}
// ResolveArchiveSessionPrefix resolves archive session prefix with manifest-first precedence.
func ResolveArchiveSessionPrefix(cfg *config.Config, m *manifest.Manifest) (string, error) {
if m != nil && strings.TrimSpace(m.S3SessionPrefix) != "" {
return strings.TrimSpace(m.S3SessionPrefix), nil
}
if cfg == nil || cfg.Session == nil || cfg.Pipeline == nil {
return "", fmt.Errorf("resolved config is required")
}
sessionID := strings.TrimSpace(cfg.Session.SessionID)
if sessionID == "" && m != nil {
sessionID = strings.TrimSpace(m.SessionID)
}
campaign := strings.TrimSpace(cfg.Session.Campaign)
if campaign == "" && m != nil {
campaign = strings.TrimSpace(m.Campaign)
}
if cfg.Pipeline.Storage.S3 == nil {
return "", fmt.Errorf("pipeline.storage.s3 configuration is required")
}
sessionPrefix := S3SessionPrefix(cfg.Pipeline.Storage.S3.RootPrefix, campaign, sessionID)
if strings.TrimSpace(sessionPrefix) == "" {
return "", fmt.Errorf("session prefix is required")
}
return sessionPrefix, nil
}
// ResolveArchiveRunPrefix resolves archive run prefix with manifest-first precedence.
func ResolveArchiveRunPrefix(cfg *config.Config, m *manifest.Manifest) (string, error) {
if m != nil {
runPrefix := strings.TrimSpace(m.S3RunPrefix)
if runPrefix != "" {
return runPrefix, nil
}
}
sessionPrefix, err := ResolveArchiveSessionPrefix(cfg, m)
if err != nil {
return "", err
}
runID := ""
if m != nil {
runID = strings.TrimSpace(m.RunID)
}
if runID == "" {
return "", fmt.Errorf("run id is required")
}
return S3RunPrefix(sessionPrefix, runID), nil
}
// ResolveArchiveCurrentStateKeys returns current pointer keys for a session prefix.
func ResolveArchiveCurrentStateKeys(sessionPrefix string) (manifestKey, runIDKey string) {
return S3CurrentManifestKey(sessionPrefix), S3CurrentRunPointerKey(sessionPrefix)
}

View File

@@ -0,0 +1,130 @@
package artifacts
import (
"strings"
"testing"
"gitea.maximumdirect.net/eric/narratio/internal/config"
"gitea.maximumdirect.net/eric/narratio/internal/manifest"
)
func TestResolveArchiveBucketPrefersManifestThenConfig(t *testing.T) {
cfg := &config.Config{
Pipeline: &config.PipelineConfig{
Storage: config.StorageConfig{
S3: &config.StorageS3Config{Bucket: "cfg-bucket"},
},
},
}
if got := ResolveArchiveBucket(cfg, &manifest.Manifest{S3Bucket: "manifest-bucket"}); got != "manifest-bucket" {
t.Fatalf("bucket = %q, want manifest-bucket", got)
}
if got := ResolveArchiveBucket(cfg, &manifest.Manifest{}); got != "cfg-bucket" {
t.Fatalf("bucket = %q, want cfg-bucket", got)
}
}
func TestResolveArchiveSessionPrefixPrefersManifestThenConfig(t *testing.T) {
cfg := &config.Config{
Pipeline: &config.PipelineConfig{
Storage: config.StorageConfig{
S3: &config.StorageS3Config{RootPrefix: "dnd"},
},
},
Session: &config.SessionConfig{
SessionID: "2026-04-19",
Campaign: "forsaken",
},
}
m := &manifest.Manifest{S3SessionPrefix: "manifest/session/prefix/"}
got, err := ResolveArchiveSessionPrefix(cfg, m)
if err != nil {
t.Fatalf("ResolveArchiveSessionPrefix() error = %v", err)
}
if got != "manifest/session/prefix/" {
t.Fatalf("session prefix = %q, want manifest/session/prefix/", got)
}
got, err = ResolveArchiveSessionPrefix(cfg, &manifest.Manifest{})
if err != nil {
t.Fatalf("ResolveArchiveSessionPrefix() error = %v", err)
}
want := "dnd/campaigns/forsaken/sessions/2026-04-19/"
if got != want {
t.Fatalf("session prefix = %q, want %q", got, want)
}
}
func TestResolveArchiveRunPrefixPrefersManifestThenDerived(t *testing.T) {
cfg := &config.Config{
Pipeline: &config.PipelineConfig{
Storage: config.StorageConfig{
S3: &config.StorageS3Config{RootPrefix: "dnd"},
},
},
Session: &config.SessionConfig{
SessionID: "2026-04-19",
Campaign: "forsaken",
},
}
m := &manifest.Manifest{
RunID: "20260516T010203Z-1a2b3c4d",
S3RunPrefix: "manifest/run/prefix/",
}
got, err := ResolveArchiveRunPrefix(cfg, m)
if err != nil {
t.Fatalf("ResolveArchiveRunPrefix() error = %v", err)
}
if got != "manifest/run/prefix/" {
t.Fatalf("run prefix = %q, want manifest/run/prefix/", got)
}
m = &manifest.Manifest{
RunID: "20260516T010203Z-1a2b3c4d",
}
got, err = ResolveArchiveRunPrefix(cfg, m)
if err != nil {
t.Fatalf("ResolveArchiveRunPrefix() error = %v", err)
}
want := "dnd/campaigns/forsaken/sessions/2026-04-19/runs/20260516T010203Z-1a2b3c4d/"
if got != want {
t.Fatalf("run prefix = %q, want %q", got, want)
}
}
func TestResolveArchiveIdentityErrorsAreDeterministic(t *testing.T) {
cfgNoS3 := &config.Config{
Pipeline: &config.PipelineConfig{},
Session: &config.SessionConfig{SessionID: "2026-04-19", Campaign: "forsaken"},
}
_, err := ResolveArchiveSessionPrefix(cfgNoS3, &manifest.Manifest{})
if err == nil || !strings.Contains(err.Error(), "pipeline.storage.s3 configuration is required") {
t.Fatalf("error = %v, want missing storage.s3", err)
}
cfg := &config.Config{
Pipeline: &config.PipelineConfig{
Storage: config.StorageConfig{
S3: &config.StorageS3Config{RootPrefix: "dnd"},
},
},
Session: &config.SessionConfig{SessionID: "2026-04-19", Campaign: "forsaken"},
}
_, err = ResolveArchiveRunPrefix(cfg, &manifest.Manifest{})
if err == nil || !strings.Contains(err.Error(), "run id is required") {
t.Fatalf("error = %v, want missing run id", err)
}
}
func TestResolveArchiveCurrentStateKeys(t *testing.T) {
manifestKey, runIDKey := ResolveArchiveCurrentStateKeys("dnd/campaigns/forsaken/sessions/2026-04-19/")
if manifestKey != "dnd/campaigns/forsaken/sessions/2026-04-19/current/manifest.json" {
t.Fatalf("manifest key = %q", manifestKey)
}
if runIDKey != "dnd/campaigns/forsaken/sessions/2026-04-19/current/run_id.txt" {
t.Fatalf("run id key = %q", runIDKey)
}
}

View File

@@ -0,0 +1,313 @@
package artifacts
import (
"encoding/json"
"errors"
"fmt"
"os"
"path/filepath"
"regexp"
"strings"
"gitea.maximumdirect.net/eric/narratio/internal/manifest"
)
const (
ArtifactTranscriptMerged = "narratio.transcript.merged"
ArtifactTranscriptPolished = "narratio.transcript.polished"
ArtifactTranscriptFull = "narratio.transcript.full"
ArtifactTranscriptTrimmed = "narratio.transcript.trimmed"
ArtifactBoundsSession = "narratio.bounds.session"
)
// ErrSessionArtifactNotFound is returned when no readable artifact exists for a known ID.
var ErrSessionArtifactNotFound = errors.New("session artifact not found")
var configuredArtifactSourceRE = regexp.MustCompile(`^narratio\.artifact\.[a-z][a-z0-9_]*$`)
type artifactContentKind string
const (
contentTranscriptJSON artifactContentKind = "transcript_json"
contentJSON artifactContentKind = "json"
contentText artifactContentKind = "text"
)
type artifactSpec struct {
ID string
CanonicalRelPath string
ProducerStage string
OutputKind string
ContentKind artifactContentKind
}
var artifactRegistry = map[string]artifactSpec{
ArtifactTranscriptMerged: {
ID: ArtifactTranscriptMerged,
CanonicalRelPath: "transcripts/merged.json",
ProducerStage: "merge",
OutputKind: "transcript_merged",
ContentKind: contentTranscriptJSON,
},
ArtifactTranscriptPolished: {
ID: ArtifactTranscriptPolished,
CanonicalRelPath: "transcripts/processed.json",
ProducerStage: "polish",
OutputKind: "transcript_processed",
ContentKind: contentTranscriptJSON,
},
ArtifactTranscriptFull: {
ID: ArtifactTranscriptFull,
CanonicalRelPath: "transcripts/normalized.json",
ProducerStage: "normalize",
OutputKind: "transcript_normalized",
ContentKind: contentTranscriptJSON,
},
ArtifactTranscriptTrimmed: {
ID: ArtifactTranscriptTrimmed,
CanonicalRelPath: "transcripts/trimmed.json",
ProducerStage: "trim",
OutputKind: "transcript_trimmed",
ContentKind: contentTranscriptJSON,
},
ArtifactBoundsSession: {
ID: ArtifactBoundsSession,
CanonicalRelPath: "artifacts/session_bounds.json",
ProducerStage: "trim",
OutputKind: "session_bounds",
ContentKind: contentJSON,
},
}
// ResolvedSessionArtifact describes one session-level artifact lookup result.
type ResolvedSessionArtifact struct {
ID string
Path string
ProducerStage string
OutputKind string
ProducerRunID string
Provenance string
}
// SessionArtifactNotFoundError includes context when a known artifact cannot be read.
type SessionArtifactNotFoundError struct {
ArtifactID string
}
func (e *SessionArtifactNotFoundError) Error() string {
return fmt.Sprintf("%s: %q", ErrSessionArtifactNotFound, e.ArtifactID)
}
func (e *SessionArtifactNotFoundError) Unwrap() error {
return ErrSessionArtifactNotFound
}
// NormalizeSessionArtifactSource validates canonical artifact IDs.
func NormalizeSessionArtifactSource(source string) (string, error) {
normalized := strings.TrimSpace(source)
if normalized == "" {
return "", fmt.Errorf("artifact source is required")
}
if _, ok := artifactRegistry[normalized]; !ok {
return "", fmt.Errorf("unsupported artifact source %q", source)
}
return normalized, nil
}
// IsConfiguredArtifactSource returns true when source is narratio.artifact.<name>.
func IsConfiguredArtifactSource(source string) bool {
return configuredArtifactSourceRE.MatchString(strings.TrimSpace(source))
}
// ResolveSessionArtifact resolves a symbolic source to a readable local session artifact path.
// Resolution order is manifest producer outputs first, then canonical session path fallback.
func ResolveSessionArtifact(paths SessionPaths, m *manifest.Manifest, source string) (ResolvedSessionArtifact, error) {
id, err := NormalizeSessionArtifactSource(source)
if err != nil {
return ResolvedSessionArtifact{}, err
}
spec := artifactRegistry[id]
for _, candidate := range manifestArtifactCandidates(paths, m, spec) {
exists, isDir, statErr := pathExists(candidate.Path)
if statErr != nil {
return ResolvedSessionArtifact{}, fmt.Errorf("stat %q: %w", candidate.Path, statErr)
}
if !exists || isDir {
continue
}
resolved := candidate
resolved.ID = spec.ID
resolved.ProducerStage = spec.ProducerStage
resolved.OutputKind = spec.OutputKind
if err := validateResolvedContent(resolved.Path, spec.ContentKind); err != nil {
return ResolvedSessionArtifact{}, fmt.Errorf("validate %q: %w", resolved.ID, err)
}
return resolved, nil
}
fallbackPath := filepath.Join(paths.Root, filepath.FromSlash(spec.CanonicalRelPath))
exists, isDir, statErr := pathExists(fallbackPath)
if statErr != nil {
return ResolvedSessionArtifact{}, fmt.Errorf("stat %q: %w", fallbackPath, statErr)
}
if exists && !isDir {
if err := validateResolvedContent(fallbackPath, spec.ContentKind); err != nil {
return ResolvedSessionArtifact{}, fmt.Errorf("validate %q: %w", spec.ID, err)
}
return ResolvedSessionArtifact{
ID: spec.ID,
Path: filepath.Clean(fallbackPath),
ProducerStage: spec.ProducerStage,
OutputKind: spec.OutputKind,
Provenance: "fallback.canonical_path",
}, nil
}
return ResolvedSessionArtifact{}, &SessionArtifactNotFoundError{ArtifactID: spec.ID}
}
// ResolveSessionArtifactWithCatalog resolves built-in sources using existing rules and resolves
// configured narratio.artifact.<name> sources through runtime catalog availability.
func ResolveSessionArtifactWithCatalog(paths SessionPaths, m *manifest.Manifest, source string, catalog *ArtifactCatalog) (ResolvedSessionArtifact, error) {
normalized := strings.TrimSpace(source)
if !IsConfiguredArtifactSource(normalized) {
return ResolveSessionArtifact(paths, m, normalized)
}
if catalog == nil {
return ResolvedSessionArtifact{}, fmt.Errorf("configured artifact source %q requires runtime artifact catalog", source)
}
entry, ok := catalog.Lookup(normalized)
if !ok {
return ResolvedSessionArtifact{}, fmt.Errorf("unsupported artifact source %q", source)
}
if !entry.Available {
return ResolvedSessionArtifact{}, &SessionArtifactNotFoundError{ArtifactID: normalized}
}
if err := validateResolvedContent(entry.Path, contentText); err != nil {
return ResolvedSessionArtifact{}, fmt.Errorf("validate %q: %w", normalized, err)
}
return ResolvedSessionArtifact{
ID: normalized,
Path: filepath.Clean(entry.Path),
ProducerStage: entry.ProducerStage,
OutputKind: entry.OutputKind,
Provenance: entry.Provenance,
}, nil
}
func manifestArtifactCandidates(paths SessionPaths, m *manifest.Manifest, spec artifactSpec) []ResolvedSessionArtifact {
if m == nil || len(m.Stages) == 0 || spec.ProducerStage == "" || spec.OutputKind == "" {
return nil
}
sr := m.Stages[spec.ProducerStage]
if sr == nil {
return nil
}
candidates := make([]ResolvedSessionArtifact, 0, len(sr.Outputs))
for _, out := range sr.Outputs {
if strings.TrimSpace(out.Kind) != spec.OutputKind {
continue
}
if strings.TrimSpace(out.LocalPath) == "" {
continue
}
resolved := filepath.Clean(ResolveSessionLocalPathForRead(paths, out.LocalPath))
if resolved == "" {
continue
}
candidates = append(candidates, ResolvedSessionArtifact{
Path: resolved,
ProducerRunID: strings.TrimSpace(out.ProducerRunID),
Provenance: "manifest." + spec.ProducerStage + ".outputs",
})
}
return dedupeResolvedArtifacts(candidates)
}
func dedupeResolvedArtifacts(values []ResolvedSessionArtifact) []ResolvedSessionArtifact {
seen := map[string]struct{}{}
out := make([]ResolvedSessionArtifact, 0, len(values))
for _, value := range values {
key := filepath.Clean(strings.TrimSpace(value.Path))
if key == "" {
continue
}
if _, ok := seen[key]; ok {
continue
}
seen[key] = struct{}{}
value.Path = key
out = append(out, value)
}
return out
}
func pathExists(path string) (exists bool, isDir bool, err error) {
info, err := os.Stat(path)
if err == nil {
return true, info.IsDir(), nil
}
if errors.Is(err, os.ErrNotExist) {
return false, false, nil
}
return false, false, err
}
func validateResolvedContent(path string, kind artifactContentKind) error {
switch kind {
case contentTranscriptJSON:
return validateTranscriptSegmentsJSON(path)
case contentJSON:
return validateJSONContent(path)
case contentText:
return validateNonEmptyContent(path)
default:
return fmt.Errorf("unsupported content kind %q", kind)
}
}
func validateTranscriptSegmentsJSON(path string) error {
data, err := os.ReadFile(path)
if err != nil {
return fmt.Errorf("read file: %w", err)
}
var payload map[string]any
if err := json.Unmarshal(data, &payload); err != nil {
return fmt.Errorf("decode json: %w", err)
}
segments, ok := payload["segments"]
if !ok {
return fmt.Errorf("top-level segments is required")
}
if _, ok := segments.([]any); !ok {
return fmt.Errorf("top-level segments must be an array")
}
return nil
}
func validateJSONContent(path string) error {
data, err := os.ReadFile(path)
if err != nil {
return fmt.Errorf("read file: %w", err)
}
var payload any
if err := json.Unmarshal(data, &payload); err != nil {
return fmt.Errorf("decode json: %w", err)
}
return nil
}
func validateNonEmptyContent(path string) error {
info, err := os.Stat(path)
if err != nil {
return fmt.Errorf("stat file: %w", err)
}
if info.IsDir() {
return fmt.Errorf("path is a directory")
}
if info.Size() <= 0 {
return fmt.Errorf("file is empty")
}
return nil
}

View File

@@ -0,0 +1,268 @@
package artifacts
import (
"errors"
"os"
"path/filepath"
"strings"
"testing"
"time"
"gitea.maximumdirect.net/eric/narratio/internal/manifest"
)
func TestNormalizeSessionArtifactSource(t *testing.T) {
tests := []struct {
name string
source string
wantID string
wantErr string
}{
{name: "legacy alias processed unsupported", source: "processed_transcript", wantErr: "unsupported artifact source"},
{name: "legacy alias normalized unsupported", source: "normalized_transcript", wantErr: "unsupported artifact source"},
{name: "legacy alias trimmed unsupported", source: "trimmed_transcript", wantErr: "unsupported artifact source"},
{name: "configured source unsupported in built-in normalization", source: "narratio.artifact.session_recap", wantErr: "unsupported artifact source"},
{name: "canonical", source: ArtifactTranscriptTrimmed, wantID: ArtifactTranscriptTrimmed},
{name: "unsupported", source: "narratio.unknown", wantErr: "unsupported artifact source"},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
got, err := NormalizeSessionArtifactSource(tt.source)
if tt.wantErr != "" {
if err == nil || !strings.Contains(err.Error(), tt.wantErr) {
t.Fatalf("NormalizeSessionArtifactSource() error = %v, want contains %q", err, tt.wantErr)
}
return
}
if err != nil {
t.Fatalf("NormalizeSessionArtifactSource() error = %v", err)
}
if got != tt.wantID {
t.Fatalf("NormalizeSessionArtifactSource() = %q, want %q", got, tt.wantID)
}
})
}
}
func TestResolveSessionArtifactPrefersManifestOutput(t *testing.T) {
workspace := t.TempDir()
paths := buildSessionPaths(workspace, "campaign", "session")
if err := os.MkdirAll(paths.ArtifactsDir, 0o755); err != nil {
t.Fatalf("MkdirAll() error = %v", err)
}
manifestPath := filepath.Join(paths.ArtifactsDir, "normalized.from-manifest.json")
if err := os.WriteFile(manifestPath, []byte(`{"segments":[]}`), 0o644); err != nil {
t.Fatalf("WriteFile() error = %v", err)
}
canonicalPath := filepath.Join(paths.TranscriptsDir, "normalized.json")
if err := os.MkdirAll(filepath.Dir(canonicalPath), 0o755); err != nil {
t.Fatalf("MkdirAll() error = %v", err)
}
if err := os.WriteFile(canonicalPath, []byte(`{"segments":[{"id":123}]}`), 0o644); err != nil {
t.Fatalf("WriteFile() error = %v", err)
}
m := manifest.New("session", time.Now().UTC())
m.MarkStageSucceeded("normalize", time.Now().UTC(), []manifest.ArtifactRecord{
{Kind: "transcript_normalized", LocalPath: manifestPath, ProducerRunID: "run-123"},
})
resolved, err := ResolveSessionArtifact(paths, m, ArtifactTranscriptFull)
if err != nil {
t.Fatalf("ResolveSessionArtifact() error = %v", err)
}
if resolved.Path != manifestPath {
t.Fatalf("resolved path = %q, want %q", resolved.Path, manifestPath)
}
if resolved.Provenance != "manifest.normalize.outputs" {
t.Fatalf("provenance = %q, want manifest.normalize.outputs", resolved.Provenance)
}
if resolved.ProducerRunID != "run-123" {
t.Fatalf("producer run id = %q, want run-123", resolved.ProducerRunID)
}
}
func TestResolveSessionArtifactFallsBackToCanonicalPath(t *testing.T) {
workspace := t.TempDir()
paths := buildSessionPaths(workspace, "campaign", "session")
canonicalPath := filepath.Join(paths.TranscriptsDir, "trimmed.json")
if err := os.MkdirAll(filepath.Dir(canonicalPath), 0o755); err != nil {
t.Fatalf("MkdirAll() error = %v", err)
}
if err := os.WriteFile(canonicalPath, []byte(`{"segments":[]}`), 0o644); err != nil {
t.Fatalf("WriteFile() error = %v", err)
}
resolved, err := ResolveSessionArtifact(paths, nil, ArtifactTranscriptTrimmed)
if err != nil {
t.Fatalf("ResolveSessionArtifact() error = %v", err)
}
if resolved.Path != canonicalPath {
t.Fatalf("resolved path = %q, want %q", resolved.Path, canonicalPath)
}
if resolved.Provenance != "fallback.canonical_path" {
t.Fatalf("provenance = %q, want fallback.canonical_path", resolved.Provenance)
}
}
func TestResolveSessionArtifactMissingReturnsTypedError(t *testing.T) {
workspace := t.TempDir()
paths := buildSessionPaths(workspace, "campaign", "session")
_, err := ResolveSessionArtifact(paths, nil, ArtifactTranscriptTrimmed)
if err == nil {
t.Fatal("expected error, got nil")
}
if !errors.Is(err, ErrSessionArtifactNotFound) {
t.Fatalf("errors.Is(err, ErrSessionArtifactNotFound) = false; err=%v", err)
}
}
func TestResolveSessionArtifactValidatesTranscriptShape(t *testing.T) {
workspace := t.TempDir()
paths := buildSessionPaths(workspace, "campaign", "session")
canonicalPath := filepath.Join(paths.TranscriptsDir, "processed.json")
if err := os.MkdirAll(filepath.Dir(canonicalPath), 0o755); err != nil {
t.Fatalf("MkdirAll() error = %v", err)
}
if err := os.WriteFile(canonicalPath, []byte(`{"not_segments":[]}`), 0o644); err != nil {
t.Fatalf("WriteFile() error = %v", err)
}
_, err := ResolveSessionArtifact(paths, nil, ArtifactTranscriptPolished)
if err == nil {
t.Fatal("expected error, got nil")
}
if !strings.Contains(err.Error(), "top-level segments is required") {
t.Fatalf("error = %q, want segments validation error", err.Error())
}
}
func TestResolveSessionArtifactWithCatalogBuiltInBehaviorUnchanged(t *testing.T) {
workspace := t.TempDir()
paths := buildSessionPaths(workspace, "campaign", "session")
canonicalPath := filepath.Join(paths.TranscriptsDir, "trimmed.json")
if err := os.MkdirAll(filepath.Dir(canonicalPath), 0o755); err != nil {
t.Fatalf("MkdirAll() error = %v", err)
}
if err := os.WriteFile(canonicalPath, []byte(`{"segments":[]}`), 0o644); err != nil {
t.Fatalf("WriteFile() error = %v", err)
}
resolved, err := ResolveSessionArtifactWithCatalog(paths, nil, ArtifactTranscriptTrimmed, NewArtifactCatalog())
if err != nil {
t.Fatalf("ResolveSessionArtifactWithCatalog() error = %v", err)
}
if resolved.Path != canonicalPath {
t.Fatalf("resolved.Path = %q, want %q", resolved.Path, canonicalPath)
}
if resolved.Provenance != "fallback.canonical_path" {
t.Fatalf("provenance = %q, want fallback.canonical_path", resolved.Provenance)
}
}
func TestResolveSessionArtifactWithCatalogConfiguredAvailableGenerated(t *testing.T) {
workspace := t.TempDir()
paths := buildSessionPaths(workspace, "campaign", "session")
outputPath := filepath.Join(paths.ArtifactsDir, "session_recap.md")
if err := os.MkdirAll(filepath.Dir(outputPath), 0o755); err != nil {
t.Fatalf("MkdirAll() error = %v", err)
}
if err := os.WriteFile(outputPath, []byte("recap\n"), 0o644); err != nil {
t.Fatalf("WriteFile() error = %v", err)
}
catalog := NewArtifactCatalog()
if err := catalog.RegisterConfiguredArtifacts(
map[string]ConfiguredArtifactDefinition{
"session_recap": {Enabled: true, OutputPath: "artifacts/session_recap.md"},
},
nil,
); err != nil {
t.Fatalf("RegisterConfiguredArtifacts() error = %v", err)
}
sourceID := ConfiguredArtifactSourceID("session_recap")
if err := catalog.MarkAvailableGenerated(sourceID, outputPath); err != nil {
t.Fatalf("MarkAvailableGenerated() error = %v", err)
}
resolved, err := ResolveSessionArtifactWithCatalog(paths, nil, sourceID, catalog)
if err != nil {
t.Fatalf("ResolveSessionArtifactWithCatalog() error = %v", err)
}
if resolved.Path != outputPath {
t.Fatalf("resolved.Path = %q, want %q", resolved.Path, outputPath)
}
if resolved.Provenance != ArtifactProvenanceGeneratedCurrentAnalyzeRun {
t.Fatalf("provenance = %q, want %q", resolved.Provenance, ArtifactProvenanceGeneratedCurrentAnalyzeRun)
}
}
func TestResolveSessionArtifactWithCatalogConfiguredAvailableFromDisk(t *testing.T) {
workspace := t.TempDir()
paths := buildSessionPaths(workspace, "campaign", "session")
outputPath := filepath.Join(paths.ArtifactsDir, "player_handout.md")
if err := os.MkdirAll(filepath.Dir(outputPath), 0o755); err != nil {
t.Fatalf("MkdirAll() error = %v", err)
}
if err := os.WriteFile(outputPath, []byte("handout\n"), 0o644); err != nil {
t.Fatalf("WriteFile() error = %v", err)
}
catalog := NewArtifactCatalog()
if err := catalog.RegisterConfiguredArtifacts(
map[string]ConfiguredArtifactDefinition{
"player_handout": {Enabled: false, OutputPath: "artifacts/player_handout.md"},
},
nil,
); err != nil {
t.Fatalf("RegisterConfiguredArtifacts() error = %v", err)
}
sourceID := ConfiguredArtifactSourceID("player_handout")
if err := catalog.MarkAvailableFromDisk(sourceID, outputPath); err != nil {
t.Fatalf("MarkAvailableFromDisk() error = %v", err)
}
resolved, err := ResolveSessionArtifactWithCatalog(paths, nil, sourceID, catalog)
if err != nil {
t.Fatalf("ResolveSessionArtifactWithCatalog() error = %v", err)
}
if resolved.Provenance != ArtifactProvenanceDisabledFromDisk {
t.Fatalf("provenance = %q, want %q", resolved.Provenance, ArtifactProvenanceDisabledFromDisk)
}
}
func TestResolveSessionArtifactWithCatalogConfiguredPlannedButUnavailable(t *testing.T) {
workspace := t.TempDir()
paths := buildSessionPaths(workspace, "campaign", "session")
catalog := NewArtifactCatalog()
if err := catalog.RegisterConfiguredArtifacts(
map[string]ConfiguredArtifactDefinition{
"session_recap": {Enabled: true, OutputPath: "artifacts/session_recap.md"},
},
nil,
); err != nil {
t.Fatalf("RegisterConfiguredArtifacts() error = %v", err)
}
_, err := ResolveSessionArtifactWithCatalog(paths, nil, ConfiguredArtifactSourceID("session_recap"), catalog)
if err == nil {
t.Fatal("expected error, got nil")
}
if !errors.Is(err, ErrSessionArtifactNotFound) {
t.Fatalf("errors.Is(err, ErrSessionArtifactNotFound)=false; err=%v", err)
}
}
func TestResolveSessionArtifactWithCatalogUnsupportedConfiguredSourceFails(t *testing.T) {
workspace := t.TempDir()
paths := buildSessionPaths(workspace, "campaign", "session")
catalog := NewArtifactCatalog()
_, err := ResolveSessionArtifactWithCatalog(paths, nil, "narratio.artifact.unknown", catalog)
if err == nil {
t.Fatal("expected error, got nil")
}
if !strings.Contains(err.Error(), "unsupported artifact source") {
t.Fatalf("error = %q, want unsupported artifact source", err.Error())
}
}

View File

@@ -0,0 +1,224 @@
package artifacts
import (
"fmt"
"sort"
"strings"
)
const (
ArtifactProvenanceGeneratedCurrentAnalyzeRun = "generated.current_analyze_run"
ArtifactProvenanceDisabledFromDisk = "filesystem.disabled_artifact_output"
)
// ConfiguredArtifactDefinition describes one configured analyze artifact.
type ConfiguredArtifactDefinition struct {
Enabled bool
OutputPath string
}
// CatalogEntry is one runtime catalog entry resolved by source ID.
type CatalogEntry struct {
SourceID string
ConfiguredKey string
CanonicalRelPath string
ProducerStage string
OutputKind string
Planned bool
Executable bool
Available bool
Path string
Provenance string
}
// ArtifactCatalog tracks built-in and configured artifact definitions and runtime state.
type ArtifactCatalog struct {
entries map[string]CatalogEntry
configuredIndex map[string]string
}
// NewArtifactCatalog returns an empty runtime artifact catalog.
func NewArtifactCatalog() *ArtifactCatalog {
return &ArtifactCatalog{
entries: map[string]CatalogEntry{},
configuredIndex: map[string]string{},
}
}
// ConfiguredArtifactSourceID converts a configured artifact key into canonical source ID.
func ConfiguredArtifactSourceID(key string) string {
return "narratio.artifact." + strings.TrimSpace(key)
}
// RegisterBuiltIns registers built-in source definitions used by runtime artifact resolution.
func (c *ArtifactCatalog) RegisterBuiltIns() error {
for _, id := range runtimeBuiltInArtifactIDs() {
spec, ok := artifactRegistry[id]
if !ok {
return fmt.Errorf("register built-ins: source %q not found in artifact registry", id)
}
if err := c.addEntry(CatalogEntry{
SourceID: spec.ID,
CanonicalRelPath: spec.CanonicalRelPath,
ProducerStage: spec.ProducerStage,
OutputKind: spec.OutputKind,
Planned: true,
}); err != nil {
return fmt.Errorf("register built-ins: %w", err)
}
}
return nil
}
// RegisterConfiguredArtifacts registers configured artifacts and applies executable selection.
func (c *ArtifactCatalog) RegisterConfiguredArtifacts(
configured map[string]ConfiguredArtifactDefinition,
selected []string,
) error {
keys := make([]string, 0, len(configured))
for key := range configured {
keys = append(keys, key)
}
sort.Strings(keys)
selectedSet := map[string]struct{}{}
for _, key := range selected {
trimmed := strings.TrimSpace(key)
if trimmed == "" {
return fmt.Errorf("selected artifact keys must be non-empty")
}
selectedSet[trimmed] = struct{}{}
}
for _, key := range keys {
trimmed := strings.TrimSpace(key)
if trimmed == "" {
return fmt.Errorf("configured artifact keys must be non-empty")
}
def := configured[key]
sourceID := ConfiguredArtifactSourceID(trimmed)
if _, exists := c.configuredIndex[trimmed]; exists {
return fmt.Errorf("duplicate configured artifact key %q", trimmed)
}
executable := def.Enabled
if len(selectedSet) > 0 {
_, executable = selectedSet[trimmed]
}
if err := c.addEntry(CatalogEntry{
SourceID: sourceID,
ConfiguredKey: trimmed,
CanonicalRelPath: strings.TrimSpace(def.OutputPath),
ProducerStage: "analyze",
OutputKind: "scriptorium_artifact",
Planned: true,
Executable: executable,
}); err != nil {
return fmt.Errorf("register configured artifact %q: %w", trimmed, err)
}
c.configuredIndex[trimmed] = sourceID
}
if len(selectedSet) > 0 {
for key := range selectedSet {
if _, ok := c.configuredIndex[key]; !ok {
return fmt.Errorf("selected artifact %q is not configured", key)
}
}
}
return nil
}
// Lookup returns one catalog entry by source ID.
func (c *ArtifactCatalog) Lookup(sourceID string) (CatalogEntry, bool) {
if c == nil {
return CatalogEntry{}, false
}
entry, ok := c.entries[strings.TrimSpace(sourceID)]
return entry, ok
}
// SourceIDForConfiguredKey returns canonical source ID for one configured key.
func (c *ArtifactCatalog) SourceIDForConfiguredKey(key string) (string, bool) {
if c == nil {
return "", false
}
sourceID, ok := c.configuredIndex[strings.TrimSpace(key)]
return sourceID, ok
}
// ListConfigured returns configured entries sorted by configured key.
func (c *ArtifactCatalog) ListConfigured() []CatalogEntry {
if c == nil || len(c.configuredIndex) == 0 {
return nil
}
keys := make([]string, 0, len(c.configuredIndex))
for key := range c.configuredIndex {
keys = append(keys, key)
}
sort.Strings(keys)
out := make([]CatalogEntry, 0, len(keys))
for _, key := range keys {
sourceID := c.configuredIndex[key]
out = append(out, c.entries[sourceID])
}
return out
}
// MarkAvailableGenerated marks one source as available in current analyze execution.
func (c *ArtifactCatalog) MarkAvailableGenerated(sourceID, path string) error {
return c.markAvailable(sourceID, path, ArtifactProvenanceGeneratedCurrentAnalyzeRun)
}
// MarkAvailableFromDisk marks one source as available from disabled artifact on disk.
func (c *ArtifactCatalog) MarkAvailableFromDisk(sourceID, path string) error {
return c.markAvailable(sourceID, path, ArtifactProvenanceDisabledFromDisk)
}
func (c *ArtifactCatalog) markAvailable(sourceID, path, provenance string) error {
if c == nil {
return fmt.Errorf("artifact catalog is nil")
}
normalizedID := strings.TrimSpace(sourceID)
entry, ok := c.entries[normalizedID]
if !ok {
return fmt.Errorf("unknown artifact source %q", sourceID)
}
trimmedPath := strings.TrimSpace(path)
if trimmedPath == "" {
return fmt.Errorf("artifact path is required")
}
entry.Available = true
entry.Path = trimmedPath
entry.Provenance = provenance
c.entries[normalizedID] = entry
return nil
}
func (c *ArtifactCatalog) addEntry(entry CatalogEntry) error {
if c == nil {
return fmt.Errorf("artifact catalog is nil")
}
sourceID := strings.TrimSpace(entry.SourceID)
if sourceID == "" {
return fmt.Errorf("source id is required")
}
if _, exists := c.entries[sourceID]; exists {
return fmt.Errorf("source id %q is already registered", sourceID)
}
entry.SourceID = sourceID
c.entries[sourceID] = entry
return nil
}
func runtimeBuiltInArtifactIDs() []string {
return []string{
ArtifactTranscriptMerged,
ArtifactTranscriptPolished,
ArtifactTranscriptFull,
ArtifactTranscriptTrimmed,
ArtifactBoundsSession,
}
}

View File

@@ -0,0 +1,188 @@
package artifacts
import "testing"
func TestArtifactCatalogRegisterBuiltInsAndLookup(t *testing.T) {
catalog := NewArtifactCatalog()
if err := catalog.RegisterBuiltIns(); err != nil {
t.Fatalf("RegisterBuiltIns() error = %v", err)
}
entry, ok := catalog.Lookup(ArtifactTranscriptFull)
if !ok {
t.Fatalf("Lookup(%q) ok = false, want true", ArtifactTranscriptFull)
}
if !entry.Planned {
t.Fatalf("entry.Planned = false, want true")
}
if entry.Executable {
t.Fatalf("entry.Executable = true, want false")
}
if entry.CanonicalRelPath != "transcripts/normalized.json" {
t.Fatalf("entry.CanonicalRelPath = %q, want transcripts/normalized.json", entry.CanonicalRelPath)
}
}
func TestArtifactCatalogRegisterConfiguredArtifactsDefaultsToEnabled(t *testing.T) {
catalog := NewArtifactCatalog()
if err := catalog.RegisterConfiguredArtifacts(map[string]ConfiguredArtifactDefinition{
"player_handout": {Enabled: false, OutputPath: "artifacts/player_handout.md"},
"session_recap": {Enabled: true, OutputPath: "artifacts/session_recap.md"},
}, nil); err != nil {
t.Fatalf("RegisterConfiguredArtifacts() error = %v", err)
}
recapID, ok := catalog.SourceIDForConfiguredKey("session_recap")
if !ok {
t.Fatal("SourceIDForConfiguredKey(session_recap) ok = false, want true")
}
recap, ok := catalog.Lookup(recapID)
if !ok {
t.Fatalf("Lookup(%q) ok = false, want true", recapID)
}
if !recap.Executable {
t.Fatalf("recap.Executable = false, want true")
}
handoutID, ok := catalog.SourceIDForConfiguredKey("player_handout")
if !ok {
t.Fatal("SourceIDForConfiguredKey(player_handout) ok = false, want true")
}
handout, ok := catalog.Lookup(handoutID)
if !ok {
t.Fatalf("Lookup(%q) ok = false, want true", handoutID)
}
if handout.Executable {
t.Fatalf("handout.Executable = true, want false")
}
}
func TestArtifactCatalogRegisterConfiguredArtifactsSelectedSetOverridesEnabled(t *testing.T) {
catalog := NewArtifactCatalog()
if err := catalog.RegisterConfiguredArtifacts(
map[string]ConfiguredArtifactDefinition{
"session_recap": {Enabled: true, OutputPath: "artifacts/session_recap.md"},
"player_handout": {Enabled: false, OutputPath: "artifacts/player_handout.md"},
},
[]string{"player_handout"},
); err != nil {
t.Fatalf("RegisterConfiguredArtifacts() error = %v", err)
}
entries := catalog.ListConfigured()
if len(entries) != 2 {
t.Fatalf("ListConfigured() len = %d, want 2", len(entries))
}
if entries[0].ConfiguredKey != "player_handout" || entries[0].Executable != true {
t.Fatalf("entries[0] = %+v, want player_handout executable", entries[0])
}
if entries[1].ConfiguredKey != "session_recap" || entries[1].Executable != false {
t.Fatalf("entries[1] = %+v, want session_recap disabled by selection", entries[1])
}
}
func TestArtifactCatalogRejectsSelectedUnknownArtifact(t *testing.T) {
catalog := NewArtifactCatalog()
err := catalog.RegisterConfiguredArtifacts(
map[string]ConfiguredArtifactDefinition{
"session_recap": {Enabled: true, OutputPath: "artifacts/session_recap.md"},
},
[]string{"unknown"},
)
if err == nil {
t.Fatal("RegisterConfiguredArtifacts() error = nil, want non-nil")
}
}
func TestArtifactCatalogRejectsConfiguredSourceConflictAcrossRegistrations(t *testing.T) {
catalog := NewArtifactCatalog()
if err := catalog.RegisterConfiguredArtifacts(
map[string]ConfiguredArtifactDefinition{
"session_recap": {Enabled: true, OutputPath: "artifacts/session_recap.md"},
},
nil,
); err != nil {
t.Fatalf("RegisterConfiguredArtifacts() first call error = %v", err)
}
err := catalog.RegisterConfiguredArtifacts(
map[string]ConfiguredArtifactDefinition{
"session_recap": {Enabled: true, OutputPath: "artifacts/session_recap.md"},
},
nil,
)
if err == nil {
t.Fatal("RegisterConfiguredArtifacts() error = nil, want non-nil")
}
}
func TestArtifactCatalogMarkAvailableGenerated(t *testing.T) {
catalog := NewArtifactCatalog()
if err := catalog.RegisterConfiguredArtifacts(
map[string]ConfiguredArtifactDefinition{
"session_recap": {Enabled: true, OutputPath: "artifacts/session_recap.md"},
},
nil,
); err != nil {
t.Fatalf("RegisterConfiguredArtifacts() error = %v", err)
}
sourceID, _ := catalog.SourceIDForConfiguredKey("session_recap")
if err := catalog.MarkAvailableGenerated(sourceID, "/tmp/session_recap.md"); err != nil {
t.Fatalf("MarkAvailableGenerated() error = %v", err)
}
entry, _ := catalog.Lookup(sourceID)
if !entry.Available {
t.Fatalf("entry.Available = false, want true")
}
if entry.Provenance != ArtifactProvenanceGeneratedCurrentAnalyzeRun {
t.Fatalf("entry.Provenance = %q, want %q", entry.Provenance, ArtifactProvenanceGeneratedCurrentAnalyzeRun)
}
}
func TestArtifactCatalogMarkAvailableFromDisk(t *testing.T) {
catalog := NewArtifactCatalog()
if err := catalog.RegisterConfiguredArtifacts(
map[string]ConfiguredArtifactDefinition{
"session_recap": {Enabled: false, OutputPath: "artifacts/session_recap.md"},
},
nil,
); err != nil {
t.Fatalf("RegisterConfiguredArtifacts() error = %v", err)
}
sourceID, _ := catalog.SourceIDForConfiguredKey("session_recap")
if err := catalog.MarkAvailableFromDisk(sourceID, "/tmp/session_recap.md"); err != nil {
t.Fatalf("MarkAvailableFromDisk() error = %v", err)
}
entry, _ := catalog.Lookup(sourceID)
if !entry.Available {
t.Fatalf("entry.Available = false, want true")
}
if entry.Provenance != ArtifactProvenanceDisabledFromDisk {
t.Fatalf("entry.Provenance = %q, want %q", entry.Provenance, ArtifactProvenanceDisabledFromDisk)
}
}
func TestArtifactCatalogLookupPlannedButUnavailable(t *testing.T) {
catalog := NewArtifactCatalog()
if err := catalog.RegisterConfiguredArtifacts(
map[string]ConfiguredArtifactDefinition{
"session_recap": {Enabled: true, OutputPath: "artifacts/session_recap.md"},
},
nil,
); err != nil {
t.Fatalf("RegisterConfiguredArtifacts() error = %v", err)
}
sourceID, _ := catalog.SourceIDForConfiguredKey("session_recap")
entry, ok := catalog.Lookup(sourceID)
if !ok {
t.Fatalf("Lookup(%q) ok = false, want true", sourceID)
}
if !entry.Planned {
t.Fatalf("entry.Planned = false, want true")
}
if entry.Available {
t.Fatalf("entry.Available = true, want false")
}
}

View File

@@ -30,13 +30,13 @@ func NewLocalStore(workspaceRoot string) *LocalStore {
return &LocalStore{WorkspaceRoot: workspaceRoot}
}
// SessionPaths resolves canonical paths for a session workdir.
func (s *LocalStore) SessionPaths(sessionID string) SessionPaths {
return buildSessionPaths(s.WorkspaceRoot, sessionID)
// SessionPathsFor resolves canonical campaign-aware paths for a session workdir.
func (s *LocalStore) SessionPathsFor(campaign, sessionID string) SessionPaths {
return buildSessionPaths(s.WorkspaceRoot, campaign, sessionID)
}
// EnsureLayout creates and verifies the canonical session workdir directory layout.
func (s *LocalStore) EnsureLayout(sessionID string) (SessionPaths, error) {
// EnsureLayoutFor creates and verifies campaign-aware session layout.
func (s *LocalStore) EnsureLayoutFor(campaign, sessionID string) (SessionPaths, error) {
if strings.TrimSpace(s.WorkspaceRoot) == "" {
return SessionPaths{}, fmt.Errorf("workspace root is required")
}
@@ -44,7 +44,22 @@ func (s *LocalStore) EnsureLayout(sessionID string) (SessionPaths, error) {
return SessionPaths{}, fmt.Errorf("sessionID is required")
}
paths := s.SessionPaths(sessionID)
campaign = strings.TrimSpace(campaign)
if campaign == "" {
return SessionPaths{}, fmt.Errorf("campaign is required")
}
return s.ensureLayout(s.SessionPathsFor(campaign, sessionID))
}
func (s *LocalStore) ensureLayout(paths SessionPaths) (SessionPaths, error) {
if strings.TrimSpace(s.WorkspaceRoot) == "" {
return SessionPaths{}, fmt.Errorf("workspace root is required")
}
if strings.TrimSpace(paths.SessionID) == "" {
return SessionPaths{}, fmt.Errorf("sessionID is required")
}
dirs := []string{
paths.Root,
paths.InputsDir,
@@ -53,8 +68,11 @@ func (s *LocalStore) EnsureLayout(sessionID string) (SessionPaths, error) {
paths.TranscriptsRawDir,
paths.TranscriptsTrimmedDir,
paths.ArtifactsDir,
paths.ReportsDir,
paths.ConfigDir,
paths.LogsDir,
paths.CurrentDir,
paths.RunsDir,
}
for _, dir := range dirs {
@@ -66,13 +84,16 @@ func (s *LocalStore) EnsureLayout(sessionID string) (SessionPaths, error) {
return paths, nil
}
// CopyInput copies an input file into the session workdir under destRelativePath.
func (s *LocalStore) CopyInput(sessionID, srcPath, destRelativePath string) (Ref, error) {
paths, err := s.EnsureLayout(sessionID)
// CopyInputFor copies an input file into the campaign-aware session workdir under destRelativePath.
func (s *LocalStore) CopyInputFor(campaign, sessionID, srcPath, destRelativePath string) (Ref, error) {
paths, err := s.EnsureLayoutFor(campaign, sessionID)
if err != nil {
return Ref{}, err
}
return s.copyInputWithPaths(paths, sessionID, srcPath, destRelativePath)
}
func (s *LocalStore) copyInputWithPaths(paths SessionPaths, sessionID, srcPath, destRelativePath string) (Ref, error) {
destAbs, err := resolveInRoot(paths.Root, destRelativePath)
if err != nil {
return Ref{}, fmt.Errorf("copy input: %w", err)
@@ -173,13 +194,16 @@ func (s *LocalStore) Checksum(path string) (string, error) {
return digest, nil
}
// AcquireSessionLock acquires an exclusive lock file for a session workdir.
func (s *LocalStore) AcquireSessionLock(sessionID string) (*LockHandle, error) {
paths, err := s.EnsureLayout(sessionID)
// AcquireSessionLockFor acquires an exclusive lock file for a campaign/session workdir.
func (s *LocalStore) AcquireSessionLockFor(campaign, sessionID string) (*LockHandle, error) {
paths, err := s.EnsureLayoutFor(campaign, sessionID)
if err != nil {
return nil, err
}
return s.acquireSessionLockForPaths(paths)
}
func (s *LocalStore) acquireSessionLockForPaths(paths SessionPaths) (*LockHandle, error) {
f, err := os.OpenFile(paths.LockPath, os.O_CREATE|os.O_EXCL|os.O_WRONLY, 0o644)
if err != nil {
if errors.Is(err, os.ErrExist) {

View File

@@ -10,9 +10,9 @@ import (
func TestEnsureLayoutCreatesExpectedDirectories(t *testing.T) {
store := NewLocalStore(t.TempDir())
paths, err := store.EnsureLayout("session-1")
paths, err := store.EnsureLayoutFor("sample-campaign", "session-1")
if err != nil {
t.Fatalf("EnsureLayout() error = %v", err)
t.Fatalf("EnsureLayoutFor() error = %v", err)
}
checkDirExists(t, paths.Root)
@@ -22,8 +22,11 @@ func TestEnsureLayoutCreatesExpectedDirectories(t *testing.T) {
checkDirExists(t, paths.TranscriptsRawDir)
checkDirExists(t, paths.TranscriptsTrimmedDir)
checkDirExists(t, paths.ArtifactsDir)
checkDirExists(t, paths.ReportsDir)
checkDirExists(t, paths.ConfigDir)
checkDirExists(t, paths.LogsDir)
checkDirExists(t, paths.CurrentDir)
checkDirExists(t, paths.RunsDir)
if filepath.Base(paths.ManifestPath) != "manifest.json" {
t.Fatalf("ManifestPath = %q, want basename manifest.json", paths.ManifestPath)
@@ -33,6 +36,17 @@ func TestEnsureLayoutCreatesExpectedDirectories(t *testing.T) {
}
}
func TestEnsureLayoutForRequiresCampaign(t *testing.T) {
store := NewLocalStore(t.TempDir())
_, err := store.EnsureLayoutFor("", "session-1")
if err == nil {
t.Fatal("expected campaign-required error, got nil")
}
if !strings.Contains(err.Error(), "campaign is required") {
t.Fatalf("error = %v, want campaign-required error", err)
}
}
func TestChecksumCalculation(t *testing.T) {
store := NewLocalStore(t.TempDir())
path := filepath.Join(t.TempDir(), "sample.txt")
@@ -53,9 +67,9 @@ func TestChecksumCalculation(t *testing.T) {
func TestLockAcquireRelease(t *testing.T) {
store := NewLocalStore(t.TempDir())
lock, err := store.AcquireSessionLock("session-1")
lock, err := store.AcquireSessionLockFor("sample-campaign", "session-1")
if err != nil {
t.Fatalf("AcquireSessionLock() error = %v", err)
t.Fatalf("AcquireSessionLockFor() error = %v", err)
}
exists, err := store.Exists(lock.path)
@@ -81,15 +95,15 @@ func TestLockAcquireRelease(t *testing.T) {
func TestLockConflict(t *testing.T) {
store := NewLocalStore(t.TempDir())
lock1, err := store.AcquireSessionLock("session-1")
lock1, err := store.AcquireSessionLockFor("sample-campaign", "session-1")
if err != nil {
t.Fatalf("first AcquireSessionLock() error = %v", err)
t.Fatalf("first AcquireSessionLockFor() error = %v", err)
}
defer func() {
_ = store.ReleaseSessionLock(lock1)
}()
_, err = store.AcquireSessionLock("session-1")
_, err = store.AcquireSessionLockFor("sample-campaign", "session-1")
if err == nil {
t.Fatal("expected lock conflict error, got nil")
}
@@ -138,9 +152,9 @@ func TestCopyInput(t *testing.T) {
t.Fatalf("WriteFile() error = %v", err)
}
ref, err := store.CopyInput("session-1", srcPath, "inputs/speakers.yml")
ref, err := store.CopyInputFor("sample-campaign", "session-1", srcPath, "inputs/speakers.yml")
if err != nil {
t.Fatalf("CopyInput() error = %v", err)
t.Fatalf("CopyInputFor() error = %v", err)
}
if ref.Kind != "input" {

View File

@@ -9,6 +9,8 @@ import (
// SessionPaths contains canonical local paths for one session work directory.
type SessionPaths struct {
WorkspaceRoot string
CampaignID string
SessionID string
Root string
InputsDir string
AudioDir string
@@ -16,20 +18,43 @@ type SessionPaths struct {
TranscriptsRawDir string
TranscriptsTrimmedDir string
ArtifactsDir string
ReportsDir string
ConfigDir string
LogsDir string
CurrentDir string
RunsDir string
ManifestPath string
LockPath string
}
// SessionWorkDir returns the work directory for one session.
func SessionWorkDir(rootDir, sessionID string) string {
return filepath.Join(rootDir, config.PathWorkDirSegment, sessionID)
// SessionWorkDirForCampaign returns the canonical campaign-aware work directory for one session.
func SessionWorkDirForCampaign(rootDir, campaign, sessionID string) string {
return filepath.Join(rootDir, config.PathWorkDirSegment, campaign, sessionID)
}
// SessionRunWorkDir returns the campaign/session/run scoped local work directory.
func SessionRunWorkDir(rootDir, campaign, sessionID, runID string) string {
return filepath.Join(rootDir, config.PathWorkDirSegment, campaign, sessionID, runID)
// SessionManifestPathForCampaign returns the canonical session manifest path.
func SessionManifestPathForCampaign(rootDir, campaign, sessionID string) string {
return filepath.Join(SessionWorkDirForCampaign(rootDir, campaign, sessionID), config.PathManifestFile)
}
// SessionRunsDirForCampaign returns the canonical runs directory for one session.
func SessionRunsDirForCampaign(rootDir, campaign, sessionID string) string {
return filepath.Join(SessionWorkDirForCampaign(rootDir, campaign, sessionID), config.PathRunsDirSegment)
}
// SessionRunRootForCampaign returns the canonical run root under runs/{run_id}.
func SessionRunRootForCampaign(rootDir, campaign, sessionID, runID string) string {
return filepath.Join(SessionRunsDirForCampaign(rootDir, campaign, sessionID), runID)
}
// SessionRunManifestPathForCampaign returns the canonical run manifest path under runs/{run_id}/manifest.json.
func SessionRunManifestPathForCampaign(rootDir, campaign, sessionID, runID string) string {
return filepath.Join(SessionRunRootForCampaign(rootDir, campaign, sessionID, runID), config.PathManifestFile)
}
// SessionRunStageDirForCampaign returns the canonical stage directory under runs/{run_id}/{stage}.
func SessionRunStageDirForCampaign(rootDir, campaign, sessionID, runID, stageName string) string {
return filepath.Join(SessionRunRootForCampaign(rootDir, campaign, sessionID, runID), stageName)
}
// SessionSpoolAudioDir returns the campaign/session/run scoped local spool audio path.
@@ -37,10 +62,16 @@ func SessionSpoolAudioDir(spoolRoot, campaign, sessionID, runID string) string {
return filepath.Join(spoolRoot, campaign, sessionID, runID, config.PathAudioDirSegment)
}
func buildSessionPaths(workspaceRoot, sessionID string) SessionPaths {
root := SessionWorkDir(workspaceRoot, sessionID)
func buildSessionPaths(workspaceRoot, campaign, sessionID string) SessionPaths {
root := SessionWorkDirForCampaign(workspaceRoot, campaign, sessionID)
return buildSessionPathsFromRoot(workspaceRoot, campaign, sessionID, root)
}
func buildSessionPathsFromRoot(workspaceRoot, campaign, sessionID, root string) SessionPaths {
return SessionPaths{
WorkspaceRoot: workspaceRoot,
CampaignID: campaign,
SessionID: sessionID,
Root: root,
InputsDir: filepath.Join(root, config.PathInputsDirSegment),
AudioDir: filepath.Join(root, config.PathAudioDirSegment),
@@ -48,8 +79,11 @@ func buildSessionPaths(workspaceRoot, sessionID string) SessionPaths {
TranscriptsRawDir: filepath.Join(root, filepath.FromSlash(config.PathTranscriptsRaw)),
TranscriptsTrimmedDir: filepath.Join(root, filepath.FromSlash(config.PathTranscriptsTrimmed)),
ArtifactsDir: filepath.Join(root, config.PathArtifactsDirSegment),
ReportsDir: filepath.Join(root, config.PathReportsDirSegment),
ConfigDir: filepath.Join(root, config.PathConfigDirSegment),
LogsDir: filepath.Join(root, config.PathLogsDirSegment),
CurrentDir: filepath.Join(root, config.PathCurrentDirSegment),
RunsDir: filepath.Join(root, config.PathRunsDirSegment),
ManifestPath: filepath.Join(root, config.PathManifestFile),
LockPath: filepath.Join(root, config.PathLockFile),
}

View File

@@ -5,12 +5,47 @@ import (
"testing"
)
func TestSessionRunWorkDir(t *testing.T) {
func TestSessionWorkDirForCampaign(t *testing.T) {
root := "/tmp/workspace"
got := SessionRunWorkDir(root, "forsaken", "2026-04-19", "20260515T031522Z-a1b2c3d4")
want := filepath.Join(root, "work", "forsaken", "2026-04-19", "20260515T031522Z-a1b2c3d4")
got := SessionWorkDirForCampaign(root, "forsaken", "2026-04-19")
want := filepath.Join(root, "work", "forsaken", "2026-04-19")
if got != want {
t.Fatalf("SessionRunWorkDir() = %q, want %q", got, want)
t.Fatalf("SessionWorkDirForCampaign() = %q, want %q", got, want)
}
}
func TestSessionManifestPathForCampaign(t *testing.T) {
root := "/tmp/workspace"
got := SessionManifestPathForCampaign(root, "forsaken", "2026-04-19")
want := filepath.Join(root, "work", "forsaken", "2026-04-19", "manifest.json")
if got != want {
t.Fatalf("SessionManifestPathForCampaign() = %q, want %q", got, want)
}
}
func TestSessionRunRootAndStageDirForCampaign(t *testing.T) {
root := "/tmp/workspace"
runID := "20260515T031522Z-a1b2c3d4"
runRoot := SessionRunRootForCampaign(root, "forsaken", "2026-04-19", runID)
wantRoot := filepath.Join(root, "work", "forsaken", "2026-04-19", "runs", runID)
if runRoot != wantRoot {
t.Fatalf("SessionRunRootForCampaign() = %q, want %q", runRoot, wantRoot)
}
stageDir := SessionRunStageDirForCampaign(root, "forsaken", "2026-04-19", runID, "transcribe")
wantStage := filepath.Join(root, "work", "forsaken", "2026-04-19", "runs", runID, "transcribe")
if stageDir != wantStage {
t.Fatalf("SessionRunStageDirForCampaign() = %q, want %q", stageDir, wantStage)
}
}
func TestSessionRunManifestPathForCampaign(t *testing.T) {
root := "/tmp/workspace"
runID := "20260515T031522Z-a1b2c3d4"
got := SessionRunManifestPathForCampaign(root, "forsaken", "2026-04-19", runID)
want := filepath.Join(root, "work", "forsaken", "2026-04-19", "runs", runID, "manifest.json")
if got != want {
t.Fatalf("SessionRunManifestPathForCampaign() = %q, want %q", got, want)
}
}

View File

@@ -8,7 +8,7 @@ import (
func TestResolveSessionLocalPathForRead(t *testing.T) {
workspace := t.TempDir()
paths := buildSessionPaths(workspace, "s-1")
paths := buildSessionPaths(workspace, "sample-campaign", "s-1")
if err := os.MkdirAll(paths.TranscriptsRawDir, 0o755); err != nil {
t.Fatalf("MkdirAll() error = %v", err)
}
@@ -41,7 +41,7 @@ func TestResolveSessionLocalPathForReadRelativeWorkspaceRootQualifiedPath(t *tes
t.Fatalf("Rel() error = %v", err)
}
paths := buildSessionPaths(workspaceRel, "s-1")
paths := buildSessionPaths(workspaceRel, "sample-campaign", "s-1")
target := filepath.Join(paths.TranscriptsRawDir, "alice.json")
if err := os.MkdirAll(filepath.Dir(target), 0o755); err != nil {
t.Fatalf("MkdirAll() error = %v", err)
@@ -50,7 +50,7 @@ func TestResolveSessionLocalPathForReadRelativeWorkspaceRootQualifiedPath(t *tes
t.Fatalf("WriteFile() error = %v", err)
}
manifestPath := filepath.Join(workspaceRel, "work", "s-1", "transcripts", "raw", "alice.json")
manifestPath := filepath.Join(workspaceRel, "work", "sample-campaign", "s-1", "transcripts", "raw", "alice.json")
got := ResolveSessionLocalPathForRead(paths, manifestPath)
if got != filepath.Clean(manifestPath) {
t.Fatalf("resolution = %q, want %q", got, filepath.Clean(manifestPath))

View File

@@ -11,19 +11,19 @@ type S3Store struct {
Prefix string
}
// SessionPaths is not implemented for S3-backed storage.
func (s *S3Store) SessionPaths(_ string) SessionPaths {
// SessionPathsFor is not implemented for S3-backed storage.
func (s *S3Store) SessionPathsFor(_, _ string) SessionPaths {
return SessionPaths{}
}
// EnsureLayout returns a not-yet-implemented error in the scaffold.
func (s *S3Store) EnsureLayout(_ string) (SessionPaths, error) {
return SessionPaths{}, fmt.Errorf("artifacts s3 ensure layout: not yet implemented")
// EnsureLayoutFor returns a not-yet-implemented error in the scaffold.
func (s *S3Store) EnsureLayoutFor(_, _ string) (SessionPaths, error) {
return SessionPaths{}, fmt.Errorf("artifacts s3 ensure layout for campaign/session: not yet implemented")
}
// CopyInput returns a not-yet-implemented error in the scaffold.
func (s *S3Store) CopyInput(_, _, _ string) (Ref, error) {
return Ref{}, fmt.Errorf("artifacts s3 copy input: not yet implemented")
// CopyInputFor returns a not-yet-implemented error in the scaffold.
func (s *S3Store) CopyInputFor(_, _, _, _ string) (Ref, error) {
return Ref{}, fmt.Errorf("artifacts s3 copy input for campaign/session: not yet implemented")
}
// Exists returns a not-yet-implemented error in the scaffold.
@@ -46,9 +46,9 @@ func (s *S3Store) Checksum(_ string) (string, error) {
return "", fmt.Errorf("artifacts s3 checksum: not yet implemented")
}
// AcquireSessionLock returns a not-yet-implemented error in the scaffold.
func (s *S3Store) AcquireSessionLock(_ string) (*LockHandle, error) {
return nil, fmt.Errorf("artifacts s3 acquire lock: not yet implemented")
// AcquireSessionLockFor returns a not-yet-implemented error in the scaffold.
func (s *S3Store) AcquireSessionLockFor(_, _ string) (*LockHandle, error) {
return nil, fmt.Errorf("artifacts s3 acquire lock for campaign/session: not yet implemented")
}
// ReleaseSessionLock returns a not-yet-implemented error in the scaffold.

View File

@@ -15,13 +15,13 @@ type Ref struct {
// Store is the local artifact/workdir abstraction used by orchestration code.
type Store interface {
SessionPaths(sessionID string) SessionPaths
EnsureLayout(sessionID string) (SessionPaths, error)
CopyInput(sessionID, srcPath, destRelativePath string) (Ref, error)
SessionPathsFor(campaign, sessionID string) SessionPaths
EnsureLayoutFor(campaign, sessionID string) (SessionPaths, error)
CopyInputFor(campaign, sessionID, srcPath, destRelativePath string) (Ref, error)
Exists(path string) (bool, error)
ExistsRef(ref Ref) (bool, error)
WriteFileAtomic(path string, data []byte, perm os.FileMode) error
Checksum(path string) (string, error)
AcquireSessionLock(sessionID string) (*LockHandle, error)
AcquireSessionLockFor(campaign, sessionID string) (*LockHandle, error)
ReleaseSessionLock(lock *LockHandle) error
}

View File

@@ -79,8 +79,8 @@ type ArchiveConfig struct {
// ArchivePromotionRule configures one artifact promotion mapping.
type ArchivePromotionRule struct {
From string `yaml:"from"`
To string `yaml:"to"`
Source string `yaml:"source"`
Dest string `yaml:"dest"`
Required *bool `yaml:"required"`
}
@@ -176,6 +176,7 @@ type ScriptoriumConfig struct {
// ScriptoriumArtifactConfig configures one named output artifact workflow.
type ScriptoriumArtifactConfig struct {
Enabled bool `yaml:"enabled"`
DependsOn []string `yaml:"depends_on"`
RenderDebug *bool `yaml:"render_debug"`
PromptID string `yaml:"prompt_id"`
ProfileID string `yaml:"profile_id"`

View File

@@ -11,6 +11,7 @@ const (
DefaultS3AccessKeyIDEnv = "OBJECT_STORAGE_KEY_ID"
DefaultS3SecretAccessKeyEnv = "OBJECT_STORAGE_KEY"
DefaultStorageS3RootPrefix = "dnd"
DefaultWorkspaceRoot = "/var/lib/narratio"
DefaultSpoolRoot = "/var/spool/narratio"
DefaultWhisperXLanguage = "en"
@@ -29,8 +30,9 @@ const (
DefaultAuditaTimeout = "3h"
DefaultAuditaReport = true
DefaultScriptoriumBinary = "scriptorium"
DefaultScriptoriumTimeout = "10m"
DefaultScriptoriumBinary = "scriptorium"
DefaultScriptoriumTimeout = "10m"
DefaultScriptoriumArtifactOutputRoot = "artifacts"
DefaultTrimBoundsTimeout = "10m"
DefaultTrimSeriatimReport = false
@@ -49,8 +51,11 @@ const (
PathTranscriptsRaw = "transcripts/raw"
PathTranscriptsTrimmed = "transcripts/trimmed"
PathArtifactsDirSegment = "artifacts"
PathReportsDirSegment = "reports"
PathConfigDirSegment = "config"
PathLogsDirSegment = "logs"
PathCurrentDirSegment = "current"
PathRunsDirSegment = "runs"
PathManifestFile = "manifest.json"
PathLockFile = ".lock"
PathTranscriptMerged = "transcripts/merged.json"
@@ -69,8 +74,7 @@ const (
// DefaultArchivePromoteArtifacts defines the default archive promotion rules.
// Callers should copy this slice before mutating.
var DefaultArchivePromoteArtifacts = []ArchivePromotionRule{
{From: PathTranscriptTrimmed, To: PathTranscriptTrimmed},
{From: "artifacts/session_recap.md", To: "artifacts/session_recap.md"},
{Source: "narratio.transcript.trimmed", Dest: PathTranscriptTrimmed},
}
// DefaultPipelineConfigSearchPaths defines the default search order for

View File

@@ -152,6 +152,7 @@ func applyPipelineDefaults(cfg *PipelineConfig) {
if cfg == nil {
return
}
applyWorkspaceDefaults(&cfg.Workspace)
applyStorageDefaults(&cfg.Storage)
applySpoolDefaults(&cfg.Spool)
applyArchiveDefaults(&cfg.Archive)
@@ -166,6 +167,15 @@ func applyPipelineDefaults(cfg *PipelineConfig) {
applyScriptoriumDefaults(cfg.Scriptorium)
}
func applyWorkspaceDefaults(cfg *WorkspaceConfig) {
if cfg == nil {
return
}
if cfg.Root == "" {
cfg.Root = DefaultWorkspaceRoot
}
}
func applyStorageDefaults(cfg *StorageConfig) {
if cfg == nil {
return

View File

@@ -15,6 +15,7 @@ func TestLoadAndValidate(t *testing.T) {
wantLoadErr string
wantValidate string
checkDefault bool
wantRoot string
}{
{
name: "valid minimal config",
@@ -39,6 +40,7 @@ inputs:
glossary_file: ./glossary.yml
`,
checkDefault: true,
wantRoot: "/tmp/narratio",
},
{
name: "seriatim and audita sections can be omitted",
@@ -59,6 +61,26 @@ inputs:
glossary_file: ./glossary.yml
`,
checkDefault: true,
wantRoot: "/tmp/narratio",
},
{
name: "workspace root defaults when omitted",
pipelineYAML: `whisperx:
transcribe_url: https://transcription.ai.rakestrawhome.com/transcribe
analyzer:
timeout: 20m
notification:
timeout: 15s
`,
sessionYAML: `session_id: 2026-05-03
inputs:
audio_dir: ./audio
speakers_file: ./speakers.yml
autocorrect_file: ./autocorrect.yml
glossary_file: ./glossary.yml
`,
checkDefault: true,
wantRoot: DefaultWorkspaceRoot,
},
{
name: "unknown pipeline field fails",
@@ -713,6 +735,9 @@ inputs:
t.Fatalf("SessionPath = %q, want %q", cfg.SessionPath, sessionPath)
}
if tt.checkDefault {
if tt.wantRoot != "" && cfg.Pipeline.Workspace.Root != tt.wantRoot {
t.Fatalf("workspace.root = %q, want %q", cfg.Pipeline.Workspace.Root, tt.wantRoot)
}
if cfg.Pipeline.WhisperX.Language != "en" {
t.Fatalf("whisperx.language = %q, want %q", cfg.Pipeline.WhisperX.Language, "en")
}
@@ -866,15 +891,59 @@ func TestValidateMissingAudioSource(t *testing.T) {
}
func TestExamplesLoadAndValidate(t *testing.T) {
pipelinePath := filepath.Join("..", "..", "examples", "pipeline.minimal.yml")
sessionPath := filepath.Join("..", "..", "examples", "session.minimal.yml")
cfg, err := Load(pipelinePath, sessionPath)
if err != nil {
t.Fatalf("Load(examples) error = %v", err)
examplesDir := filepath.Join("..", "..", "examples")
tests := []struct {
name string
pipelineFile string
sessionFile string
sessionOpts SessionLoadOptions
}{
{
name: "minimal pipeline with local audio session",
pipelineFile: "pipeline.minimal.yml",
sessionFile: "session.local-audio.yml",
},
{
name: "production pipeline with s3 audio session",
pipelineFile: "pipeline.production.yml",
sessionFile: "session.s3-audio.yml",
},
{
name: "full annotated pipeline with local audio session",
pipelineFile: "pipeline.full.annotated.yml",
sessionFile: "session.local-audio.yml",
},
{
name: "template session renders with session_id option",
pipelineFile: "pipeline.minimal.yml",
sessionFile: "session.template.yml",
sessionOpts: SessionLoadOptions{
SessionID: "2026-05-03",
},
},
}
if err := Validate(cfg); err != nil {
t.Fatalf("Validate(examples) error = %v", err)
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
pipelinePath := filepath.Join(examplesDir, tt.pipelineFile)
sessionPath := filepath.Join(examplesDir, tt.sessionFile)
var (
cfg *Config
err error
)
if strings.TrimSpace(tt.sessionOpts.SessionID) == "" {
cfg, err = Load(pipelinePath, sessionPath)
} else {
cfg, err = LoadWithSessionOptions(pipelinePath, sessionPath, tt.sessionOpts)
}
if err != nil {
t.Fatalf("load example config error = %v", err)
}
if err := Validate(cfg); err != nil {
t.Fatalf("validate example config error = %v", err)
}
})
}
}

View File

@@ -24,7 +24,7 @@ func TestScriptoriumLoadAndValidate(t *testing.T) {
output_path: artifacts/session_recap.md
inputs:
transcript:
source: processed_transcript
source: narratio.transcript.polished
required: true
vars:
session_id: true
@@ -104,7 +104,7 @@ func TestScriptoriumLoadAndValidate(t *testing.T) {
output_path: artifacts/session_recap.md
inputs:
transcript:
source: processed_transcript
source: narratio.transcript.polished
required: true
previous_recap:
source: previous_session_artifact
@@ -116,6 +116,37 @@ func TestScriptoriumLoadAndValidate(t *testing.T) {
output_kind: session_recap
`,
},
{
name: "canonical artifact source is accepted",
scriptoriumYAML: `scriptorium:
binary: scriptorium
artifacts:
session_recap:
enabled: true
prompt_id: dnd.session_recap
output_path: artifacts/session_recap.md
inputs:
transcript:
source: narratio.transcript.trimmed
required: true
`,
},
{
name: "unknown artifact source fails validation",
scriptoriumYAML: `scriptorium:
binary: scriptorium
artifacts:
session_recap:
enabled: true
prompt_id: dnd.session_recap
output_path: artifacts/session_recap.md
inputs:
transcript:
source: narratio.unknown
required: true
`,
wantValidateErr: `pipeline.scriptorium.artifacts.session_recap.inputs.transcript.source "narratio.unknown" is unsupported`,
},
{
name: "artifact render_debug override is accepted",
scriptoriumYAML: `scriptorium:
@@ -129,7 +160,7 @@ func TestScriptoriumLoadAndValidate(t *testing.T) {
output_path: artifacts/session_recap.md
inputs:
transcript:
source: processed_transcript
source: narratio.transcript.polished
required: true
`,
assert: func(t *testing.T, cfg *Config) {
@@ -151,7 +182,7 @@ func TestScriptoriumLoadAndValidate(t *testing.T) {
output_path: artifacts/session_recap.md
inputs:
transcript:
source: processed_transcript
source: narratio.transcript.polished
required: true
player_summary:
enabled: true
@@ -161,7 +192,7 @@ func TestScriptoriumLoadAndValidate(t *testing.T) {
timeout: 3m
inputs:
transcript:
source: processed_transcript
source: narratio.transcript.polished
required: true
`,
assert: func(t *testing.T, cfg *Config) {
@@ -174,6 +205,195 @@ func TestScriptoriumLoadAndValidate(t *testing.T) {
}
},
},
{
name: "valid artifact dependency is accepted",
scriptoriumYAML: `scriptorium:
binary: scriptorium
artifacts:
session_recap:
enabled: true
prompt_id: dnd.session_recap
output_path: artifacts/session_recap.md
inputs:
transcript:
source: narratio.transcript.trimmed
required: true
player_handout:
enabled: true
depends_on:
- session_recap
prompt_id: dnd.player_handout
output_path: artifacts/player_handout.md
inputs:
recap:
source: narratio.artifact.session_recap
required: true
`,
},
{
name: "valid dependency on disabled artifact with output path is accepted",
scriptoriumYAML: `scriptorium:
binary: scriptorium
artifacts:
session_recap:
enabled: false
output_path: artifacts/session_recap.md
player_handout:
enabled: true
depends_on:
- session_recap
prompt_id: dnd.player_handout
output_path: artifacts/player_handout.md
inputs:
recap:
source: narratio.artifact.session_recap
required: true
`,
},
{
name: "invalid artifact name fails validation",
scriptoriumYAML: `scriptorium:
binary: scriptorium
artifacts:
SessionRecap:
enabled: true
prompt_id: dnd.session_recap
output_path: artifacts/session_recap.md
`,
wantValidateErr: "pipeline.scriptorium.artifacts keys must match ^[a-z][a-z0-9_]*$",
},
{
name: "artifact output path outside artifacts root fails validation",
scriptoriumYAML: `scriptorium:
binary: scriptorium
artifacts:
session_recap:
enabled: true
prompt_id: dnd.session_recap
output_path: transcripts/session_recap.md
`,
wantValidateErr: "pipeline.scriptorium.artifacts.session_recap.output_path must be under artifacts/",
},
{
name: "missing depends_on for artifact source fails validation",
scriptoriumYAML: `scriptorium:
binary: scriptorium
artifacts:
session_recap:
enabled: true
prompt_id: dnd.session_recap
output_path: artifacts/session_recap.md
inputs:
transcript:
source: narratio.transcript.polished
required: true
player_handout:
enabled: true
prompt_id: dnd.player_handout
output_path: artifacts/player_handout.md
inputs:
recap:
source: narratio.artifact.session_recap
required: true
`,
wantValidateErr: `pipeline.scriptorium.artifacts.player_handout.inputs.recap.source "narratio.artifact.session_recap" requires depends_on entry "session_recap"`,
},
{
name: "dependency on unknown artifact fails validation",
scriptoriumYAML: `scriptorium:
binary: scriptorium
artifacts:
player_handout:
enabled: true
depends_on:
- session_recap
prompt_id: dnd.player_handout
output_path: artifacts/player_handout.md
inputs:
recap:
source: narratio.artifact.session_recap
required: true
`,
wantValidateErr: `pipeline.scriptorium.artifacts.player_handout.depends_on[0] "session_recap" is not a configured artifact key`,
},
{
name: "self dependency fails validation",
scriptoriumYAML: `scriptorium:
binary: scriptorium
artifacts:
session_recap:
enabled: true
depends_on:
- session_recap
prompt_id: dnd.session_recap
output_path: artifacts/session_recap.md
`,
wantValidateErr: "pipeline.scriptorium.artifacts.session_recap.depends_on must not include itself",
},
{
name: "enabled dependency cycle fails validation",
scriptoriumYAML: `scriptorium:
binary: scriptorium
artifacts:
artifact_a:
enabled: true
depends_on:
- artifact_b
prompt_id: dnd.a
output_path: artifacts/a.md
inputs:
b:
source: narratio.artifact.artifact_b
required: true
artifact_b:
enabled: true
depends_on:
- artifact_a
prompt_id: dnd.b
output_path: artifacts/b.md
inputs:
a:
source: narratio.artifact.artifact_a
required: true
`,
wantValidateErr: "pipeline.scriptorium.artifacts enabled dependencies must not contain cycles",
},
{
name: "artifact source typo fails validation",
scriptoriumYAML: `scriptorium:
binary: scriptorium
artifacts:
player_handout:
enabled: true
prompt_id: dnd.player_handout
output_path: artifacts/player_handout.md
inputs:
recap:
source: narratio.artifact.session-recap
required: true
`,
wantValidateErr: `pipeline.scriptorium.artifacts.player_handout.inputs.recap.source "narratio.artifact.session-recap" is unsupported`,
},
{
name: "referenced disabled artifact missing output path fails validation",
scriptoriumYAML: `scriptorium:
binary: scriptorium
artifacts:
session_recap:
enabled: false
player_handout:
enabled: true
depends_on:
- session_recap
prompt_id: dnd.player_handout
output_path: artifacts/player_handout.md
inputs:
recap:
source: narratio.artifact.session_recap
required: true
`,
wantValidateErr: "pipeline.scriptorium.artifacts.session_recap.output_path is required when artifact is referenced",
},
}
for _, tt := range tests {
@@ -227,6 +447,7 @@ audita:
`
const testSessionBaseYAML = `session_id: 2026-05-03
campaign: test-campaign
inputs:
audio_dir: ./audio
speakers_file: ./speakers.yml

View File

@@ -136,40 +136,86 @@ func TestSpoolAndArchiveDefaults(t *testing.T) {
if cfg.Pipeline.Archive.UploadRun == nil || !*cfg.Pipeline.Archive.UploadRun {
t.Fatalf("archive.upload_run = %#v, want true", cfg.Pipeline.Archive.UploadRun)
}
if len(cfg.Pipeline.Archive.PromoteArtifacts) != 2 {
t.Fatalf("archive.promote_artifacts len = %d, want 2 defaults", len(cfg.Pipeline.Archive.PromoteArtifacts))
if len(cfg.Pipeline.Archive.PromoteArtifacts) != 1 {
t.Fatalf("archive.promote_artifacts len = %d, want 1 default", len(cfg.Pipeline.Archive.PromoteArtifacts))
}
for i, item := range cfg.Pipeline.Archive.PromoteArtifacts {
if item.Required == nil || !*item.Required {
t.Fatalf("archive.promote_artifacts[%d].required = %#v, want true", i, item.Required)
}
item := cfg.Pipeline.Archive.PromoteArtifacts[0]
if item.Required == nil || !*item.Required {
t.Fatalf("archive.promote_artifacts[0].required = %#v, want true", item.Required)
}
if item.Source != "narratio.transcript.trimmed" {
t.Fatalf("archive.promote_artifacts[0].source = %q, want narratio.transcript.trimmed", item.Source)
}
if item.Dest != "transcripts/trimmed.json" {
t.Fatalf("archive.promote_artifacts[0].dest = %q, want transcripts/trimmed.json", item.Dest)
}
}
func TestArchivePromotionPathValidation(t *testing.T) {
func TestArchivePromotionValidation(t *testing.T) {
tests := []struct {
name string
ruleYML string
wantErr string
}{
{
name: "absolute from path rejected",
name: "absolute dest path rejected",
ruleYML: `archive:
promote_artifacts:
- from: "/transcripts/trimmed.json"
to: "transcripts/trimmed.json"
- source: "narratio.transcript.trimmed"
dest: "/transcripts/trimmed.json"
`,
wantErr: "must be a relative path",
},
{
name: "traversal to path rejected",
name: "traversal dest path rejected",
ruleYML: `archive:
promote_artifacts:
- from: "transcripts/trimmed.json"
to: "../trimmed.json"
- source: "narratio.transcript.trimmed"
dest: "../trimmed.json"
`,
wantErr: "must not contain path traversal",
},
{
name: "invalid source rejected",
ruleYML: `archive:
promote_artifacts:
- source: "narratio.unknown"
dest: "transcripts/trimmed.json"
`,
wantErr: "source \"narratio.unknown\" is unsupported",
},
{
name: "duplicate destination rejected",
ruleYML: `archive:
promote_artifacts:
- source: "narratio.transcript.trimmed"
dest: "artifacts/shared.md"
- source: "narratio.transcript.full"
dest: "artifacts/shared.md"
`,
wantErr: "duplicates another archive promotion destination",
},
{
name: "configured source requires configured artifact key",
ruleYML: `archive:
promote_artifacts:
- source: "narratio.artifact.session_recap"
dest: "artifacts/session_recap.md"
`,
wantErr: "configured artifact \"session_recap\" is not defined in pipeline.scriptorium.artifacts",
},
{
name: "configured source without output path fails when dest omitted",
ruleYML: `scriptorium:
artifacts:
session_recap:
enabled: false
archive:
promote_artifacts:
- source: "narratio.artifact.session_recap"
`,
wantErr: "destination cannot be derived",
},
}
for _, tt := range tests {
@@ -189,6 +235,72 @@ func TestArchivePromotionPathValidation(t *testing.T) {
}
}
func TestArchivePromotionDerivesDestinationWhenOmitted(t *testing.T) {
tests := []struct {
name string
pipelineYML string
wantDest string
}{
{
name: "built in source derives canonical destination",
pipelineYML: testPipelineBaseYAML + `
archive:
promote_artifacts:
- source: narratio.transcript.full
`,
wantDest: "transcripts/normalized.json",
},
{
name: "configured source derives configured output path",
pipelineYML: testPipelineBaseYAML + `
scriptorium:
artifacts:
session_recap:
enabled: true
prompt_id: dnd.session_recap
output_path: artifacts/session_recap.md
archive:
promote_artifacts:
- source: narratio.artifact.session_recap
`,
wantDest: "artifacts/session_recap.md",
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
pipelinePath, sessionPath := writeConfigFiles(t, tt.pipelineYML, testSessionBaseYAML)
cfg, err := Load(pipelinePath, sessionPath)
if err != nil {
t.Fatalf("Load() error = %v", err)
}
if err := Validate(cfg); err != nil {
t.Fatalf("Validate() error = %v", err)
}
if len(cfg.Pipeline.Archive.PromoteArtifacts) != 1 {
t.Fatalf("archive.promote_artifacts len = %d, want 1", len(cfg.Pipeline.Archive.PromoteArtifacts))
}
if cfg.Pipeline.Archive.PromoteArtifacts[0].Dest != tt.wantDest {
t.Fatalf("archive.promote_artifacts[0].dest = %q, want %q", cfg.Pipeline.Archive.PromoteArtifacts[0].Dest, tt.wantDest)
}
})
}
}
func TestArchiveLegacyFromToFailsStrictDecode(t *testing.T) {
pipelineYAML := testPipelineBaseYAML + `
archive:
promote_artifacts:
- from: transcripts/trimmed.json
to: transcripts/trimmed.json
`
pipelinePath, sessionPath := writeConfigFiles(t, pipelineYAML, testSessionBaseYAML)
_, err := Load(pipelinePath, sessionPath)
if err == nil || !strings.Contains(err.Error(), "strict decode failed") {
t.Fatalf("Load() error = %v, want strict decode failed", err)
}
}
func TestSessionAudioS3Validation(t *testing.T) {
tests := []struct {
name string

View File

@@ -47,7 +47,7 @@ func validatePipeline(cfg *PipelineConfig) error {
if err := validateSpool(cfg.Spool); err != nil {
return err
}
if err := validateArchive(cfg.Archive); err != nil {
if err := validateArchive(cfg.Archive, cfg.Scriptorium); err != nil {
return err
}
if err := validateWhisperX(cfg.WhisperX); err != nil {
@@ -104,28 +104,91 @@ func validateSpool(cfg SpoolConfig) error {
return nil
}
func validateArchive(cfg *ArchiveConfig) error {
func validateArchive(cfg *ArchiveConfig, scriptorium *ScriptoriumConfig) error {
if cfg == nil {
return nil
}
seenDest := map[string]struct{}{}
for i, item := range cfg.PromoteArtifacts {
prefix := fmt.Sprintf("pipeline.archive.promote_artifacts[%d]", i)
if strings.TrimSpace(item.From) == "" {
return fmt.Errorf("%s.from is required", prefix)
source := strings.TrimSpace(item.Source)
if source == "" {
return fmt.Errorf("%s.source is required", prefix)
}
if strings.TrimSpace(item.To) == "" {
return fmt.Errorf("%s.to is required", prefix)
if _, err := archiveSourceKnown(source, scriptorium); err != nil {
return fmt.Errorf("%s.source %q is unsupported: %w", prefix, item.Source, err)
}
if err := validateRelativeSafePath(prefix+".from", item.From); err != nil {
dest := strings.TrimSpace(item.Dest)
if dest == "" {
derivedDest, err := deriveArchivePromotionDest(source, scriptorium)
if err != nil {
return fmt.Errorf("%s.dest is required when destination cannot be derived from %q: %w", prefix, source, err)
}
dest = derivedDest
cfg.PromoteArtifacts[i].Dest = derivedDest
}
if err := validateRelativeSafePath(prefix+".dest", dest); err != nil {
return err
}
if err := validateRelativeSafePath(prefix+".to", item.To); err != nil {
return err
normalizedDest := filepath.ToSlash(filepath.Clean(dest))
if _, ok := seenDest[normalizedDest]; ok {
return fmt.Errorf("%s.dest %q duplicates another archive promotion destination", prefix, dest)
}
seenDest[normalizedDest] = struct{}{}
}
return nil
}
func archiveSourceKnown(source string, scriptorium *ScriptoriumConfig) (string, error) {
trimmed := strings.TrimSpace(source)
switch trimmed {
case "narratio.transcript.merged",
"narratio.transcript.polished",
"narratio.transcript.full",
"narratio.transcript.trimmed",
"narratio.bounds.session":
return "", nil
}
matches := narratioArtifactSourceRE.FindStringSubmatch(trimmed)
if len(matches) != 2 {
return "", fmt.Errorf("must be a built-in source id or narratio.artifact.<name>")
}
artifactKey := matches[1]
if scriptorium == nil || len(scriptorium.Artifacts) == 0 {
return "", fmt.Errorf("configured artifact %q is not defined in pipeline.scriptorium.artifacts", artifactKey)
}
if _, ok := scriptorium.Artifacts[artifactKey]; !ok {
return "", fmt.Errorf("configured artifact %q is not defined in pipeline.scriptorium.artifacts", artifactKey)
}
return artifactKey, nil
}
func deriveArchivePromotionDest(source string, scriptorium *ScriptoriumConfig) (string, error) {
trimmed := strings.TrimSpace(source)
switch trimmed {
case "narratio.transcript.merged":
return PathTranscriptMerged, nil
case "narratio.transcript.polished":
return PathTranscriptProcessed, nil
case "narratio.transcript.full":
return PathTranscriptNormalized, nil
case "narratio.transcript.trimmed":
return PathTranscriptTrimmed, nil
case "narratio.bounds.session":
return filepath.ToSlash(filepath.Join(PathArtifactsDirSegment, "session_bounds.json")), nil
}
artifactKey, err := archiveSourceKnown(trimmed, scriptorium)
if err != nil {
return "", err
}
artifactCfg := scriptorium.Artifacts[artifactKey]
outputPath := strings.TrimSpace(artifactCfg.OutputPath)
if outputPath == "" {
return "", fmt.Errorf("pipeline.scriptorium.artifacts.%s.output_path is empty", artifactKey)
}
return outputPath, nil
}
func validateSecrets(cfg *SecretsConfig) error {
if cfg == nil {
return nil
@@ -324,28 +387,78 @@ func validateScriptorium(cfg *ScriptoriumConfig) error {
return err
}
for artifactName, artifactCfg := range cfg.Artifacts {
trimmedArtifactName := strings.TrimSpace(artifactName)
if trimmedArtifactName == "" {
return fmt.Errorf("pipeline.scriptorium.artifacts keys must be non-empty")
configuredArtifacts := make(map[string]struct{}, len(cfg.Artifacts))
referencedArtifacts := make(map[string]struct{})
for artifactName := range cfg.Artifacts {
if !scriptoriumArtifactKeyRE.MatchString(strings.TrimSpace(artifactName)) {
return fmt.Errorf("pipeline.scriptorium.artifacts keys must match ^[a-z][a-z0-9_]*$")
}
configuredArtifacts[artifactName] = struct{}{}
}
for artifactName, artifactCfg := range cfg.Artifacts {
if artifactCfg.Enabled && strings.TrimSpace(artifactCfg.PromptID) == "" {
return fmt.Errorf("pipeline.scriptorium.artifacts.%s.prompt_id is required when enabled", artifactName)
}
if artifactCfg.Enabled && strings.TrimSpace(artifactCfg.OutputPath) == "" {
return fmt.Errorf("pipeline.scriptorium.artifacts.%s.output_path is required when enabled", artifactName)
}
if strings.TrimSpace(artifactCfg.OutputPath) != "" {
pathField := "pipeline.scriptorium.artifacts." + artifactName + ".output_path"
if err := validateRelativeSafePath(pathField, artifactCfg.OutputPath); err != nil {
return err
}
if err := validatePathWithinRoot(pathField, artifactCfg.OutputPath, DefaultScriptoriumArtifactOutputRoot); err != nil {
return err
}
}
if err := validateDuration("pipeline.scriptorium.artifacts."+artifactName+".timeout", artifactCfg.Timeout); err != nil {
return err
}
depSet := make(map[string]struct{}, len(artifactCfg.DependsOn))
for i, depName := range artifactCfg.DependsOn {
trimmedDep := strings.TrimSpace(depName)
field := fmt.Sprintf("pipeline.scriptorium.artifacts.%s.depends_on[%d]", artifactName, i)
if trimmedDep == "" {
return fmt.Errorf("%s must be non-empty", field)
}
if _, ok := configuredArtifacts[trimmedDep]; !ok {
return fmt.Errorf("%s %q is not a configured artifact key", field, depName)
}
if trimmedDep == artifactName {
return fmt.Errorf("pipeline.scriptorium.artifacts.%s.depends_on must not include itself", artifactName)
}
depSet[trimmedDep] = struct{}{}
referencedArtifacts[trimmedDep] = struct{}{}
}
for inputName, inputCfg := range artifactCfg.Inputs {
trimmedInputName := strings.TrimSpace(inputName)
if trimmedInputName == "" {
return fmt.Errorf("pipeline.scriptorium.artifacts.%s.inputs keys must be non-empty", artifactName)
}
if strings.TrimSpace(inputCfg.Source) == "" {
source := strings.TrimSpace(inputCfg.Source)
if source == "" {
return fmt.Errorf("pipeline.scriptorium.artifacts.%s.inputs.%s.source is required", artifactName, inputName)
}
referencedArtifact, err := validateScriptoriumInputSource(artifactName, inputName, source, configuredArtifacts)
if err != nil {
return err
}
if referencedArtifact != "" {
if _, ok := depSet[referencedArtifact]; !ok {
return fmt.Errorf(
"pipeline.scriptorium.artifacts.%s.inputs.%s.source %q requires depends_on entry %q",
artifactName,
inputName,
source,
referencedArtifact,
)
}
referencedArtifacts[referencedArtifact] = struct{}{}
}
}
for varName, varValue := range artifactCfg.Vars {
if strings.TrimSpace(varName) == "" {
@@ -359,6 +472,17 @@ func validateScriptorium(cfg *ScriptoriumConfig) error {
}
}
for artifactName := range referencedArtifacts {
artifactCfg := cfg.Artifacts[artifactName]
if strings.TrimSpace(artifactCfg.OutputPath) == "" {
return fmt.Errorf("pipeline.scriptorium.artifacts.%s.output_path is required when artifact is referenced", artifactName)
}
}
if err := validateEnabledArtifactDependencyCycles(cfg.Artifacts); err != nil {
return err
}
return nil
}
@@ -437,6 +561,53 @@ func archiveUploadConfiguredForS3(pipeline *PipelineConfig) bool {
var windowsAbsPathRE = regexp.MustCompile(`^[A-Za-z]:[\\/].*`)
var envVarNameRE = regexp.MustCompile(`^[A-Za-z_][A-Za-z0-9_]*$`)
var scriptoriumArtifactKeyRE = regexp.MustCompile(`^[a-z][a-z0-9_]*$`)
var narratioArtifactSourceRE = regexp.MustCompile(`^narratio\.artifact\.([a-z][a-z0-9_]*)$`)
func validateScriptoriumInputSource(artifactName, inputName, source string, configuredArtifacts map[string]struct{}) (string, error) {
if isStaticSupportedScriptoriumInputSource(source) {
return "", nil
}
matches := narratioArtifactSourceRE.FindStringSubmatch(source)
if len(matches) != 2 {
return "", fmt.Errorf(
"pipeline.scriptorium.artifacts.%s.inputs.%s.source %q is unsupported",
artifactName,
inputName,
source,
)
}
referenced := matches[1]
if _, ok := configuredArtifacts[referenced]; !ok {
return "", fmt.Errorf(
"pipeline.scriptorium.artifacts.%s.inputs.%s.source %q references unknown artifact %q",
artifactName,
inputName,
source,
referenced,
)
}
return referenced, nil
}
func isStaticSupportedScriptoriumInputSource(source string) bool {
switch strings.TrimSpace(source) {
case "previous_session_artifact":
return true
case "narratio.transcript.merged":
return true
case "narratio.transcript.polished":
return true
case "narratio.transcript.full":
return true
case "narratio.transcript.trimmed":
return true
case "narratio.bounds.session":
return true
default:
return false
}
}
func validateEnvVarNameField(fieldName, value string) error {
trimmed := strings.TrimSpace(value)
@@ -467,6 +638,73 @@ func validateRelativeSafePath(fieldName, value string) error {
return nil
}
func validatePathWithinRoot(fieldName, value, root string) error {
normalizedValue := filepath.ToSlash(filepath.Clean(strings.TrimSpace(value)))
normalizedRoot := filepath.ToSlash(filepath.Clean(strings.TrimSpace(root)))
if normalizedValue == normalizedRoot {
return nil
}
if strings.HasPrefix(normalizedValue, normalizedRoot+"/") {
return nil
}
return fmt.Errorf("%s must be under %s/", fieldName, normalizedRoot)
}
func validateEnabledArtifactDependencyCycles(artifacts map[string]ScriptoriumArtifactConfig) error {
if len(artifacts) == 0 {
return nil
}
enabled := make(map[string]struct{}, len(artifacts))
graph := make(map[string][]string, len(artifacts))
for name, cfg := range artifacts {
if !cfg.Enabled {
continue
}
enabled[name] = struct{}{}
}
for name, cfg := range artifacts {
if !cfg.Enabled {
continue
}
for _, dep := range cfg.DependsOn {
trimmedDep := strings.TrimSpace(dep)
if _, ok := enabled[trimmedDep]; ok {
graph[name] = append(graph[name], trimmedDep)
}
}
}
visiting := make(map[string]bool, len(enabled))
visited := make(map[string]bool, len(enabled))
var visit func(node string) error
visit = func(node string) error {
if visiting[node] {
return fmt.Errorf("pipeline.scriptorium.artifacts enabled dependencies must not contain cycles")
}
if visited[node] {
return nil
}
visiting[node] = true
for _, dep := range graph[node] {
if err := visit(dep); err != nil {
return err
}
}
visiting[node] = false
visited[node] = true
return nil
}
for node := range enabled {
if err := visit(node); err != nil {
return err
}
}
return nil
}
func validateDuration(fieldName, value string) error {
trimmed := strings.TrimSpace(value)
if trimmed == "" {

View File

@@ -28,9 +28,12 @@ type InputRecord struct {
// ArtifactRecord captures one produced artifact and optional remote metadata.
type ArtifactRecord struct {
Kind string `json:"kind"`
SourceID string `json:"source_id,omitempty"`
LocalPath string `json:"local_path"`
RemoteKey string `json:"remote_key,omitempty"`
Checksum string `json:"checksum,omitempty"`
// ProducerRunID identifies the run that produced this durable artifact.
ProducerRunID string `json:"producer_run_id,omitempty"`
RemoteKey string `json:"remote_key,omitempty"`
Checksum string `json:"checksum,omitempty"`
}
// StageRecord tracks lifecycle and provenance for one pipeline stage.
@@ -120,6 +123,15 @@ func (m *Manifest) MarkStageSkipped(name string, at time.Time, reason string) {
m.UpdatedAt = at
}
// MarkStageStale marks a stage as stale so it is not skipped as idempotently complete.
func (m *Manifest) MarkStageStale(name string, at time.Time, reason string) {
s := m.ensureStage(name, at)
s.Status = StatusStale
s.Error = &ErrorRecord{Message: strings.TrimSpace(reason), Code: "stale", At: timePtr(at)}
s.UpdatedAt = at
m.UpdatedAt = at
}
func (m *Manifest) ensureStage(name string, at time.Time) *StageRecord {
if m.Stages == nil {
m.Stages = map[string]*StageRecord{}

View File

@@ -0,0 +1,164 @@
package manifest
import (
"strings"
"time"
)
type RunManifestStatus string
const (
RunManifestStatusRunning RunManifestStatus = "running"
RunManifestStatusSucceeded RunManifestStatus = "succeeded"
RunManifestStatusFailed RunManifestStatus = "failed"
)
type RunStageAction string
const (
RunStageActionRun RunStageAction = "run"
RunStageActionSkip RunStageAction = "skip"
)
// RunStageRecord tracks lifecycle and provenance for one stage within a single invocation.
type RunStageRecord struct {
Name string `json:"name"`
Action RunStageAction `json:"action"`
Status StageStatus `json:"status"`
CreatedAt time.Time `json:"created_at"`
UpdatedAt time.Time `json:"updated_at"`
StartedAt *time.Time `json:"started_at,omitempty"`
CompletedAt *time.Time `json:"completed_at,omitempty"`
Outputs []ArtifactRecord `json:"outputs,omitempty"`
Logs []string `json:"logs,omitempty"`
GeneratedConfigs []string `json:"generated_configs,omitempty"`
Error *ErrorRecord `json:"error,omitempty"`
Metadata map[string]any `json:"metadata,omitempty"`
}
// RunManifest is the invocation-scoped execution record under runs/{run_id}/manifest.json.
type RunManifest struct {
SessionID string `json:"session_id"`
Campaign string `json:"campaign,omitempty"`
RunID string `json:"run_id"`
Force bool `json:"force"`
RequestedStages []string `json:"requested_stages,omitempty"`
SessionManifestPath string `json:"session_manifest_path,omitempty"`
LocalWorkDir string `json:"local_workdir,omitempty"`
LocalSpoolDir string `json:"local_spool_dir,omitempty"`
S3Bucket string `json:"s3_bucket,omitempty"`
S3SessionPrefix string `json:"s3_session_prefix,omitempty"`
S3RunPrefix string `json:"s3_run_prefix,omitempty"`
CreatedAt time.Time `json:"created_at"`
UpdatedAt time.Time `json:"updated_at"`
StartedAt *time.Time `json:"started_at,omitempty"`
CompletedAt *time.Time `json:"completed_at,omitempty"`
Status RunManifestStatus `json:"status"`
LastError *ErrorRecord `json:"last_error,omitempty"`
Stages map[string]*RunStageRecord `json:"stages"`
Metadata map[string]any `json:"metadata,omitempty"`
}
// NewRun constructs a new run manifest with deterministic timestamps.
func NewRun(sessionID, campaign, runID string, force bool, requestedStages []string, now time.Time) *RunManifest {
return &RunManifest{
SessionID: strings.TrimSpace(sessionID),
Campaign: strings.TrimSpace(campaign),
RunID: strings.TrimSpace(runID),
Force: force,
RequestedStages: append([]string(nil), requestedStages...),
CreatedAt: now,
UpdatedAt: now,
StartedAt: timePtr(now),
Status: RunManifestStatusRunning,
Stages: map[string]*RunStageRecord{},
}
}
func (m *RunManifest) SetStageAction(name string, action RunStageAction, at time.Time) {
s := m.ensureStage(name, at)
s.Action = action
s.UpdatedAt = at
m.UpdatedAt = at
}
func (m *RunManifest) MarkStageRunning(name string, at time.Time) {
s := m.ensureStage(name, at)
s.Status = StatusRunning
s.StartedAt = timePtr(at)
s.CompletedAt = nil
s.Error = nil
s.UpdatedAt = at
m.UpdatedAt = at
}
func (m *RunManifest) MarkStageSucceeded(name string, at time.Time, outputs []ArtifactRecord) {
s := m.ensureStage(name, at)
s.Status = StatusSucceeded
s.CompletedAt = timePtr(at)
s.Error = nil
s.Outputs = append([]ArtifactRecord(nil), outputs...)
s.UpdatedAt = at
m.UpdatedAt = at
}
func (m *RunManifest) MarkStageFailed(name string, at time.Time, message string) {
s := m.ensureStage(name, at)
s.Status = StatusFailed
s.CompletedAt = timePtr(at)
s.Error = &ErrorRecord{Message: strings.TrimSpace(message), At: timePtr(at)}
s.UpdatedAt = at
m.LastError = &ErrorRecord{Message: strings.TrimSpace(message), At: timePtr(at)}
m.UpdatedAt = at
m.Status = RunManifestStatusFailed
m.CompletedAt = timePtr(at)
}
func (m *RunManifest) MarkStageSkipped(name string, at time.Time, reason string) {
s := m.ensureStage(name, at)
s.Status = StatusSkipped
s.CompletedAt = timePtr(at)
s.Error = &ErrorRecord{Message: strings.TrimSpace(reason), Code: "skipped", At: timePtr(at)}
s.UpdatedAt = at
m.UpdatedAt = at
}
func (m *RunManifest) MarkSucceeded(at time.Time) {
m.Status = RunManifestStatusSucceeded
m.CompletedAt = timePtr(at)
m.UpdatedAt = at
}
func (m *RunManifest) MarkFailed(at time.Time, message string) {
m.Status = RunManifestStatusFailed
m.CompletedAt = timePtr(at)
m.LastError = &ErrorRecord{Message: strings.TrimSpace(message), At: timePtr(at)}
m.UpdatedAt = at
}
func (m *RunManifest) ensureStage(name string, at time.Time) *RunStageRecord {
if m.Stages == nil {
m.Stages = map[string]*RunStageRecord{}
}
stageName := strings.TrimSpace(name)
s, ok := m.Stages[stageName]
if !ok || s == nil {
s = &RunStageRecord{
Name: stageName,
Action: RunStageActionRun,
Status: StatusPending,
CreatedAt: at,
UpdatedAt: at,
}
m.Stages[stageName] = s
}
if s.Name == "" {
s.Name = stageName
}
if s.CreatedAt.IsZero() {
s.CreatedAt = at
}
return s
}

View File

@@ -0,0 +1,50 @@
package manifest
import (
"testing"
"time"
)
func TestRunManifestStageMarkHelpers(t *testing.T) {
rm := NewRun("2026-05-03", "forsaken", "20260517T000000Z-abcdef12", false, []string{"prepare"}, time.Date(2026, 5, 3, 10, 0, 0, 0, time.UTC))
if rm.Status != RunManifestStatusRunning {
t.Fatalf("status = %q, want %q", rm.Status, RunManifestStatusRunning)
}
runningAt := time.Date(2026, 5, 3, 10, 1, 0, 0, time.UTC)
rm.SetStageAction("prepare", RunStageActionRun, runningAt)
rm.MarkStageRunning("prepare", runningAt)
rm.MarkStageSucceeded("prepare", runningAt.Add(30*time.Second), []ArtifactRecord{
{Kind: "input", LocalPath: "inputs/session.yml"},
})
stage := rm.Stages["prepare"]
if stage == nil {
t.Fatal("prepare stage missing")
}
if stage.Action != RunStageActionRun {
t.Fatalf("action = %q, want %q", stage.Action, RunStageActionRun)
}
if stage.Status != StatusSucceeded {
t.Fatalf("status = %q, want %q", stage.Status, StatusSucceeded)
}
skippedAt := runningAt.Add(1 * time.Minute)
rm.SetStageAction("notify", RunStageActionSkip, skippedAt)
rm.MarkStageSkipped("notify", skippedAt, "already_succeeded")
skipped := rm.Stages["notify"]
if skipped == nil {
t.Fatal("notify stage missing")
}
if skipped.Action != RunStageActionSkip {
t.Fatalf("action = %q, want %q", skipped.Action, RunStageActionSkip)
}
if skipped.Status != StatusSkipped {
t.Fatalf("status = %q, want %q", skipped.Status, StatusSkipped)
}
rm.MarkSucceeded(skippedAt.Add(10 * time.Second))
if rm.Status != RunManifestStatusSucceeded {
t.Fatalf("status = %q, want %q", rm.Status, RunManifestStatusSucceeded)
}
}

View File

@@ -129,6 +129,89 @@ func (s *LocalStore) Save(ctx context.Context, path string, m *Manifest) error {
return nil
}
// CreateRun returns a new in-memory run manifest for one invocation.
func (s *LocalStore) CreateRun(
ctx context.Context,
sessionID, campaign, runID string,
force bool,
requestedStages []string,
) (*RunManifest, error) {
if err := checkContext(ctx); err != nil {
return nil, err
}
if strings.TrimSpace(sessionID) == "" {
return nil, fmt.Errorf("create run manifest: session_id is required")
}
if strings.TrimSpace(runID) == "" {
return nil, fmt.Errorf("create run manifest: run_id is required")
}
now := time.Now().UTC()
return NewRun(sessionID, campaign, runID, force, requestedStages, now), nil
}
// LoadRun reads and validates a local JSON run manifest from path.
func (s *LocalStore) LoadRun(ctx context.Context, path string) (*RunManifest, error) {
if err := checkContext(ctx); err != nil {
return nil, err
}
if strings.TrimSpace(path) == "" {
return nil, fmt.Errorf("load run manifest: path is required")
}
data, err := os.ReadFile(path)
if err != nil {
return nil, fmt.Errorf("load run manifest %q: %w", path, err)
}
var m RunManifest
if err := json.Unmarshal(data, &m); err != nil {
return nil, fmt.Errorf("decode run manifest %q: %w", path, err)
}
if err := validateLoadedRunManifest(&m); err != nil {
return nil, fmt.Errorf("run manifest %q invalid: %w", path, err)
}
normalizeRunManifest(&m)
return &m, nil
}
// SaveRun writes the run manifest to path atomically via temp file + rename.
func (s *LocalStore) SaveRun(ctx context.Context, path string, m *RunManifest) error {
if err := checkContext(ctx); err != nil {
return err
}
if strings.TrimSpace(path) == "" {
return fmt.Errorf("save run manifest: path is required")
}
if m == nil {
return fmt.Errorf("save run manifest: manifest is nil")
}
if strings.TrimSpace(m.SessionID) == "" {
return fmt.Errorf("save run manifest: session_id is required")
}
if strings.TrimSpace(m.RunID) == "" {
return fmt.Errorf("save run manifest: run_id is required")
}
if m.CreatedAt.IsZero() {
return fmt.Errorf("save run manifest: created_at is required")
}
m.UpdatedAt = time.Now().UTC()
if m.Stages == nil {
m.Stages = map[string]*RunStageRecord{}
}
data, err := json.MarshalIndent(m, "", " ")
if err != nil {
return fmt.Errorf("save run manifest: marshal: %w", err)
}
data = append(data, '\n')
return writeJSONAtomically(ctx, path, ".run-manifest.json.tmp-*", data)
}
func validateLoadedManifest(m *Manifest) error {
if m == nil {
return fmt.Errorf("manifest is nil")
@@ -161,6 +244,88 @@ func normalizeManifest(m *Manifest) {
}
}
func validateLoadedRunManifest(m *RunManifest) error {
if m == nil {
return fmt.Errorf("manifest is nil")
}
if strings.TrimSpace(m.SessionID) == "" {
return fmt.Errorf("session_id is required")
}
if strings.TrimSpace(m.RunID) == "" {
return fmt.Errorf("run_id is required")
}
if m.CreatedAt.IsZero() {
return fmt.Errorf("created_at is required")
}
if m.UpdatedAt.IsZero() {
return fmt.Errorf("updated_at is required")
}
return nil
}
func normalizeRunManifest(m *RunManifest) {
if m.Stages == nil {
m.Stages = map[string]*RunStageRecord{}
}
for name, stage := range m.Stages {
if stage == nil {
stage = &RunStageRecord{
Name: name,
Action: RunStageActionRun,
Status: StatusPending,
CreatedAt: m.CreatedAt,
UpdatedAt: m.UpdatedAt,
}
m.Stages[name] = stage
}
if stage.Name == "" {
stage.Name = name
}
}
}
func writeJSONAtomically(ctx context.Context, path, tempPattern string, data []byte) error {
dir := filepath.Dir(path)
if err := os.MkdirAll(dir, 0o755); err != nil {
return fmt.Errorf("create directory %q: %w", dir, err)
}
tmp, err := os.CreateTemp(dir, tempPattern)
if err != nil {
return fmt.Errorf("create temp file: %w", err)
}
tmpName := tmp.Name()
removeTmp := true
defer func() {
if removeTmp {
_ = os.Remove(tmpName)
}
}()
if _, err := tmp.Write(data); err != nil {
_ = tmp.Close()
return fmt.Errorf("write temp file: %w", err)
}
if err := tmp.Sync(); err != nil {
_ = tmp.Close()
return fmt.Errorf("sync temp file: %w", err)
}
if err := tmp.Close(); err != nil {
return fmt.Errorf("close temp file: %w", err)
}
if err := checkContext(ctx); err != nil {
return err
}
if err := os.Rename(tmpName, path); err != nil {
return fmt.Errorf("rename temp file: %w", err)
}
removeTmp = false
return nil
}
func checkContext(ctx context.Context) error {
if ctx == nil {
return nil

View File

@@ -150,3 +150,73 @@ func TestLoadRejectsInvalidManifest(t *testing.T) {
t.Fatalf("error = %q, want session_id validation", err.Error())
}
}
func TestLocalStoreCreateSaveLoadRunManifestRoundTrip(t *testing.T) {
store := &LocalStore{}
ctx := context.Background()
run, err := store.CreateRun(
ctx,
"2026-05-03",
"forsaken",
"20260517T000000Z-abcdef12",
true,
[]string{"prepare", "transcribe"},
)
if err != nil {
t.Fatalf("CreateRun() error = %v", err)
}
run.SessionManifestPath = "/var/lib/narratio/work/forsaken/2026-05-03/manifest.json"
run.MarkStageRunning("prepare", time.Date(2026, 5, 3, 12, 1, 0, 0, time.UTC))
run.MarkStageSucceeded("prepare", time.Date(2026, 5, 3, 12, 2, 0, 0, time.UTC), []ArtifactRecord{
{Kind: "input", LocalPath: "inputs/session.yml"},
})
run.MarkSucceeded(time.Date(2026, 5, 3, 12, 3, 0, 0, time.UTC))
path := filepath.Join(t.TempDir(), "run-manifest.json")
if err := store.SaveRun(ctx, path, run); err != nil {
t.Fatalf("SaveRun() error = %v", err)
}
loaded, err := store.LoadRun(ctx, path)
if err != nil {
t.Fatalf("LoadRun() error = %v", err)
}
if loaded.SessionID != "2026-05-03" {
t.Fatalf("SessionID = %q, want %q", loaded.SessionID, "2026-05-03")
}
if loaded.Campaign != "forsaken" {
t.Fatalf("Campaign = %q, want %q", loaded.Campaign, "forsaken")
}
if loaded.RunID != "20260517T000000Z-abcdef12" {
t.Fatalf("RunID = %q, want %q", loaded.RunID, "20260517T000000Z-abcdef12")
}
if loaded.Status != RunManifestStatusSucceeded {
t.Fatalf("Status = %q, want %q", loaded.Status, RunManifestStatusSucceeded)
}
if loaded.Stages["prepare"] == nil || loaded.Stages["prepare"].Status != StatusSucceeded {
t.Fatalf("prepare stage = %#v, want succeeded", loaded.Stages["prepare"])
}
if loaded.Stages["prepare"].Action != RunStageActionRun {
t.Fatalf("prepare action = %q, want %q", loaded.Stages["prepare"].Action, RunStageActionRun)
}
}
func TestLoadRunRejectsInvalidManifest(t *testing.T) {
store := &LocalStore{}
ctx := context.Background()
path := filepath.Join(t.TempDir(), "run-manifest.json")
if err := os.WriteFile(path, []byte(`{"session_id":"2026-05-03"}`), 0o644); err != nil {
t.Fatalf("WriteFile() error = %v", err)
}
_, err := store.LoadRun(ctx, path)
if err == nil {
t.Fatal("expected error, got nil")
}
if !strings.Contains(err.Error(), "run_id is required") {
t.Fatalf("error = %q, want run_id validation", err.Error())
}
}

View File

@@ -3,6 +3,7 @@ package stage
import (
"context"
"encoding/json"
"errors"
"fmt"
"os"
"path/filepath"
@@ -27,12 +28,23 @@ func (analyzeStage) Declares() IODecl {
{Kind: "transcript_normalized", Category: "transcripts", RelativePath: "transcripts/normalized.json"},
{Kind: "transcript_trimmed", Category: "transcripts", RelativePath: "transcripts/trimmed.json"},
},
Outputs: []artifacts.Ref{
{Kind: "session_recap", Category: "artifacts", RelativePath: "artifacts/session_recap.md"},
},
Outputs: nil,
}
}
type analyzeArtifactExecutionPlan struct {
Name string
Cfg config.ScriptoriumArtifactConfig
}
type analyzeArtifactExecutionResult struct {
Output artifacts.Ref
Logs []string
GeneratedConfigs []string
Metadata map[string]any
ReusedArtifacts []map[string]any
}
func (analyzeStage) Run(ctx context.Context, env *Env, m *manifest.Manifest) (*StageResult, error) {
if env == nil || env.Config == nil {
return nil, fmt.Errorf("analyze: stage environment config is required")
@@ -58,96 +70,314 @@ func (analyzeStage) Run(ctx context.Context, env *Env, m *manifest.Manifest) (*S
return nil, fmt.Errorf("analyze: session id is required")
}
paths := env.ArtifactStore.SessionPaths(sessionID)
paths := sessionPathsForEnv(env, sessionID)
runLayout, err := resolveRunStageLayout(env, m, paths, sessionID, "analyze")
if err != nil {
return nil, fmt.Errorf("analyze: resolve run-stage layout: %w", err)
}
if env.Config.Pipeline.Scriptorium == nil {
return &StageResult{
Metadata: map[string]any{
"stage": "analyze",
"skipped": true,
"reason": "pipeline.scriptorium is not configured",
},
}, nil
return &StageResult{Metadata: map[string]any{
"stage": "analyze",
"skipped": true,
"reason": "pipeline.scriptorium is not configured",
}}, nil
}
artifactName, artifactCfg, skipReason, err := selectAnalyzeArtifact(env.Config.Pipeline.Scriptorium)
runtimeCatalog, err := buildAnalyzeRuntimeArtifactCatalog(paths, env.Config.Pipeline.Scriptorium, env.SelectedAnalyzeArtifacts)
if err != nil {
return nil, fmt.Errorf("analyze: build runtime artifact catalog: %w", err)
}
plans, skipReason, err := buildAnalyzeExecutionPlans(env.Config.Pipeline.Scriptorium, runtimeCatalog)
if err != nil {
return nil, fmt.Errorf("analyze: %w", err)
}
if skipReason != "" {
return &StageResult{
Metadata: map[string]any{
"stage": "analyze",
"skipped": true,
"reason": skipReason,
},
}, nil
return &StageResult{Metadata: map[string]any{
"stage": "analyze",
"skipped": true,
"reason": skipReason,
}}, nil
}
processedTranscriptPath, processedSource, err := discoverProcessedTranscript(m, paths)
if err != nil {
return nil, fmt.Errorf("analyze: resolve processed transcript: %w", err)
}
normalizedTranscriptPath, normalizedSource, err := discoverNormalizedTranscript(m, paths)
if err != nil {
return nil, fmt.Errorf("analyze: resolve normalized transcript: %w", err)
}
trimmedTranscriptPath, trimmedSource, err := discoverTrimmedTranscript(m, paths)
if err != nil {
return nil, fmt.Errorf("analyze: resolve trimmed transcript: %w", err)
transcriptRefs := discoverAnalyzeTranscriptRefs(m, paths)
sessionDir := filepath.Dir(strings.TrimSpace(env.Config.SessionPath))
outputs := make([]artifacts.Ref, 0, len(plans))
logs := []string{}
generatedConfigs := []string{}
artifactMetadata := make([]map[string]any, 0, len(plans))
reusedArtifacts := []map[string]any{}
reusedSeen := map[string]struct{}{}
for _, plan := range plans {
artifactResult, err := executeAnalyzeArtifact(
ctx,
env,
m,
paths,
runLayout,
sessionID,
sessionDir,
plan,
transcriptRefs,
runtimeCatalog,
)
if err != nil {
return nil, err
}
outputs = append(outputs, artifactResult.Output)
logs = append(logs, artifactResult.Logs...)
generatedConfigs = append(generatedConfigs, artifactResult.GeneratedConfigs...)
artifactMetadata = append(artifactMetadata, artifactResult.Metadata)
for _, reused := range artifactResult.ReusedArtifacts {
sourceID, _ := reused["source_id"].(string)
path, _ := reused["path"].(string)
key := sourceID + "|" + path
if _, exists := reusedSeen[key]; exists {
continue
}
reusedSeen[key] = struct{}{}
reusedArtifacts = append(reusedArtifacts, reused)
}
sourceID, ok := runtimeCatalog.SourceIDForConfiguredKey(plan.Name)
if !ok {
return nil, fmt.Errorf("analyze: source id not found for artifact %q", plan.Name)
}
if err := runtimeCatalog.MarkAvailableGenerated(sourceID, artifactResult.Output.AbsolutePath); err != nil {
return nil, fmt.Errorf("analyze: mark generated artifact %q available: %w", sourceID, err)
}
}
transcriptInputs := analyzeTranscriptInputs{
ProcessedPath: processedTranscriptPath,
ProcessedSource: processedSource,
NormalizedPath: normalizedTranscriptPath,
NormalizedSource: normalizedSource,
TrimmedPath: trimmedTranscriptPath,
TrimmedSource: trimmedSource,
metadata := map[string]any{
"stage": "analyze",
"selected_artifacts": extractPlanNames(plans),
"generated_artifacts": artifactMetadata,
"reused_artifacts": reusedArtifacts,
"artifact_count": len(artifactMetadata),
"reused_artifact_count": len(reusedArtifacts),
}
if len(artifactMetadata) == 1 {
for k, v := range artifactMetadata[0] {
metadata[k] = v
}
}
return &StageResult{
Outputs: outputs,
Logs: dedupeAndSortPaths(logs),
GeneratedConfigs: dedupeAndSortPaths(generatedConfigs),
Metadata: metadata,
}, nil
}
func buildAnalyzeExecutionPlans(
scriptoriumCfg *config.ScriptoriumConfig,
catalog *artifacts.ArtifactCatalog,
) ([]analyzeArtifactExecutionPlan, string, error) {
if scriptoriumCfg == nil {
return nil, "pipeline.scriptorium is not configured", nil
}
if len(scriptoriumCfg.Artifacts) == 0 {
return nil, "no scriptorium artifacts configured", nil
}
entries := catalog.ListConfigured()
selected := make([]string, 0, len(entries))
for _, entry := range entries {
if entry.Executable {
selected = append(selected, entry.ConfiguredKey)
}
}
if len(selected) == 0 {
return nil, "no selected scriptorium artifacts to execute", nil
}
ordered, err := orderSelectedScriptoriumArtifacts(scriptoriumCfg.Artifacts, selected, catalog)
if err != nil {
return nil, "", err
}
plans := make([]analyzeArtifactExecutionPlan, 0, len(ordered))
for _, name := range ordered {
artifactCfg, ok := scriptoriumCfg.Artifacts[name]
if !ok {
return nil, "", fmt.Errorf("selected artifact %q is not configured", name)
}
plans = append(plans, analyzeArtifactExecutionPlan{Name: name, Cfg: artifactCfg})
}
return plans, "", nil
}
func orderSelectedScriptoriumArtifacts(
artifactsCfg map[string]config.ScriptoriumArtifactConfig,
selected []string,
catalog *artifacts.ArtifactCatalog,
) ([]string, error) {
selectedSet := map[string]struct{}{}
for _, key := range selected {
trimmed := strings.TrimSpace(key)
if trimmed == "" {
return nil, fmt.Errorf("selected artifact key must be non-empty")
}
selectedSet[trimmed] = struct{}{}
}
for selectedKey := range selectedSet {
cfg, ok := artifactsCfg[selectedKey]
if !ok {
return nil, fmt.Errorf("selected artifact %q is not configured", selectedKey)
}
for _, dep := range cfg.DependsOn {
trimmedDep := strings.TrimSpace(dep)
if trimmedDep == "" {
continue
}
if _, ok := selectedSet[trimmedDep]; ok {
continue
}
sourceID, ok := catalog.SourceIDForConfiguredKey(trimmedDep)
if !ok {
return nil, fmt.Errorf("artifact %q depends on unknown configured artifact %q", selectedKey, trimmedDep)
}
entry, ok := catalog.Lookup(sourceID)
if !ok || !entry.Available {
return nil, fmt.Errorf("artifact %q depends on %q, but %q is unavailable", selectedKey, trimmedDep, sourceID)
}
}
}
indegree := map[string]int{}
edges := map[string][]string{}
for key := range selectedSet {
indegree[key] = 0
}
for key := range selectedSet {
cfg := artifactsCfg[key]
for _, dep := range cfg.DependsOn {
trimmedDep := strings.TrimSpace(dep)
if _, ok := selectedSet[trimmedDep]; !ok {
continue
}
edges[trimmedDep] = append(edges[trimmedDep], key)
indegree[key]++
}
}
for key := range edges {
sort.Strings(edges[key])
}
ready := make([]string, 0, len(indegree))
for key, degree := range indegree {
if degree == 0 {
ready = append(ready, key)
}
}
sort.Strings(ready)
order := make([]string, 0, len(selectedSet))
for len(ready) > 0 {
node := ready[0]
ready = ready[1:]
order = append(order, node)
for _, dep := range edges[node] {
indegree[dep]--
if indegree[dep] == 0 {
ready = append(ready, dep)
sort.Strings(ready)
}
}
}
if len(order) != len(selectedSet) {
return nil, fmt.Errorf("selected scriptorium artifacts contain a dependency cycle")
}
return order, nil
}
func executeAnalyzeArtifact(
ctx context.Context,
env *Env,
m *manifest.Manifest,
paths artifacts.SessionPaths,
runLayout runStageLayout,
sessionID string,
sessionDir string,
plan analyzeArtifactExecutionPlan,
transcriptRefs analyzeTranscriptInputs,
runtimeCatalog *artifacts.ArtifactCatalog,
) (*analyzeArtifactExecutionResult, error) {
artifactName := plan.Name
artifactCfg := plan.Cfg
inputPaths := map[string]string{}
omittedOptionalInputs := []string{}
sessionDir := filepath.Dir(strings.TrimSpace(env.Config.SessionPath))
reusedArtifacts := []map[string]any{}
inputNames := sortedScriptoriumInputNames(artifactCfg.Inputs)
for _, inputName := range inputNames {
inputCfg := artifactCfg.Inputs[inputName]
resolvedPath, resolved, resolveErr := resolveScriptoriumInput(inputName, inputCfg, transcriptInputs, paths, sessionDir)
resolvedPath, resolved, resolvedArtifact, resolveErr := resolveScriptoriumInput(inputName, inputCfg, m, paths, sessionDir, runtimeCatalog)
if resolveErr != nil {
return nil, fmt.Errorf("analyze: resolve input %q: %w", inputName, resolveErr)
return nil, fmt.Errorf("analyze: resolve input %q for artifact %q: %w", inputName, artifactName, resolveErr)
}
if !resolved {
if inputCfg.Required {
return nil, fmt.Errorf("analyze: required input %q could not be resolved", inputName)
return nil, fmt.Errorf("analyze: required input %q for artifact %q could not be resolved", inputName, artifactName)
}
omittedOptionalInputs = append(omittedOptionalInputs, inputName)
continue
}
inputPaths[inputName] = resolvedPath
if resolvedArtifact != nil && resolvedArtifact.Provenance == artifacts.ArtifactProvenanceDisabledFromDisk {
reusedArtifacts = append(reusedArtifacts, map[string]any{
"name": configuredArtifactNameFromSourceID(resolvedArtifact.ID),
"source_id": resolvedArtifact.ID,
"path": resolvedArtifact.Path,
"provenance": resolvedArtifact.Provenance,
})
}
}
vars, err := buildScriptoriumVars(artifactCfg.Vars, env.Config.Session)
if err != nil {
return nil, fmt.Errorf("analyze: resolve vars: %w", err)
return nil, fmt.Errorf("analyze: resolve vars for artifact %q: %w", artifactName, err)
}
outputPath, err := resolveScriptoriumOutputPath(paths, artifactCfg.OutputPath)
canonicalOutputPath, err := resolveScriptoriumOutputPath(paths, artifactCfg.OutputPath)
if err != nil {
return nil, fmt.Errorf("analyze: resolve output path: %w", err)
return nil, fmt.Errorf("analyze: resolve output path for artifact %q: %w", artifactName, err)
}
outputPath, err := runLocalPathForCanonical(runLayout, paths, canonicalOutputPath)
if err != nil {
return nil, fmt.Errorf("analyze: resolve run-local output path for artifact %q: %w", artifactName, err)
}
stdoutLogPath := filepath.Join(paths.LogsDir, "scriptorium."+artifactName+".stdout.log")
stderrLogPath := filepath.Join(paths.LogsDir, "scriptorium."+artifactName+".stderr.log")
generatedConfigPath := filepath.Join(paths.ConfigDir, "scriptorium."+artifactName+".generated.yml")
if runLayout.Enabled {
stdoutLogPath = filepath.Join(runLayout.LogsDir, "scriptorium."+artifactName+".stdout.log")
stderrLogPath = filepath.Join(runLayout.LogsDir, "scriptorium."+artifactName+".stderr.log")
generatedConfigPath = filepath.Join(runLayout.ConfigDir, "scriptorium."+artifactName+".generated.yml")
}
timeout, err := resolveScriptoriumTimeout(env.Config.Pipeline.Scriptorium.Timeout, artifactCfg.Timeout)
if err != nil {
return nil, fmt.Errorf("analyze: resolve timeout: %w", err)
return nil, fmt.Errorf("analyze: resolve timeout for artifact %q: %w", artifactName, err)
}
logPaths := []string{}
generatedConfigs := []string{}
meta := map[string]any{
"stage": "analyze",
"name": artifactName,
"artifact_name": artifactName,
"source_id": artifacts.ConfiguredArtifactSourceID(artifactName),
"output_kind": "scriptorium_artifact",
"prompt_id": artifactCfg.PromptID,
"profile_id": artifactCfg.ProfileID,
"binary": env.Config.Pipeline.Scriptorium.Binary,
@@ -157,20 +387,28 @@ func (analyzeStage) Run(ctx context.Context, env *Env, m *manifest.Manifest) (*S
"omitted_optional_inputs": omittedOptionalInputs,
"vars": vars,
"timeout": timeout.String(),
"processed_transcript_path": processedTranscriptPath,
"processed_transcript_source": processedSource,
"normalized_transcript_path": normalizedTranscriptPath,
"normalized_transcript_source": normalizedSource,
"trimmed_transcript_path": trimmedTranscriptPath,
"trimmed_transcript_source": trimmedSource,
"processed_transcript_path": transcriptRefs.ProcessedPath,
"processed_transcript_source": transcriptRefs.ProcessedSource,
"normalized_transcript_path": transcriptRefs.NormalizedPath,
"normalized_transcript_source": transcriptRefs.NormalizedSource,
"trimmed_transcript_path": transcriptRefs.TrimmedPath,
"trimmed_transcript_source": transcriptRefs.TrimmedSource,
"render_debug_enabled": resolveRenderDebugEnabled(env.Config.Pipeline.Scriptorium.RenderDebug, artifactCfg.RenderDebug),
}
if meta["render_debug_enabled"] == true {
renderOutputPath := filepath.Join(paths.ArtifactsDir, artifactName+".render.json")
if runLayout.Enabled {
renderOutputPath = filepath.Join(runLayout.ReportsDir, artifactName+".render.json")
}
renderStdoutPath := filepath.Join(paths.LogsDir, "scriptorium."+artifactName+".render.stdout.log")
renderStderrPath := filepath.Join(paths.LogsDir, "scriptorium."+artifactName+".render.stderr.log")
renderGeneratedConfigPath := filepath.Join(paths.ConfigDir, "scriptorium."+artifactName+".render.generated.yml")
if runLayout.Enabled {
renderStdoutPath = filepath.Join(runLayout.LogsDir, "scriptorium."+artifactName+".render.stdout.log")
renderStderrPath = filepath.Join(runLayout.LogsDir, "scriptorium."+artifactName+".render.stderr.log")
renderGeneratedConfigPath = filepath.Join(runLayout.ConfigDir, "scriptorium."+artifactName+".render.generated.yml")
}
renderReq := scriptorium.RenderArtifactRequest{
Binary: env.Config.Pipeline.Scriptorium.Binary,
@@ -187,10 +425,10 @@ func (analyzeStage) Run(ctx context.Context, env *Env, m *manifest.Manifest) (*S
}
renderRes, renderErr := env.Scriptorium.RenderArtifact(ctx, renderReq)
if renderErr != nil {
return nil, fmt.Errorf("analyze: scriptorium render failed: %w", renderErr)
return nil, fmt.Errorf("analyze: scriptorium render failed for artifact %q: %w", artifactName, renderErr)
}
if renderRes.ValidationFailed {
return nil, fmt.Errorf("analyze: scriptorium render returned validation_failed=true")
return nil, fmt.Errorf("analyze: scriptorium render returned validation_failed=true for artifact %q", artifactName)
}
finalRenderOutputPath := coalesceString(renderRes.OutputPath, renderReq.OutputPath)
if err := requireNonEmptyFile(finalRenderOutputPath, artifactName+" render output"); err != nil {
@@ -238,7 +476,8 @@ func (analyzeStage) Run(ctx context.Context, env *Env, m *manifest.Manifest) (*S
if runErr != nil {
if res.ValidationFailed {
return nil, fmt.Errorf(
"analyze: scriptorium validation failed (prompt_id=%q, output_path=%q, exit_code=%d, stdout_log=%q, stderr_log=%q): %w",
"analyze: scriptorium validation failed (artifact=%q, prompt_id=%q, output_path=%q, exit_code=%d, stdout_log=%q, stderr_log=%q): %w",
artifactName,
req.PromptID,
coalesceString(res.OutputPath, req.OutputPath),
res.ExitCode,
@@ -247,27 +486,30 @@ func (analyzeStage) Run(ctx context.Context, env *Env, m *manifest.Manifest) (*S
runErr,
)
}
return nil, fmt.Errorf("analyze: scriptorium run failed: %w", runErr)
return nil, fmt.Errorf("analyze: scriptorium run failed for artifact %q: %w", artifactName, runErr)
}
if res.ValidationFailed {
return nil, fmt.Errorf("analyze: scriptorium run returned validation_failed=true")
return nil, fmt.Errorf("analyze: scriptorium run returned validation_failed=true for artifact %q", artifactName)
}
finalOutputPath := coalesceString(res.OutputPath, req.OutputPath)
if err := requireNonEmptyFile(finalOutputPath, artifactName+" output"); err != nil {
return nil, fmt.Errorf("analyze: %w", err)
}
artifactRef := artifacts.Ref{
Kind: artifactName,
Category: "artifacts",
SessionID: sessionID,
AbsolutePath: finalOutputPath,
promotedArtifact, err := promoteRunLocalOutput(env.ArtifactStore, finalOutputPath, canonicalOutputPath, artifacts.Ref{
Kind: artifactName,
Category: "artifacts",
SessionID: sessionID,
})
if err != nil {
return nil, fmt.Errorf("analyze: promote artifact output for %q: %w", artifactName, err)
}
logPaths = append(logPaths, stdoutLogPath, stderrLogPath)
generatedConfigs = append(generatedConfigs, generatedConfigPath)
meta["output_path"] = finalOutputPath
meta["run_output_path"] = finalOutputPath
meta["path"] = canonicalOutputPath
meta["output_path"] = canonicalOutputPath
meta["generated_config_path"] = generatedConfigPath
meta["stdout_log_path"] = stdoutLogPath
meta["stderr_log_path"] = stderrLogPath
@@ -281,39 +523,38 @@ func (analyzeStage) Run(ctx context.Context, env *Env, m *manifest.Manifest) (*S
meta["adapter_generated_config"] = res.GeneratedConfigPath
meta["adapter_stdout_log_path"] = res.StdoutLogPath
meta["adapter_stderr_log_path"] = res.StderrLogPath
meta["provenance"] = artifacts.ArtifactProvenanceGeneratedCurrentAnalyzeRun
if res.Metadata != nil {
meta["adapter_metadata"] = res.Metadata
}
return &StageResult{
Outputs: []artifacts.Ref{artifactRef},
return &analyzeArtifactExecutionResult{
Output: promotedArtifact,
Logs: logPaths,
GeneratedConfigs: generatedConfigs,
Metadata: meta,
ReusedArtifacts: reusedArtifacts,
}, nil
}
func selectAnalyzeArtifact(cfg *config.ScriptoriumConfig) (string, config.ScriptoriumArtifactConfig, string, error) {
if cfg == nil {
return "", config.ScriptoriumArtifactConfig{}, "pipeline.scriptorium is not configured", nil
func extractPlanNames(plans []analyzeArtifactExecutionPlan) []string {
if len(plans) == 0 {
return nil
}
out := make([]string, 0, len(plans))
for _, plan := range plans {
out = append(out, plan.Name)
}
return out
}
enabled := []string{}
for name, artifact := range cfg.Artifacts {
if artifact.Enabled {
enabled = append(enabled, name)
}
func configuredArtifactNameFromSourceID(sourceID string) string {
trimmed := strings.TrimSpace(sourceID)
const prefix = "narratio.artifact."
if !strings.HasPrefix(trimmed, prefix) {
return ""
}
sort.Strings(enabled)
if len(enabled) == 0 {
return "", config.ScriptoriumArtifactConfig{}, "no enabled scriptorium artifacts configured", nil
}
sessionRecapCfg, ok := cfg.Artifacts["session_recap"]
if !ok || !sessionRecapCfg.Enabled {
return "", config.ScriptoriumArtifactConfig{}, "", fmt.Errorf("only artifacts.session_recap is supported in this analyze implementation; enabled=%s", strings.Join(enabled, ","))
}
return "session_recap", sessionRecapCfg, "", nil
return strings.TrimPrefix(trimmed, prefix)
}
func discoverProcessedTranscript(m *manifest.Manifest, paths artifacts.SessionPaths) (string, string, error) {
@@ -350,40 +591,6 @@ func discoverProcessedTranscript(m *manifest.Manifest, paths artifacts.SessionPa
return "", "", nil
}
func discoverTrimmedTranscript(m *manifest.Manifest, paths artifacts.SessionPaths) (string, string, error) {
candidates := []string{}
if m != nil && m.Stages != nil {
if sr := m.Stages["trim"]; sr != nil {
for _, out := range sr.Outputs {
if out.Kind != "transcript_trimmed" {
continue
}
p := strings.TrimSpace(out.LocalPath)
if p == "" {
continue
}
resolved := artifacts.ResolveSessionLocalPathForRead(paths, p)
candidates = append(candidates, filepath.Clean(resolved))
}
}
}
deduped := dedupeAndSortPaths(candidates)
for _, p := range deduped {
if info, err := os.Stat(p); err == nil && !info.IsDir() {
return p, "manifest.trim.outputs", nil
}
}
fallback := filepath.Join(paths.TranscriptsDir, "trimmed.json")
if info, err := os.Stat(fallback); err == nil && !info.IsDir() {
return filepath.Clean(fallback), "fallback.transcripts_dir", nil
}
if len(deduped) > 0 {
return deduped[0], "manifest.trim.outputs", nil
}
return "", "", nil
}
type analyzeTranscriptInputs struct {
ProcessedPath string
ProcessedSource string
@@ -393,52 +600,125 @@ type analyzeTranscriptInputs struct {
TrimmedSource string
}
func discoverAnalyzeTranscriptRefs(m *manifest.Manifest, paths artifacts.SessionPaths) analyzeTranscriptInputs {
processedPath, processedSource := discoverAnalyzeArtifactRef(m, paths, artifacts.ArtifactTranscriptPolished)
normalizedPath, normalizedSource := discoverAnalyzeArtifactRef(m, paths, artifacts.ArtifactTranscriptFull)
trimmedPath, trimmedSource := discoverAnalyzeArtifactRef(m, paths, artifacts.ArtifactTranscriptTrimmed)
return analyzeTranscriptInputs{
ProcessedPath: processedPath,
ProcessedSource: processedSource,
NormalizedPath: normalizedPath,
NormalizedSource: normalizedSource,
TrimmedPath: trimmedPath,
TrimmedSource: trimmedSource,
}
}
func discoverAnalyzeArtifactRef(m *manifest.Manifest, paths artifacts.SessionPaths, source string) (string, string) {
resolved, err := artifacts.ResolveSessionArtifact(paths, m, source)
if err != nil {
return "", ""
}
return resolved.Path, resolved.Provenance
}
func resolveScriptoriumInput(
inputName string,
inputCfg config.ScriptoriumInputConfig,
transcriptInputs analyzeTranscriptInputs,
m *manifest.Manifest,
paths artifacts.SessionPaths,
sessionDir string,
) (string, bool, error) {
switch strings.TrimSpace(inputCfg.Source) {
case "processed_transcript":
if strings.TrimSpace(transcriptInputs.ProcessedPath) == "" {
return "", false, nil
}
if err := validateProcessedTranscriptOutput(transcriptInputs.ProcessedPath); err != nil {
return "", false, fmt.Errorf("processed transcript %q invalid: %w", transcriptInputs.ProcessedPath, err)
}
return transcriptInputs.ProcessedPath, true, nil
case "normalized_transcript":
if strings.TrimSpace(transcriptInputs.NormalizedPath) == "" {
return "", false, fmt.Errorf("normalized transcript input is unavailable; run normalize stage first")
}
if err := validateProcessedTranscriptOutput(transcriptInputs.NormalizedPath); err != nil {
return "", false, fmt.Errorf("normalized transcript %q invalid: %w", transcriptInputs.NormalizedPath, err)
}
return transcriptInputs.NormalizedPath, true, nil
case "trimmed_transcript":
if strings.TrimSpace(transcriptInputs.TrimmedPath) == "" {
return "", false, fmt.Errorf("trimmed transcript input is unavailable; run trim stage first")
}
if err := validateProcessedTranscriptOutput(transcriptInputs.TrimmedPath); err != nil {
return "", false, fmt.Errorf("trimmed transcript %q invalid: %w", transcriptInputs.TrimmedPath, err)
}
return transcriptInputs.TrimmedPath, true, nil
runtimeCatalog *artifacts.ArtifactCatalog,
) (string, bool, *artifacts.ResolvedSessionArtifact, error) {
source := strings.TrimSpace(inputCfg.Source)
switch source {
case "previous_session_artifact":
if strings.TrimSpace(inputCfg.Path) == "" {
return "", false, nil
return "", false, nil, nil
}
resolved := resolveInputPathForRead(paths, sessionDir, inputCfg.Path)
if err := requireFile(resolved, "scriptorium input "+inputName); err != nil {
return "", false, nil
return "", false, nil, nil
}
return resolved, true, nil
return resolved, true, nil, nil
default:
return "", false, fmt.Errorf("unsupported source %q", inputCfg.Source)
resolved, err := artifacts.ResolveSessionArtifactWithCatalog(paths, m, source, runtimeCatalog)
if err == nil {
copy := resolved
return resolved.Path, true, &copy, nil
}
if errors.Is(err, artifacts.ErrSessionArtifactNotFound) {
if artifacts.IsConfiguredArtifactSource(source) {
if inputCfg.Required {
return "", false, nil, fmt.Errorf("configured artifact source %q is unavailable", source)
}
return "", false, nil, nil
}
normalized, normalizeErr := artifacts.NormalizeSessionArtifactSource(source)
if normalizeErr != nil {
return "", false, nil, normalizeErr
}
switch normalized {
case artifacts.ArtifactTranscriptPolished:
return "", false, nil, nil
case artifacts.ArtifactTranscriptFull:
return "", false, nil, fmt.Errorf("normalized transcript input is unavailable; run normalize stage first")
case artifacts.ArtifactTranscriptTrimmed:
return "", false, nil, fmt.Errorf("trimmed transcript input is unavailable; run trim stage first")
default:
return "", false, nil, nil
}
}
return "", false, nil, err
}
}
func buildAnalyzeRuntimeArtifactCatalog(
paths artifacts.SessionPaths,
scriptoriumCfg *config.ScriptoriumConfig,
selectedArtifacts []string,
) (*artifacts.ArtifactCatalog, error) {
catalog := artifacts.NewArtifactCatalog()
if err := catalog.RegisterBuiltIns(); err != nil {
return nil, err
}
if scriptoriumCfg == nil {
return catalog, nil
}
configured := map[string]artifacts.ConfiguredArtifactDefinition{}
for key, artifactCfg := range scriptoriumCfg.Artifacts {
configured[key] = artifacts.ConfiguredArtifactDefinition{
Enabled: artifactCfg.Enabled,
OutputPath: artifactCfg.OutputPath,
}
}
if err := catalog.RegisterConfiguredArtifacts(configured, selectedArtifacts); err != nil {
return nil, err
}
for _, entry := range catalog.ListConfigured() {
if entry.Executable {
continue
}
if strings.TrimSpace(entry.CanonicalRelPath) == "" {
continue
}
resolvedPath, err := resolveScriptoriumOutputPath(paths, entry.CanonicalRelPath)
if err != nil {
continue
}
if err := requireNonEmptyFile(resolvedPath, "configured artifact "+entry.SourceID); err != nil {
continue
}
if err := catalog.MarkAvailableFromDisk(entry.SourceID, resolvedPath); err != nil {
return nil, err
}
}
return catalog, nil
}
func resolveInputPathForRead(paths artifacts.SessionPaths, sessionDir, pathValue string) string {
trimmed := strings.TrimSpace(pathValue)
if trimmed == "" {

View File

@@ -17,7 +17,7 @@ import (
func TestAnalyzeGeneratesSessionRecapFromTrimmedTranscript(t *testing.T) {
env, m, fake := setupAnalyzeEnv(t)
paths := env.ArtifactStore.SessionPaths(m.SessionID)
paths := sessionPathsForEnv(env, m.SessionID)
writeAnalyzeFile(t, filepath.Join(paths.TranscriptsDir, "trimmed.json"), `{"segments":[]}`)
result, err := (analyzeStage{}).Run(context.Background(), env, m)
@@ -70,7 +70,7 @@ func TestAnalyzeGeneratesSessionRecapFromTrimmedTranscript(t *testing.T) {
func TestAnalyzeRenderDebugFalseDoesNotCallRenderArtifact(t *testing.T) {
env, m, fake := setupAnalyzeEnv(t)
paths := env.ArtifactStore.SessionPaths(m.SessionID)
paths := sessionPathsForEnv(env, m.SessionID)
writeAnalyzeFile(t, filepath.Join(paths.TranscriptsDir, "trimmed.json"), `{"segments":[]}`)
env.Config.Pipeline.Scriptorium.RenderDebug = false
@@ -86,7 +86,7 @@ func TestAnalyzeRenderDebugFalseDoesNotCallRenderArtifact(t *testing.T) {
func TestAnalyzeRenderDebugArtifactOverrideFalseWinsOverGlobalTrue(t *testing.T) {
env, m, fake := setupAnalyzeEnv(t)
paths := env.ArtifactStore.SessionPaths(m.SessionID)
paths := sessionPathsForEnv(env, m.SessionID)
writeAnalyzeFile(t, filepath.Join(paths.TranscriptsDir, "trimmed.json"), `{"segments":[]}`)
env.Config.Pipeline.Scriptorium.RenderDebug = true
@@ -109,7 +109,7 @@ func TestAnalyzeRenderDebugArtifactOverrideFalseWinsOverGlobalTrue(t *testing.T)
func TestAnalyzeRenderDebugTrueCallsRenderBeforeRun(t *testing.T) {
env, m, _ := setupAnalyzeEnv(t)
paths := env.ArtifactStore.SessionPaths(m.SessionID)
paths := sessionPathsForEnv(env, m.SessionID)
writeAnalyzeFile(t, filepath.Join(paths.TranscriptsDir, "trimmed.json"), `{"segments":[]}`)
env.Config.Pipeline.Scriptorium.RenderDebug = true
@@ -130,7 +130,7 @@ func TestAnalyzeRenderDebugTrueCallsRenderBeforeRun(t *testing.T) {
func TestAnalyzeRenderOutputPathIsRecorded(t *testing.T) {
env, m, fake := setupAnalyzeEnv(t)
paths := env.ArtifactStore.SessionPaths(m.SessionID)
paths := sessionPathsForEnv(env, m.SessionID)
writeAnalyzeFile(t, filepath.Join(paths.TranscriptsDir, "trimmed.json"), `{"segments":[]}`)
env.Config.Pipeline.Scriptorium.RenderDebug = true
@@ -154,7 +154,7 @@ func TestAnalyzeRenderOutputPathIsRecorded(t *testing.T) {
func TestAnalyzeRenderFailurePreventsRun(t *testing.T) {
env, m, fake := setupAnalyzeEnv(t)
paths := env.ArtifactStore.SessionPaths(m.SessionID)
paths := sessionPathsForEnv(env, m.SessionID)
writeAnalyzeFile(t, filepath.Join(paths.TranscriptsDir, "trimmed.json"), `{"segments":[]}`)
env.Config.Pipeline.Scriptorium.RenderDebug = true
fake.RenderErr = errors.New("render boom")
@@ -176,7 +176,7 @@ func TestAnalyzeRenderFailurePreventsRun(t *testing.T) {
func TestAnalyzeRenderInvalidJSONFailsClearly(t *testing.T) {
env, m, _ := setupAnalyzeEnv(t)
paths := env.ArtifactStore.SessionPaths(m.SessionID)
paths := sessionPathsForEnv(env, m.SessionID)
writeAnalyzeFile(t, filepath.Join(paths.TranscriptsDir, "trimmed.json"), `{"segments":[]}`)
env.Config.Pipeline.Scriptorium.RenderDebug = true
runner := &orderedScriptoriumRunner{
@@ -199,7 +199,7 @@ func TestAnalyzeRenderInvalidJSONFailsClearly(t *testing.T) {
func TestAnalyzeRunStillSucceedsWhenRenderSucceeds(t *testing.T) {
env, m, fake := setupAnalyzeEnv(t)
paths := env.ArtifactStore.SessionPaths(m.SessionID)
paths := sessionPathsForEnv(env, m.SessionID)
writeAnalyzeFile(t, filepath.Join(paths.TranscriptsDir, "trimmed.json"), `{"segments":[]}`)
env.Config.Pipeline.Scriptorium.RenderDebug = true
@@ -220,7 +220,7 @@ func TestAnalyzeRunStillSucceedsWhenRenderSucceeds(t *testing.T) {
func TestAnalyzeOmitsOptionalPreviousRecapWhenUnavailable(t *testing.T) {
env, m, fake := setupAnalyzeEnv(t)
paths := env.ArtifactStore.SessionPaths(m.SessionID)
paths := sessionPathsForEnv(env, m.SessionID)
writeAnalyzeFile(t, filepath.Join(paths.TranscriptsDir, "processed.json"), `{"segments":[]}`)
env.Config.Pipeline.Scriptorium.Artifacts["session_recap"] = config.ScriptoriumArtifactConfig{
@@ -231,7 +231,7 @@ func TestAnalyzeOmitsOptionalPreviousRecapWhenUnavailable(t *testing.T) {
Timeout: "2m",
Inputs: map[string]config.ScriptoriumInputConfig{
"transcript": {
Source: "processed_transcript",
Source: "narratio.transcript.polished",
Required: true,
},
"previous_recap": {
@@ -256,6 +256,32 @@ func TestAnalyzeOmitsOptionalPreviousRecapWhenUnavailable(t *testing.T) {
}
}
func TestAnalyzeUsesRunLocalPathsAndPromotesCanonical(t *testing.T) {
env, m, fake := setupAnalyzeEnv(t)
env.Config.Session.Campaign = "sample-campaign"
m.Campaign = "sample-campaign"
m.RunID = "20260518T010203Z-abcdef12"
paths := sessionPathsForEnv(env, m.SessionID)
writeAnalyzeFile(t, filepath.Join(paths.TranscriptsDir, "trimmed.json"), `{"segments":[]}`)
result, err := (analyzeStage{}).Run(context.Background(), env, m)
if err != nil {
t.Fatalf("Run() error = %v", err)
}
if len(fake.RunRequests) != 1 {
t.Fatalf("run requests = %d, want 1", len(fake.RunRequests))
}
if !strings.Contains(fake.RunRequests[0].OutputPath, filepath.Join("runs", m.RunID, "analyze", "outputs")) {
t.Fatalf("run output path = %q, want run-local path", fake.RunRequests[0].OutputPath)
}
if len(result.Outputs) != 1 {
t.Fatalf("outputs len = %d, want 1", len(result.Outputs))
}
if strings.Contains(result.Outputs[0].AbsolutePath, string(filepath.Separator)+"runs"+string(filepath.Separator)) {
t.Fatalf("promoted output path = %q, want canonical session path", result.Outputs[0].AbsolutePath)
}
}
type orderedScriptoriumRunner struct {
Calls []string
RenderErr error
@@ -318,7 +344,7 @@ func (r *orderedScriptoriumRunner) RunArtifact(_ context.Context, req scriptoriu
func TestAnalyzeIncludesPreviousRecapWhenConfiguredAndAvailable(t *testing.T) {
env, m, fake := setupAnalyzeEnv(t)
paths := env.ArtifactStore.SessionPaths(m.SessionID)
paths := sessionPathsForEnv(env, m.SessionID)
writeAnalyzeFile(t, filepath.Join(paths.TranscriptsDir, "processed.json"), `{"segments":[]}`)
previousRecapPath := filepath.Join(filepath.Dir(env.Config.SessionPath), "previous", "session_recap.md")
@@ -332,7 +358,7 @@ func TestAnalyzeIncludesPreviousRecapWhenConfiguredAndAvailable(t *testing.T) {
Timeout: "2m",
Inputs: map[string]config.ScriptoriumInputConfig{
"transcript": {
Source: "processed_transcript",
Source: "narratio.transcript.polished",
Required: true,
},
"previous_recap": {
@@ -359,7 +385,7 @@ func TestAnalyzeIncludesPreviousRecapWhenConfiguredAndAvailable(t *testing.T) {
func TestAnalyzeFailsWhenRequiredPreviousRecapMissing(t *testing.T) {
env, m, _ := setupAnalyzeEnv(t)
paths := env.ArtifactStore.SessionPaths(m.SessionID)
paths := sessionPathsForEnv(env, m.SessionID)
writeAnalyzeFile(t, filepath.Join(paths.TranscriptsDir, "processed.json"), `{"segments":[]}`)
env.Config.Pipeline.Scriptorium.Artifacts["session_recap"] = config.ScriptoriumArtifactConfig{
@@ -368,7 +394,7 @@ func TestAnalyzeFailsWhenRequiredPreviousRecapMissing(t *testing.T) {
OutputPath: "artifacts/session_recap.md",
Inputs: map[string]config.ScriptoriumInputConfig{
"transcript": {
Source: "processed_transcript",
Source: "narratio.transcript.polished",
Required: true,
},
"previous_recap": {
@@ -388,9 +414,312 @@ func TestAnalyzeFailsWhenRequiredPreviousRecapMissing(t *testing.T) {
}
}
func TestAnalyzeResolvesConfiguredArtifactInputFromDisabledArtifactOutput(t *testing.T) {
env, m, fake := setupAnalyzeEnv(t)
paths := sessionPathsForEnv(env, m.SessionID)
writeAnalyzeFile(t, filepath.Join(paths.TranscriptsDir, "trimmed.json"), `{"segments":[]}`)
playerHandoutPath := filepath.Join(paths.ArtifactsDir, "player_handout.md")
writeAnalyzeFile(t, playerHandoutPath, "handout\n")
sessionRecap := env.Config.Pipeline.Scriptorium.Artifacts["session_recap"]
sessionRecap.Inputs["recap"] = config.ScriptoriumInputConfig{
Source: "narratio.artifact.player_handout",
Required: true,
}
env.Config.Pipeline.Scriptorium.Artifacts["session_recap"] = sessionRecap
env.Config.Pipeline.Scriptorium.Artifacts["player_handout"] = config.ScriptoriumArtifactConfig{
Enabled: false,
OutputPath: "artifacts/player_handout.md",
}
_, err := (analyzeStage{}).Run(context.Background(), env, m)
if err != nil {
t.Fatalf("Run() error = %v", err)
}
if len(fake.RunRequests) != 1 {
t.Fatalf("run requests = %d, want 1", len(fake.RunRequests))
}
if got := fake.RunRequests[0].InputPaths["recap"]; got != playerHandoutPath {
t.Fatalf("recap input = %q, want %q", got, playerHandoutPath)
}
}
func TestAnalyzeMetadataIncludesGeneratedAndReusedArtifacts(t *testing.T) {
env, m, _ := setupAnalyzeEnv(t)
paths := sessionPathsForEnv(env, m.SessionID)
writeAnalyzeFile(t, filepath.Join(paths.TranscriptsDir, "trimmed.json"), `{"segments":[]}`)
playerHandoutPath := filepath.Join(paths.ArtifactsDir, "player_handout.md")
writeAnalyzeFile(t, playerHandoutPath, "handout\n")
sessionRecap := env.Config.Pipeline.Scriptorium.Artifacts["session_recap"]
sessionRecap.Inputs["recap"] = config.ScriptoriumInputConfig{
Source: "narratio.artifact.player_handout",
Required: true,
}
env.Config.Pipeline.Scriptorium.Artifacts["session_recap"] = sessionRecap
env.Config.Pipeline.Scriptorium.Artifacts["player_handout"] = config.ScriptoriumArtifactConfig{
Enabled: false,
OutputPath: "artifacts/player_handout.md",
}
result, err := (analyzeStage{}).Run(context.Background(), env, m)
if err != nil {
t.Fatalf("Run() error = %v", err)
}
generated := mustArtifactEntryList(t, result.Metadata, "generated_artifacts")
if len(generated) != 1 {
t.Fatalf("generated_artifacts len = %d, want 1 (%#v)", len(generated), generated)
}
g0 := generated[0]
if g0["name"] != "session_recap" {
t.Fatalf("generated[0].name = %#v, want session_recap", g0["name"])
}
if g0["source_id"] != "narratio.artifact.session_recap" {
t.Fatalf("generated[0].source_id = %#v, want narratio.artifact.session_recap", g0["source_id"])
}
if g0["output_kind"] != "scriptorium_artifact" {
t.Fatalf("generated[0].output_kind = %#v, want scriptorium_artifact", g0["output_kind"])
}
if g0["path"] != filepath.Join(paths.ArtifactsDir, "session_recap.md") {
t.Fatalf("generated[0].path = %#v, want session recap path", g0["path"])
}
if g0["prompt_id"] != "dnd.session_recap" {
t.Fatalf("generated[0].prompt_id = %#v, want dnd.session_recap", g0["prompt_id"])
}
if g0["profile_id"] != "local-quality" {
t.Fatalf("generated[0].profile_id = %#v, want local-quality", g0["profile_id"])
}
if g0["provenance"] != artifacts.ArtifactProvenanceGeneratedCurrentAnalyzeRun {
t.Fatalf("generated[0].provenance = %#v, want %q", g0["provenance"], artifacts.ArtifactProvenanceGeneratedCurrentAnalyzeRun)
}
reused := mustArtifactEntryList(t, result.Metadata, "reused_artifacts")
if len(reused) != 1 {
t.Fatalf("reused_artifacts len = %d, want 1 (%#v)", len(reused), reused)
}
r0 := reused[0]
if r0["name"] != "player_handout" {
t.Fatalf("reused[0].name = %#v, want player_handout", r0["name"])
}
if r0["source_id"] != "narratio.artifact.player_handout" {
t.Fatalf("reused[0].source_id = %#v, want narratio.artifact.player_handout", r0["source_id"])
}
if r0["path"] != playerHandoutPath {
t.Fatalf("reused[0].path = %#v, want %q", r0["path"], playerHandoutPath)
}
if r0["provenance"] != artifacts.ArtifactProvenanceDisabledFromDisk {
t.Fatalf("reused[0].provenance = %#v, want %q", r0["provenance"], artifacts.ArtifactProvenanceDisabledFromDisk)
}
}
func TestAnalyzeRunsMultipleIndependentArtifactsInDeterministicOrder(t *testing.T) {
env, m, fake := setupAnalyzeEnv(t)
paths := sessionPathsForEnv(env, m.SessionID)
writeAnalyzeFile(t, filepath.Join(paths.TranscriptsDir, "trimmed.json"), `{"segments":[]}`)
env.Config.Pipeline.Scriptorium.Artifacts["player_handout"] = config.ScriptoriumArtifactConfig{
Enabled: true,
PromptID: "dnd.player_handout",
ProfileID: "local-quality",
OutputPath: "artifacts/player_handout.md",
Inputs: map[string]config.ScriptoriumInputConfig{
"transcript": {
Source: "narratio.transcript.trimmed",
Required: true,
},
},
}
result, err := (analyzeStage{}).Run(context.Background(), env, m)
if err != nil {
t.Fatalf("Run() error = %v", err)
}
if len(fake.RunRequests) != 2 {
t.Fatalf("run requests = %d, want 2", len(fake.RunRequests))
}
if fake.RunRequests[0].PromptID != "dnd.player_handout" {
t.Fatalf("first prompt id = %q, want dnd.player_handout", fake.RunRequests[0].PromptID)
}
if fake.RunRequests[1].PromptID != "dnd.session_recap" {
t.Fatalf("second prompt id = %q, want dnd.session_recap", fake.RunRequests[1].PromptID)
}
selected, ok := result.Metadata["selected_artifacts"].([]string)
if !ok {
t.Fatalf("selected_artifacts = %#v, want []string", result.Metadata["selected_artifacts"])
}
if len(selected) != 2 || selected[0] != "player_handout" || selected[1] != "session_recap" {
t.Fatalf("selected_artifacts = %#v, want [player_handout session_recap]", selected)
}
}
func TestAnalyzeRunsDependenciesBeforeDependents(t *testing.T) {
env, m, fake := setupAnalyzeEnv(t)
paths := sessionPathsForEnv(env, m.SessionID)
writeAnalyzeFile(t, filepath.Join(paths.TranscriptsDir, "trimmed.json"), `{"segments":[]}`)
env.Config.Pipeline.Scriptorium.Artifacts["player_handout"] = config.ScriptoriumArtifactConfig{
Enabled: true,
DependsOn: []string{"session_recap"},
PromptID: "dnd.player_handout",
ProfileID: "local-quality",
OutputPath: "artifacts/player_handout.md",
Inputs: map[string]config.ScriptoriumInputConfig{
"recap": {
Source: "narratio.artifact.session_recap",
Required: true,
},
"transcript": {
Source: "narratio.transcript.trimmed",
Required: true,
},
},
}
_, err := (analyzeStage{}).Run(context.Background(), env, m)
if err != nil {
t.Fatalf("Run() error = %v", err)
}
if len(fake.RunRequests) != 2 {
t.Fatalf("run requests = %d, want 2", len(fake.RunRequests))
}
if fake.RunRequests[0].PromptID != "dnd.session_recap" {
t.Fatalf("first prompt id = %q, want dnd.session_recap", fake.RunRequests[0].PromptID)
}
if fake.RunRequests[1].PromptID != "dnd.player_handout" {
t.Fatalf("second prompt id = %q, want dnd.player_handout", fake.RunRequests[1].PromptID)
}
if got := fake.RunRequests[1].InputPaths["recap"]; got != filepath.Join(paths.ArtifactsDir, "session_recap.md") {
t.Fatalf("dependent recap path = %q, want %q", got, filepath.Join(paths.ArtifactsDir, "session_recap.md"))
}
}
func TestAnalyzeAppliesSelectedArtifactsFilter(t *testing.T) {
env, m, fake := setupAnalyzeEnv(t)
paths := sessionPathsForEnv(env, m.SessionID)
writeAnalyzeFile(t, filepath.Join(paths.TranscriptsDir, "trimmed.json"), `{"segments":[]}`)
env.Config.Pipeline.Scriptorium.Artifacts["player_handout"] = config.ScriptoriumArtifactConfig{
Enabled: true,
PromptID: "dnd.player_handout",
ProfileID: "local-quality",
OutputPath: "artifacts/player_handout.md",
Inputs: map[string]config.ScriptoriumInputConfig{
"transcript": {
Source: "narratio.transcript.trimmed",
Required: true,
},
},
}
env.SelectedAnalyzeArtifacts = []string{"player_handout"}
result, err := (analyzeStage{}).Run(context.Background(), env, m)
if err != nil {
t.Fatalf("Run() error = %v", err)
}
if len(fake.RunRequests) != 1 {
t.Fatalf("run requests = %d, want 1", len(fake.RunRequests))
}
if fake.RunRequests[0].PromptID != "dnd.player_handout" {
t.Fatalf("prompt id = %q, want dnd.player_handout", fake.RunRequests[0].PromptID)
}
if len(result.Outputs) != 1 || result.Outputs[0].Kind != "player_handout" {
t.Fatalf("outputs = %#v, want only player_handout", result.Outputs)
}
}
func TestAnalyzeMetadataIncludesMultipleGeneratedArtifacts(t *testing.T) {
env, m, _ := setupAnalyzeEnv(t)
paths := sessionPathsForEnv(env, m.SessionID)
writeAnalyzeFile(t, filepath.Join(paths.TranscriptsDir, "trimmed.json"), `{"segments":[]}`)
env.Config.Pipeline.Scriptorium.Artifacts["player_handout"] = config.ScriptoriumArtifactConfig{
Enabled: true,
PromptID: "dnd.player_handout",
ProfileID: "local-quality",
OutputPath: "artifacts/player_handout.md",
Inputs: map[string]config.ScriptoriumInputConfig{
"transcript": {
Source: "narratio.transcript.trimmed",
Required: true,
},
},
}
result, err := (analyzeStage{}).Run(context.Background(), env, m)
if err != nil {
t.Fatalf("Run() error = %v", err)
}
generated := mustArtifactEntryList(t, result.Metadata, "generated_artifacts")
if len(generated) != 2 {
t.Fatalf("generated_artifacts len = %d, want 2 (%#v)", len(generated), generated)
}
for i, entry := range generated {
for _, field := range []string{"name", "source_id", "output_kind", "path", "prompt_id", "profile_id", "provenance"} {
if _, ok := entry[field]; !ok {
t.Fatalf("generated[%d] missing field %q: %#v", i, field, entry)
}
}
}
}
func TestAnalyzeFailsWhenRequiredConfiguredArtifactMissing(t *testing.T) {
env, m, _ := setupAnalyzeEnv(t)
paths := sessionPathsForEnv(env, m.SessionID)
writeAnalyzeFile(t, filepath.Join(paths.TranscriptsDir, "trimmed.json"), `{"segments":[]}`)
sessionRecap := env.Config.Pipeline.Scriptorium.Artifacts["session_recap"]
sessionRecap.Inputs["recap"] = config.ScriptoriumInputConfig{
Source: "narratio.artifact.player_handout",
Required: true,
}
env.Config.Pipeline.Scriptorium.Artifacts["session_recap"] = sessionRecap
env.Config.Pipeline.Scriptorium.Artifacts["player_handout"] = config.ScriptoriumArtifactConfig{
Enabled: false,
OutputPath: "artifacts/player_handout.md",
}
_, err := (analyzeStage{}).Run(context.Background(), env, m)
if err == nil {
t.Fatal("expected error, got nil")
}
if !strings.Contains(err.Error(), `configured artifact source "narratio.artifact.player_handout" is unavailable`) {
t.Fatalf("error = %q, want configured artifact unavailable context", err.Error())
}
}
func TestAnalyzeOmitsOptionalMissingConfiguredArtifactInput(t *testing.T) {
env, m, fake := setupAnalyzeEnv(t)
paths := sessionPathsForEnv(env, m.SessionID)
writeAnalyzeFile(t, filepath.Join(paths.TranscriptsDir, "trimmed.json"), `{"segments":[]}`)
sessionRecap := env.Config.Pipeline.Scriptorium.Artifacts["session_recap"]
sessionRecap.Inputs["recap"] = config.ScriptoriumInputConfig{
Source: "narratio.artifact.player_handout",
Required: false,
}
env.Config.Pipeline.Scriptorium.Artifacts["session_recap"] = sessionRecap
env.Config.Pipeline.Scriptorium.Artifacts["player_handout"] = config.ScriptoriumArtifactConfig{
Enabled: false,
OutputPath: "artifacts/player_handout.md",
}
_, err := (analyzeStage{}).Run(context.Background(), env, m)
if err != nil {
t.Fatalf("Run() error = %v", err)
}
if len(fake.RunRequests) != 1 {
t.Fatalf("run requests = %d, want 1", len(fake.RunRequests))
}
if _, exists := fake.RunRequests[0].InputPaths["recap"]; exists {
t.Fatalf("optional recap input should be omitted when unavailable, got %q", fake.RunRequests[0].InputPaths["recap"])
}
}
func TestAnalyzeFailsWhenOutputPathMissing(t *testing.T) {
env, m, fake := setupAnalyzeEnv(t)
paths := env.ArtifactStore.SessionPaths(m.SessionID)
paths := sessionPathsForEnv(env, m.SessionID)
writeAnalyzeFile(t, filepath.Join(paths.TranscriptsDir, "trimmed.json"), `{"segments":[]}`)
artifact := env.Config.Pipeline.Scriptorium.Artifacts["session_recap"]
@@ -425,12 +754,12 @@ func TestAnalyzeFailsWhenTrimmedTranscriptMissing(t *testing.T) {
func TestAnalyzeSupportsProcessedTranscriptSourceWhenConfigured(t *testing.T) {
env, m, fake := setupAnalyzeEnv(t)
paths := env.ArtifactStore.SessionPaths(m.SessionID)
paths := sessionPathsForEnv(env, m.SessionID)
writeAnalyzeFile(t, filepath.Join(paths.TranscriptsDir, "processed.json"), `{"segments":[]}`)
artifact := env.Config.Pipeline.Scriptorium.Artifacts["session_recap"]
artifact.Inputs["transcript"] = config.ScriptoriumInputConfig{
Source: "processed_transcript",
Source: "narratio.transcript.polished",
Required: true,
}
env.Config.Pipeline.Scriptorium.Artifacts["session_recap"] = artifact
@@ -447,15 +776,39 @@ func TestAnalyzeSupportsProcessedTranscriptSourceWhenConfigured(t *testing.T) {
}
}
func TestAnalyzeSupportsCanonicalTrimmedTranscriptSourceWhenConfigured(t *testing.T) {
env, m, fake := setupAnalyzeEnv(t)
paths := sessionPathsForEnv(env, m.SessionID)
writeAnalyzeFile(t, filepath.Join(paths.TranscriptsDir, "trimmed.json"), `{"segments":[]}`)
artifact := env.Config.Pipeline.Scriptorium.Artifacts["session_recap"]
artifact.Inputs["transcript"] = config.ScriptoriumInputConfig{
Source: "narratio.transcript.trimmed",
Required: true,
}
env.Config.Pipeline.Scriptorium.Artifacts["session_recap"] = artifact
_, err := (analyzeStage{}).Run(context.Background(), env, m)
if err != nil {
t.Fatalf("Run() error = %v", err)
}
if len(fake.RunRequests) != 1 {
t.Fatalf("run requests = %d, want 1", len(fake.RunRequests))
}
if fake.RunRequests[0].InputPaths["transcript"] != filepath.Join(paths.TranscriptsDir, "trimmed.json") {
t.Fatalf("transcript input = %q, want trimmed transcript path", fake.RunRequests[0].InputPaths["transcript"])
}
}
func TestAnalyzeSupportsNormalizedTranscriptSourceWhenConfigured(t *testing.T) {
env, m, fake := setupAnalyzeEnv(t)
paths := env.ArtifactStore.SessionPaths(m.SessionID)
paths := sessionPathsForEnv(env, m.SessionID)
normalizedPath := filepath.Join(paths.TranscriptsDir, "normalized.json")
writeAnalyzeFile(t, normalizedPath, `{"segments":[{"id":1}]}`)
artifact := env.Config.Pipeline.Scriptorium.Artifacts["session_recap"]
artifact.Inputs["transcript"] = config.ScriptoriumInputConfig{
Source: "normalized_transcript",
Source: "narratio.transcript.full",
Required: true,
}
env.Config.Pipeline.Scriptorium.Artifacts["session_recap"] = artifact
@@ -472,9 +825,9 @@ func TestAnalyzeSupportsNormalizedTranscriptSourceWhenConfigured(t *testing.T) {
}
}
func TestAnalyzeSupportsNormalizedTranscriptSourceFromManifestOutput(t *testing.T) {
func TestAnalyzeSupportsCanonicalNormalizedTranscriptSourceFromManifestOutput(t *testing.T) {
env, m, fake := setupAnalyzeEnv(t)
paths := env.ArtifactStore.SessionPaths(m.SessionID)
paths := sessionPathsForEnv(env, m.SessionID)
fallbackPath := filepath.Join(paths.TranscriptsDir, "normalized.json")
manifestPath := filepath.Join(paths.ArtifactsDir, "normalized.from-manifest.json")
writeAnalyzeFile(t, fallbackPath, `{"segments":[{"id":999}]}`)
@@ -485,7 +838,37 @@ func TestAnalyzeSupportsNormalizedTranscriptSourceFromManifestOutput(t *testing.
artifact := env.Config.Pipeline.Scriptorium.Artifacts["session_recap"]
artifact.Inputs["transcript"] = config.ScriptoriumInputConfig{
Source: "normalized_transcript",
Source: "narratio.transcript.full",
Required: true,
}
env.Config.Pipeline.Scriptorium.Artifacts["session_recap"] = artifact
_, err := (analyzeStage{}).Run(context.Background(), env, m)
if err != nil {
t.Fatalf("Run() error = %v", err)
}
if len(fake.RunRequests) != 1 {
t.Fatalf("run requests = %d, want 1", len(fake.RunRequests))
}
if fake.RunRequests[0].InputPaths["transcript"] != manifestPath {
t.Fatalf("transcript input = %q, want manifest normalized transcript path", fake.RunRequests[0].InputPaths["transcript"])
}
}
func TestAnalyzeSupportsNormalizedTranscriptSourceFromManifestOutput(t *testing.T) {
env, m, fake := setupAnalyzeEnv(t)
paths := sessionPathsForEnv(env, m.SessionID)
fallbackPath := filepath.Join(paths.TranscriptsDir, "normalized.json")
manifestPath := filepath.Join(paths.ArtifactsDir, "normalized.from-manifest.json")
writeAnalyzeFile(t, fallbackPath, `{"segments":[{"id":999}]}`)
writeAnalyzeFile(t, manifestPath, `{"segments":[{"id":10}]}`)
m.MarkStageSucceeded("normalize", time.Now().UTC(), []manifest.ArtifactRecord{
{Kind: "transcript_normalized", LocalPath: manifestPath},
})
artifact := env.Config.Pipeline.Scriptorium.Artifacts["session_recap"]
artifact.Inputs["transcript"] = config.ScriptoriumInputConfig{
Source: "narratio.transcript.full",
Required: true,
}
env.Config.Pipeline.Scriptorium.Artifacts["session_recap"] = artifact
@@ -506,7 +889,7 @@ func TestAnalyzeFailsWhenNormalizedTranscriptMissing(t *testing.T) {
env, m, fake := setupAnalyzeEnv(t)
artifact := env.Config.Pipeline.Scriptorium.Artifacts["session_recap"]
artifact.Inputs["transcript"] = config.ScriptoriumInputConfig{
Source: "normalized_transcript",
Source: "narratio.transcript.full",
Required: true,
}
env.Config.Pipeline.Scriptorium.Artifacts["session_recap"] = artifact
@@ -528,7 +911,7 @@ func TestAnalyzeFailsWhenNormalizedTranscriptMissing(t *testing.T) {
func TestAnalyzeFailsWhenProcessedTranscriptInvalidJSON(t *testing.T) {
env, m, _ := setupAnalyzeEnv(t)
paths := env.ArtifactStore.SessionPaths(m.SessionID)
paths := sessionPathsForEnv(env, m.SessionID)
writeAnalyzeFile(t, filepath.Join(paths.TranscriptsDir, "trimmed.json"), `{not-json`)
_, err := (analyzeStage{}).Run(context.Background(), env, m)
@@ -542,7 +925,7 @@ func TestAnalyzeFailsWhenProcessedTranscriptInvalidJSON(t *testing.T) {
func TestAnalyzeFailsWhenProcessedTranscriptMissingSegmentsArray(t *testing.T) {
env, m, _ := setupAnalyzeEnv(t)
paths := env.ArtifactStore.SessionPaths(m.SessionID)
paths := sessionPathsForEnv(env, m.SessionID)
writeAnalyzeFile(t, filepath.Join(paths.TranscriptsDir, "trimmed.json"), `{"not_segments":[]}`)
_, err := (analyzeStage{}).Run(context.Background(), env, m)
@@ -556,7 +939,7 @@ func TestAnalyzeFailsWhenProcessedTranscriptMissingSegmentsArray(t *testing.T) {
func TestAnalyzeRecordsRefsAndMetadata(t *testing.T) {
env, m, _ := setupAnalyzeEnv(t)
paths := env.ArtifactStore.SessionPaths(m.SessionID)
paths := sessionPathsForEnv(env, m.SessionID)
writeAnalyzeFile(t, filepath.Join(paths.TranscriptsDir, "trimmed.json"), `{"segments":[]}`)
result, err := (analyzeStage{}).Run(context.Background(), env, m)
@@ -585,7 +968,7 @@ func TestAnalyzeRecordsRefsAndMetadata(t *testing.T) {
func TestAnalyzeHandlesAdapterError(t *testing.T) {
env, m, fake := setupAnalyzeEnv(t)
paths := env.ArtifactStore.SessionPaths(m.SessionID)
paths := sessionPathsForEnv(env, m.SessionID)
writeAnalyzeFile(t, filepath.Join(paths.TranscriptsDir, "trimmed.json"), `{"segments":[]}`)
fake.RunErr = errors.New("adapter boom")
@@ -600,7 +983,7 @@ func TestAnalyzeHandlesAdapterError(t *testing.T) {
func TestAnalyzeHandlesValidationFailedResultAsError(t *testing.T) {
env, m, fake := setupAnalyzeEnv(t)
paths := env.ArtifactStore.SessionPaths(m.SessionID)
paths := sessionPathsForEnv(env, m.SessionID)
writeAnalyzeFile(t, filepath.Join(paths.TranscriptsDir, "trimmed.json"), `{"segments":[]}`)
fake.RunResult = scriptorium.ArtifactResult{
ValidationFailed: true,
@@ -619,7 +1002,7 @@ func TestAnalyzeHandlesValidationFailedResultAsError(t *testing.T) {
func TestAnalyzeSkipsWhenNoEnabledScriptoriumArtifactsConfigured(t *testing.T) {
env, m, _ := setupAnalyzeEnv(t)
paths := env.ArtifactStore.SessionPaths(m.SessionID)
paths := sessionPathsForEnv(env, m.SessionID)
writeAnalyzeFile(t, filepath.Join(paths.TranscriptsDir, "trimmed.json"), `{"segments":[]}`)
env.Config.Pipeline.Scriptorium.Artifacts["session_recap"] = config.ScriptoriumArtifactConfig{
@@ -635,6 +1018,22 @@ func TestAnalyzeSkipsWhenNoEnabledScriptoriumArtifactsConfigured(t *testing.T) {
}
}
func TestAnalyzeSkipsWhenArtifactMapEmpty(t *testing.T) {
env, m, _ := setupAnalyzeEnv(t)
env.Config.Pipeline.Scriptorium.Artifacts = map[string]config.ScriptoriumArtifactConfig{}
result, err := (analyzeStage{}).Run(context.Background(), env, m)
if err != nil {
t.Fatalf("Run() error = %v", err)
}
if result.Metadata["skipped"] != true {
t.Fatalf("metadata = %#v, want skipped=true", result.Metadata)
}
if result.Metadata["reason"] != "no scriptorium artifacts configured" {
t.Fatalf("reason = %#v, want no scriptorium artifacts configured", result.Metadata["reason"])
}
}
func setupAnalyzeEnv(t *testing.T) (*Env, *manifest.Manifest, *scriptorium.FakeRunner) {
t.Helper()
workspace := t.TempDir()
@@ -664,7 +1063,7 @@ func setupAnalyzeEnv(t *testing.T) (*Env, *manifest.Manifest, *scriptorium.FakeR
Timeout: "2m",
Inputs: map[string]config.ScriptoriumInputConfig{
"transcript": {
Source: "trimmed_transcript",
Source: "narratio.transcript.trimmed",
Required: true,
},
"previous_recap": {
@@ -692,7 +1091,7 @@ func setupAnalyzeEnv(t *testing.T) (*Env, *manifest.Manifest, *scriptorium.FakeR
}
store := artifacts.NewLocalStore(workspace)
if _, err := store.EnsureLayout("2026-05-03"); err != nil {
if _, err := store.EnsureLayoutFor("sample-campaign", "2026-05-03"); err != nil {
t.Fatalf("EnsureLayout() error = %v", err)
}
@@ -722,3 +1121,27 @@ func writeAnalyzeFileNoTest(path, contents string) {
_ = os.MkdirAll(filepath.Dir(path), 0o755)
_ = os.WriteFile(path, []byte(contents), 0o644)
}
func mustArtifactEntryList(t *testing.T, metadata map[string]any, key string) []map[string]any {
t.Helper()
raw, ok := metadata[key]
if !ok {
t.Fatalf("metadata missing key %q: %#v", key, metadata)
}
if typed, ok := raw.([]map[string]any); ok {
return typed
}
asList, ok := raw.([]any)
if !ok {
t.Fatalf("metadata[%q] = %#v, want []map[string]any", key, raw)
}
out := make([]map[string]any, 0, len(asList))
for _, item := range asList {
m, ok := item.(map[string]any)
if !ok {
t.Fatalf("metadata[%q] entry = %#v, want map[string]any", key, item)
}
out = append(out, m)
}
return out
}

View File

@@ -3,6 +3,7 @@ package stage
import (
"context"
"encoding/json"
"errors"
"fmt"
"io/fs"
"os"
@@ -33,15 +34,6 @@ var archivePrerequisiteStages = []string{
"analyze",
}
var archiveRunUploadDirs = []string{
"inputs",
"transcripts",
"artifacts",
"reports",
"config",
"logs",
}
func (archiveStage) Name() string { return "archive" }
func (archiveStage) Declares() IODecl {
@@ -87,27 +79,27 @@ func (archiveStage) Run(ctx context.Context, env *Env, m *manifest.Manifest) (*S
return nil, fmt.Errorf("archive: remote object store backend is required when archive run upload is enabled")
}
workDir, err := archiveWorkDir(env, m)
runRoot, err := resolveArchiveRunRoot(env, m)
if err != nil {
return nil, fmt.Errorf("archive: resolve local workdir: %w", err)
return nil, fmt.Errorf("archive: resolve run root: %w", err)
}
workDirInfo, err := os.Stat(workDir)
runRootInfo, err := os.Stat(runRoot)
if err != nil {
return nil, fmt.Errorf("archive: local workdir %q: %w", workDir, err)
return nil, fmt.Errorf("archive: run root %q: %w", runRoot, err)
}
if !workDirInfo.IsDir() {
return nil, fmt.Errorf("archive: local workdir %q is not a directory", workDir)
if !runRootInfo.IsDir() {
return nil, fmt.Errorf("archive: run root %q is not a directory", runRoot)
}
runPrefix, err := archiveRunPrefix(env, m)
runPrefix, err := artifacts.ResolveArchiveRunPrefix(env.Config, m)
if err != nil {
return nil, fmt.Errorf("archive: resolve s3 run prefix: %w", err)
}
sessionPrefix, err := archiveSessionPrefix(env, m)
sessionPrefix, err := artifacts.ResolveArchiveSessionPrefix(env.Config, m)
if err != nil {
return nil, fmt.Errorf("archive: resolve s3 session prefix: %w", err)
}
bucket := archiveBucket(env, m)
bucket := artifacts.ResolveArchiveBucket(env.Config, m)
if bucket == "" {
return nil, fmt.Errorf("archive: resolve s3 bucket: bucket is required")
}
@@ -116,17 +108,21 @@ func (archiveStage) Run(ctx context.Context, env *Env, m *manifest.Manifest) (*S
return nil, fmt.Errorf("archive: run id is required")
}
manifestSource, err := resolveArchiveManifestSource(env, m, workDir)
manifestSource, err := resolveArchiveRunManifestSource(runRoot)
if err != nil {
return nil, fmt.Errorf("archive: resolve manifest source: %w", err)
return nil, fmt.Errorf("archive: resolve run manifest source: %w", err)
}
runFiles, err := collectArchiveRunFiles(workDir, manifestSource)
runFiles, err := collectArchiveRunFiles(runRoot, manifestSource)
if err != nil {
return nil, fmt.Errorf("archive: collect run files: %w", err)
}
workDirCandidates := archiveSourceWorkDirs(env, m, workDir)
promotions, err := resolveArchivePromotions(workDirCandidates, env.Config.Pipeline.Archive.PromoteArtifacts)
sessionPaths := archiveSessionPaths(env, m)
runtimeCatalog, err := buildArchiveRuntimeArtifactCatalog(sessionPaths, env.Config.Pipeline.Scriptorium)
if err != nil {
return nil, fmt.Errorf("archive: build runtime artifact catalog: %w", err)
}
promotions, skippedOptional, err := resolveArchivePromotions(sessionPaths, m, runtimeCatalog, env.Config.Pipeline.Archive.PromoteArtifacts)
if err != nil {
return nil, fmt.Errorf("archive: resolve promotion rules: %w", err)
}
@@ -140,23 +136,15 @@ func (archiveStage) Run(ctx context.Context, env *Env, m *manifest.Manifest) (*S
}
promotedUploaded := make([]string, 0, len(promotions))
skippedOptional := make([]string, 0)
for _, promotion := range promotions {
if !promotion.Exists {
if promotion.Required {
return nil, fmt.Errorf("archive: required promotion source missing: %q", promotion.From)
}
skippedOptional = append(skippedOptional, promotion.To)
continue
}
key := artifacts.S3PromotedArtifactKey(sessionPrefix, promotion.To)
key := artifacts.S3PromotedArtifactKey(sessionPrefix, promotion.Dest)
if _, err := env.ObjectStore.Upload(ctx, promotion.LocalPath, key, storage.UploadOptions{}); err != nil {
return nil, fmt.Errorf("archive: upload promoted output %q to %q: %w", promotion.From, key, err)
return nil, fmt.Errorf("archive: upload promoted output source %q to %q: %w", promotion.Source, key, err)
}
promotedUploaded = append(promotedUploaded, promotion.To)
promotedUploaded = append(promotedUploaded, promotion.Dest)
}
currentManifestKey := artifacts.S3CurrentManifestKey(sessionPrefix)
currentManifestKey, currentRunPointerKey := artifacts.ResolveArchiveCurrentStateKeys(sessionPrefix)
manifestTempPath, err := writeCurrentManifestSnapshot(m, archiveMetadataPreview(
bucket,
runPrefix,
@@ -177,7 +165,6 @@ func (archiveStage) Run(ctx context.Context, env *Env, m *manifest.Manifest) (*S
return nil, fmt.Errorf("archive: upload current manifest to %q: %w", currentManifestKey, err)
}
currentRunPointerKey := artifacts.S3CurrentRunPointerKey(sessionPrefix)
runIDTempPath, err := writeCurrentRunIDPointer(runID)
if err != nil {
return nil, fmt.Errorf("archive: build current run id pointer: %w", err)
@@ -210,11 +197,11 @@ func (archiveStage) Run(ctx context.Context, env *Env, m *manifest.Manifest) (*S
}
type archivePromotion struct {
From string
To string
Required bool
LocalPath string
Exists bool
Source string
Dest string
Required bool
LocalPath string
Provenance string
}
func archiveDisabled(env *Env) bool {
@@ -249,213 +236,236 @@ func validateArchivePrerequisites(m *manifest.Manifest) error {
return nil
}
func archiveWorkDir(env *Env, m *manifest.Manifest) (string, error) {
workDir := strings.TrimSpace(m.LocalWorkDir)
if workDir != "" {
cleaned := filepath.Clean(workDir)
if info, err := os.Stat(cleaned); err == nil && info.IsDir() {
return cleaned, nil
}
}
sessionID := strings.TrimSpace(env.Config.Session.SessionID)
if sessionID == "" {
sessionID = strings.TrimSpace(m.SessionID)
}
campaign := strings.TrimSpace(env.Config.Session.Campaign)
if campaign == "" {
campaign = strings.TrimSpace(m.Campaign)
}
runID := strings.TrimSpace(m.RunID)
if runID == "" {
return "", fmt.Errorf("run id is required")
}
if campaign == "" || sessionID == "" {
return "", fmt.Errorf("campaign and session id are required")
}
runScoped := artifacts.SessionRunWorkDir(env.Config.Pipeline.Workspace.Root, campaign, sessionID, runID)
if info, err := os.Stat(runScoped); err == nil && info.IsDir() {
return runScoped, nil
}
legacy := artifacts.SessionWorkDir(env.Config.Pipeline.Workspace.Root, sessionID)
return legacy, nil
}
func archiveRunPrefix(env *Env, m *manifest.Manifest) (string, error) {
runPrefix := strings.TrimSpace(m.S3RunPrefix)
if runPrefix != "" {
return runPrefix, nil
}
sessionPrefix, err := archiveSessionPrefix(env, m)
if err != nil {
return "", err
}
runID := strings.TrimSpace(m.RunID)
if runID == "" {
return "", fmt.Errorf("run id is required")
}
return artifacts.S3RunPrefix(sessionPrefix, runID), nil
}
func archiveSessionPrefix(env *Env, m *manifest.Manifest) (string, error) {
if m != nil && strings.TrimSpace(m.S3SessionPrefix) != "" {
return strings.TrimSpace(m.S3SessionPrefix), nil
}
sessionID := strings.TrimSpace(env.Config.Session.SessionID)
if sessionID == "" {
sessionID = strings.TrimSpace(m.SessionID)
}
campaign := strings.TrimSpace(env.Config.Session.Campaign)
if campaign == "" {
campaign = strings.TrimSpace(m.Campaign)
}
if env.Config.Pipeline.Storage.S3 == nil {
return "", fmt.Errorf("pipeline.storage.s3 configuration is required")
}
sessionPrefix := artifacts.S3SessionPrefix(env.Config.Pipeline.Storage.S3.RootPrefix, campaign, sessionID)
if strings.TrimSpace(sessionPrefix) == "" {
return "", fmt.Errorf("session prefix is required")
}
return sessionPrefix, nil
}
func archiveBucket(env *Env, m *manifest.Manifest) string {
if m != nil && strings.TrimSpace(m.S3Bucket) != "" {
return strings.TrimSpace(m.S3Bucket)
}
if env == nil || env.Config == nil || env.Config.Pipeline == nil || env.Config.Pipeline.Storage.S3 == nil {
return ""
}
return strings.TrimSpace(env.Config.Pipeline.Storage.S3.Bucket)
}
func resolveArchivePromotions(workDirs []string, rules []config.ArchivePromotionRule) ([]archivePromotion, error) {
if len(workDirs) == 0 {
return nil, fmt.Errorf("at least one workdir candidate is required")
}
out := make([]archivePromotion, 0, len(rules))
for _, rule := range rules {
from := strings.TrimSpace(rule.From)
to := strings.TrimSpace(rule.To)
required := rule.Required == nil || *rule.Required
var (
localPath string
exists bool
)
for _, candidateRoot := range workDirs {
resolvedPath, err := resolveWorkDirRelativePath(candidateRoot, from)
if err != nil {
return nil, fmt.Errorf("promotion from %q: %w", from, err)
}
info, err := os.Stat(resolvedPath)
if err == nil && !info.IsDir() {
localPath = resolvedPath
exists = true
break
}
if err != nil && !os.IsNotExist(err) {
return nil, fmt.Errorf("promotion source %q: %w", from, err)
}
if localPath == "" {
localPath = resolvedPath
}
}
out = append(out, archivePromotion{
From: from,
To: to,
Required: required,
LocalPath: localPath,
Exists: exists,
})
}
return out, nil
}
func archiveSourceWorkDirs(env *Env, m *manifest.Manifest, runWorkDir string) []string {
candidates := make([]string, 0, 2)
if strings.TrimSpace(runWorkDir) != "" {
candidates = append(candidates, filepath.Clean(runWorkDir))
}
func resolveArchiveRunRoot(env *Env, m *manifest.Manifest) (string, error) {
sessionID := strings.TrimSpace(env.Config.Session.SessionID)
if sessionID == "" && m != nil {
sessionID = strings.TrimSpace(m.SessionID)
}
if sessionID != "" {
sessionWorkDir := filepath.Clean(artifacts.SessionWorkDir(env.Config.Pipeline.Workspace.Root, sessionID))
if sessionWorkDir != "" {
candidates = append(candidates, sessionWorkDir)
}
campaign := strings.TrimSpace(env.Config.Session.Campaign)
if campaign == "" && m != nil {
campaign = strings.TrimSpace(m.Campaign)
}
seen := make(map[string]struct{}, len(candidates))
out := make([]string, 0, len(candidates))
for _, c := range candidates {
if _, ok := seen[c]; ok {
continue
}
seen[c] = struct{}{}
out = append(out, c)
runID := ""
if m != nil {
runID = strings.TrimSpace(m.RunID)
}
if sessionID == "" || campaign == "" {
return "", fmt.Errorf("campaign and session id are required")
}
if runID == "" {
return "", fmt.Errorf("run id is required")
}
return out
}
func resolveWorkDirRelativePath(workDir, rel string) (string, error) {
rel = filepath.Clean(filepath.FromSlash(strings.TrimSpace(rel)))
if rel == "." || rel == "" {
return "", fmt.Errorf("relative path is required")
}
full := filepath.Join(workDir, rel)
cleanedWork := filepath.Clean(workDir)
cleanedFull := filepath.Clean(full)
relative, err := filepath.Rel(cleanedWork, cleanedFull)
canonical := filepath.Clean(artifacts.SessionRunRootForCampaign(env.Config.Pipeline.Workspace.Root, campaign, sessionID, runID))
canonicalExists, err := directoryExists(canonical)
if err != nil {
return "", fmt.Errorf("compute relative path: %w", err)
return "", fmt.Errorf("check canonical run root %q: %w", canonical, err)
}
if relative == ".." || strings.HasPrefix(relative, ".."+string(filepath.Separator)) {
return "", fmt.Errorf("path escapes workdir")
if !canonicalExists {
return "", fmt.Errorf("run root not found for campaign %q session %q run %q at canonical path %q", campaign, sessionID, runID, canonical)
}
return cleanedFull, nil
return canonical, nil
}
func collectArchiveRunFiles(workDir, manifestPath string) ([]archiveUploadFile, error) {
files := make([]archiveUploadFile, 0, 64)
func resolveArchiveSessionRoot(env *Env, m *manifest.Manifest) (string, error) {
sessionID := strings.TrimSpace(env.Config.Session.SessionID)
if sessionID == "" && m != nil {
sessionID = strings.TrimSpace(m.SessionID)
}
campaign := strings.TrimSpace(env.Config.Session.Campaign)
if campaign == "" && m != nil {
campaign = strings.TrimSpace(m.Campaign)
}
if sessionID == "" {
return "", fmt.Errorf("session id is required")
}
if campaign == "" {
return "", fmt.Errorf("campaign is required")
}
return filepath.Clean(artifacts.SessionWorkDirForCampaign(env.Config.Pipeline.Workspace.Root, campaign, sessionID)), nil
}
for _, dirName := range archiveRunUploadDirs {
fullDir := filepath.Join(workDir, dirName)
info, err := os.Stat(fullDir)
func archiveSessionPaths(env *Env, m *manifest.Manifest) artifacts.SessionPaths {
sessionID := strings.TrimSpace(env.Config.Session.SessionID)
if sessionID == "" && m != nil {
sessionID = strings.TrimSpace(m.SessionID)
}
campaign := strings.TrimSpace(env.Config.Session.Campaign)
if campaign == "" && m != nil {
campaign = strings.TrimSpace(m.Campaign)
}
store := artifacts.NewLocalStore(env.Config.Pipeline.Workspace.Root)
return store.SessionPathsFor(campaign, sessionID)
}
func resolveArchivePromotions(
paths artifacts.SessionPaths,
m *manifest.Manifest,
catalog *artifacts.ArtifactCatalog,
rules []config.ArchivePromotionRule,
) ([]archivePromotion, []string, error) {
out := make([]archivePromotion, 0, len(rules))
skippedOptional := make([]string, 0)
for _, rule := range rules {
source := strings.TrimSpace(rule.Source)
required := rule.Required == nil || *rule.Required
dest, err := resolveArchivePromotionDest(rule, catalog)
if err != nil {
if os.IsNotExist(err) {
return nil, nil, fmt.Errorf("source %q: %w", source, err)
}
resolved, err := artifacts.ResolveSessionArtifactWithCatalog(paths, m, source, catalog)
if err != nil {
if errors.Is(err, artifacts.ErrSessionArtifactNotFound) && !required {
skippedOptional = append(skippedOptional, dest)
continue
}
return nil, fmt.Errorf("stat %q: %w", fullDir, err)
if errors.Is(err, artifacts.ErrSessionArtifactNotFound) {
return nil, nil, fmt.Errorf("required promotion source unavailable: %q", source)
}
return nil, nil, fmt.Errorf("resolve source %q: %w", source, err)
}
if !info.IsDir() {
out = append(out, archivePromotion{
Source: source,
Dest: dest,
Required: required,
LocalPath: resolved.Path,
Provenance: resolved.Provenance,
})
}
return out, skippedOptional, nil
}
func resolveArchivePromotionDest(rule config.ArchivePromotionRule, catalog *artifacts.ArtifactCatalog) (string, error) {
dest := strings.TrimSpace(rule.Dest)
if dest == "" {
entry, ok := catalog.Lookup(strings.TrimSpace(rule.Source))
if !ok {
return "", fmt.Errorf("destination omitted and source is unknown")
}
dest = strings.TrimSpace(entry.CanonicalRelPath)
if dest == "" {
return "", fmt.Errorf("destination omitted and no canonical destination is available")
}
}
return normalizeArchiveRelativePath(dest)
}
func normalizeArchiveRelativePath(rel string) (string, error) {
trimmed := strings.TrimSpace(rel)
if trimmed == "" {
return "", fmt.Errorf("relative path is required")
}
cleaned := filepath.ToSlash(filepath.Clean(filepath.FromSlash(trimmed)))
if cleaned == "." || cleaned == "" {
return "", fmt.Errorf("relative path is required")
}
if filepath.IsAbs(trimmed) || strings.HasPrefix(cleaned, "/") || cleaned == ".." || strings.HasPrefix(cleaned, "../") {
return "", fmt.Errorf("path must be a clean relative path")
}
return cleaned, nil
}
func buildArchiveRuntimeArtifactCatalog(
paths artifacts.SessionPaths,
scriptoriumCfg *config.ScriptoriumConfig,
) (*artifacts.ArtifactCatalog, error) {
catalog := artifacts.NewArtifactCatalog()
if err := catalog.RegisterBuiltIns(); err != nil {
return nil, err
}
if scriptoriumCfg == nil {
return catalog, nil
}
configured := map[string]artifacts.ConfiguredArtifactDefinition{}
for key, artifactCfg := range scriptoriumCfg.Artifacts {
configured[key] = artifacts.ConfiguredArtifactDefinition{
Enabled: artifactCfg.Enabled,
OutputPath: artifactCfg.OutputPath,
}
}
if err := catalog.RegisterConfiguredArtifacts(configured, nil); err != nil {
return nil, err
}
for _, entry := range catalog.ListConfigured() {
if strings.TrimSpace(entry.CanonicalRelPath) == "" {
continue
}
if err := filepath.WalkDir(fullDir, func(path string, d fs.DirEntry, walkErr error) error {
if walkErr != nil {
return walkErr
localPath, err := resolveConfiguredArtifactLocalPath(paths, entry.CanonicalRelPath)
if err != nil {
continue
}
info, statErr := os.Stat(localPath)
if statErr != nil {
if os.IsNotExist(statErr) {
continue
}
if d.IsDir() {
return nil, fmt.Errorf("stat configured artifact %q: %w", entry.SourceID, statErr)
}
if info.IsDir() {
continue
}
if err := catalog.MarkAvailableFromDisk(entry.SourceID, localPath); err != nil {
return nil, err
}
}
return catalog, nil
}
func resolveConfiguredArtifactLocalPath(paths artifacts.SessionPaths, configured string) (string, error) {
outputPath := strings.TrimSpace(configured)
if outputPath == "" {
return "", fmt.Errorf("configured artifact output path is required")
}
if filepath.IsAbs(outputPath) {
return filepath.Clean(outputPath), nil
}
rel := filepath.Clean(outputPath)
if rel == "." || rel == "" {
return "", fmt.Errorf("relative output path is required")
}
if rel == ".." || strings.HasPrefix(rel, ".."+string(filepath.Separator)) {
return "", fmt.Errorf("relative output path escapes session root: %q", configured)
}
return filepath.Join(paths.Root, rel), nil
}
func collectArchiveRunFiles(runRoot, manifestPath string) ([]archiveUploadFile, error) {
files := make([]archiveUploadFile, 0, 64)
err := filepath.WalkDir(runRoot, func(path string, d fs.DirEntry, walkErr error) error {
if walkErr != nil {
return walkErr
}
if d.IsDir() {
if path == runRoot {
return nil
}
rel, err := filepath.Rel(workDir, path)
relDir, err := filepath.Rel(runRoot, path)
if err != nil {
return fmt.Errorf("relative path from %q to %q: %w", workDir, path, err)
return fmt.Errorf("relative dir from %q to %q: %w", runRoot, path, err)
}
relDir = filepath.ToSlash(relDir)
// Preserve existing behavior: audio is not uploaded in archive run record.
if relDir == "audio" || strings.HasPrefix(relDir, "audio/") {
return filepath.SkipDir
}
rel = filepath.ToSlash(rel)
files = append(files, archiveUploadFile{
RelativePath: rel,
LocalPath: path,
})
return nil
}); err != nil {
return nil, fmt.Errorf("walk %q: %w", fullDir, err)
}
rel, err := filepath.Rel(runRoot, path)
if err != nil {
return fmt.Errorf("relative path from %q to %q: %w", runRoot, path, err)
}
rel = filepath.ToSlash(rel)
files = append(files, archiveUploadFile{
RelativePath: rel,
LocalPath: path,
})
return nil
})
if err != nil {
return nil, fmt.Errorf("walk %q: %w", runRoot, err)
}
manifestInfo, err := os.Stat(manifestPath)
@@ -472,6 +482,14 @@ func collectArchiveRunFiles(workDir, manifestPath string) ([]archiveUploadFile,
RelativePath: "manifest.json",
LocalPath: manifestPath,
})
seen := map[string]archiveUploadFile{}
for _, file := range files {
seen[file.RelativePath] = file
}
files = files[:0]
for _, file := range seen {
files = append(files, file)
}
sort.Slice(files, func(i, j int) bool {
return files[i].RelativePath < files[j].RelativePath
@@ -479,39 +497,30 @@ func collectArchiveRunFiles(workDir, manifestPath string) ([]archiveUploadFile,
return files, nil
}
func resolveArchiveManifestSource(env *Env, m *manifest.Manifest, workDir string) (string, error) {
candidates := make([]string, 0, 3)
candidates = append(candidates, filepath.Join(workDir, "manifest.json"))
func resolveArchiveRunManifestSource(runRoot string) (string, error) {
path := filepath.Join(filepath.Clean(runRoot), "manifest.json")
info, err := os.Stat(path)
if err != nil {
if os.IsNotExist(err) {
return "", fmt.Errorf("manifest.json not found in run root %q", runRoot)
}
return "", fmt.Errorf("stat %q: %w", path, err)
}
if info.IsDir() {
return "", fmt.Errorf("manifest path %q is a directory", path)
}
return path, nil
}
sessionID := strings.TrimSpace(env.Config.Session.SessionID)
if sessionID == "" && m != nil {
sessionID = strings.TrimSpace(m.SessionID)
func directoryExists(path string) (bool, error) {
info, err := os.Stat(path)
if err == nil {
return info.IsDir(), nil
}
if sessionID != "" {
sessionManifest := filepath.Join(artifacts.SessionWorkDir(env.Config.Pipeline.Workspace.Root, sessionID), "manifest.json")
candidates = append(candidates, sessionManifest)
if os.IsNotExist(err) {
return false, nil
}
seen := make(map[string]struct{}, len(candidates))
for _, candidate := range candidates {
clean := filepath.Clean(candidate)
if _, ok := seen[clean]; ok {
continue
}
seen[clean] = struct{}{}
info, err := os.Stat(clean)
if err != nil {
if os.IsNotExist(err) {
continue
}
return "", fmt.Errorf("stat %q: %w", clean, err)
}
if info.IsDir() {
continue
}
return clean, nil
}
return "", fmt.Errorf("manifest.json not found in run workdir or session workdir")
return false, err
}
func writeCurrentManifestSnapshot(m *manifest.Manifest, archiveMetadata map[string]any) (string, error) {

View File

@@ -73,14 +73,15 @@ func TestArchiveUploadsRunRecordPromotionsAndCurrentPointer(t *testing.T) {
runPrefix := m.S3RunPrefix
sessionPrefix := m.S3SessionPrefix
wantRunUploads := []string{
"artifacts/session_recap.md",
"config/audita.generated.yml",
"inputs/session.yml",
"analyze/outputs/artifacts/session_recap.md",
"merge/config/seriatim.generated.yml",
"prepare/inputs/session.yml",
"prepare/outputs/audio/speaker.flac",
"logs/audita.stderr.log",
"manifest.json",
"reports/audita.report.json",
"transcripts/raw/speaker.json",
"transcripts/trimmed.json",
"polish/reports/audita.report.json",
"transcribe/outputs/transcripts/raw/speaker.json",
"trim/outputs/transcripts/trimmed.json",
}
for _, rel := range wantRunUploads {
key := runPrefix + rel
@@ -132,12 +133,11 @@ func TestArchiveUploadsRunRecordPromotionsAndCurrentPointer(t *testing.T) {
}
func TestArchiveUsesCustomPromotionRules(t *testing.T) {
env, m, workDir := archiveFixture(t)
env, m, _ := archiveFixture(t)
env.Config.Pipeline.Archive.PromoteArtifacts = []config.ArchivePromotionRule{
{From: "transcripts/trimmed.json", To: "published/trimmed.json", Required: boolPtr(true)},
{From: "artifacts/session_recap.md", To: "published/recap.md", Required: boolPtr(true)},
{Source: "narratio.transcript.trimmed", Dest: "published/trimmed.json", Required: boolPtr(true)},
{Source: "narratio.artifact.session_recap", Dest: "published/recap.md", Required: boolPtr(true)},
}
writeStageTestFile(t, filepath.Join(workDir, "published", "ignored.txt"), "ignore\n")
_, err := archiveStage{}.Run(context.Background(), env, m)
if err != nil {
@@ -156,8 +156,8 @@ func TestArchiveUsesCustomPromotionRules(t *testing.T) {
func TestArchiveSkipsOptionalMissingPromotion(t *testing.T) {
env, m, _ := archiveFixture(t)
env.Config.Pipeline.Archive.PromoteArtifacts = []config.ArchivePromotionRule{
{From: "transcripts/trimmed.json", To: "transcripts/trimmed.json", Required: boolPtr(true)},
{From: "artifacts/optional.md", To: "artifacts/optional.md", Required: boolPtr(false)},
{Source: "narratio.transcript.trimmed", Dest: "transcripts/trimmed.json", Required: boolPtr(true)},
{Source: "narratio.transcript.merged", Dest: "transcripts/merged.json", Required: boolPtr(false)},
}
result, err := archiveStage{}.Run(context.Background(), env, m)
@@ -165,7 +165,7 @@ func TestArchiveSkipsOptionalMissingPromotion(t *testing.T) {
t.Fatalf("Run() error = %v", err)
}
got, _ := result.Metadata["skipped_optional_promotions"].([]string)
want := []string{"artifacts/optional.md"}
want := []string{"transcripts/merged.json"}
if !reflect.DeepEqual(got, want) {
t.Fatalf("skipped_optional_promotions = %#v, want %#v", got, want)
}
@@ -174,35 +174,27 @@ func TestArchiveSkipsOptionalMissingPromotion(t *testing.T) {
func TestArchiveFailsWhenRequiredPromotionMissing(t *testing.T) {
env, m, _ := archiveFixture(t)
env.Config.Pipeline.Archive.PromoteArtifacts = []config.ArchivePromotionRule{
{From: "artifacts/missing.md", To: "artifacts/missing.md", Required: boolPtr(true)},
{Source: "narratio.transcript.merged", Dest: "transcripts/merged.json", Required: boolPtr(true)},
}
_, err := archiveStage{}.Run(context.Background(), env, m)
if err == nil || !strings.Contains(err.Error(), "required promotion source missing") {
t.Fatalf("Run() error = %v, want required promotion missing failure", err)
if err == nil || !strings.Contains(err.Error(), "required promotion source unavailable") {
t.Fatalf("Run() error = %v, want required promotion source unavailable failure", err)
}
}
func TestArchivePromotionFallsBackToSessionWorkDir(t *testing.T) {
env, m, runWorkDir := archiveFixture(t)
sessionWorkDir := artifacts.SessionWorkDir(env.Config.Pipeline.Workspace.Root, m.SessionID)
sessionTrimmed := filepath.Join(sessionWorkDir, "transcripts", "trimmed.json")
func TestArchiveFailsWhenCanonicalRunRootMissing(t *testing.T) {
env, m, runRoot := archiveFixture(t)
if err := os.RemoveAll(runRoot); err != nil {
t.Fatalf("remove run root: %v", err)
}
if err := os.Remove(filepath.Join(runWorkDir, "transcripts", "trimmed.json")); err != nil {
t.Fatalf("remove run scoped trimmed transcript: %v", err)
_, err := archiveStage{}.Run(context.Background(), env, m)
if err == nil {
t.Fatal("expected missing run-root error, got nil")
}
writeStageTestFile(t, sessionTrimmed, "{}\n")
result, err := archiveStage{}.Run(context.Background(), env, m)
if err != nil {
t.Fatalf("Run() error = %v", err)
}
if result.Metadata["promoted_files_uploaded"] != 2 {
t.Fatalf("metadata promoted_files_uploaded = %#v, want 2", result.Metadata["promoted_files_uploaded"])
}
fake := env.ObjectStore.(*storage.FakeBackend)
if _, ok := fake.Objects[m.S3SessionPrefix+"transcripts/trimmed.json"]; !ok {
t.Fatalf("missing promoted trimmed key from session fallback")
if !strings.Contains(err.Error(), "run root not found") {
t.Fatalf("error = %v, want missing run-root error", err)
}
}
@@ -264,22 +256,27 @@ func archiveFixture(t *testing.T) (*Env, *manifest.Manifest, string) {
runID := "20260516T010203Z-1a2b3c4d"
campaign := "forsaken"
sessionID := "2026-04-19"
workDir := filepath.Join(root, "work", campaign, sessionID, runID)
sessionRoot := artifacts.SessionWorkDirForCampaign(root, campaign, sessionID)
runRoot := artifacts.SessionRunRootForCampaign(root, campaign, sessionID, runID)
writeStageTestFile(t, filepath.Join(workDir, "inputs", "session.yml"), "session_id: 2026-04-19\n")
writeStageTestFile(t, filepath.Join(workDir, "transcripts", "raw", "speaker.json"), "{}\n")
writeStageTestFile(t, filepath.Join(workDir, "transcripts", "trimmed.json"), "{}\n")
writeStageTestFile(t, filepath.Join(workDir, "artifacts", "session_recap.md"), "# recap\n")
writeStageTestFile(t, filepath.Join(workDir, "reports", "audita.report.json"), "{}\n")
writeStageTestFile(t, filepath.Join(workDir, "config", "audita.generated.yml"), "key: value\n")
writeStageTestFile(t, filepath.Join(workDir, "logs", "audita.stderr.log"), "stderr\n")
writeStageTestFile(t, filepath.Join(workDir, "audio", "speaker.flac"), "flac")
writeStageTestFile(t, filepath.Join(workDir, "manifest.json"), "{}\n")
writeStageTestFile(t, filepath.Join(sessionRoot, "transcripts", "trimmed.json"), "{\"segments\":[]}\n")
writeStageTestFile(t, filepath.Join(sessionRoot, "artifacts", "session_recap.md"), "# recap\n")
writeStageTestFile(t, filepath.Join(runRoot, "prepare", "inputs", "session.yml"), "session_id: 2026-04-19\n")
writeStageTestFile(t, filepath.Join(runRoot, "prepare", "outputs", "audio", "speaker.flac"), "flac\n")
writeStageTestFile(t, filepath.Join(runRoot, "transcribe", "outputs", "transcripts", "raw", "speaker.json"), "{}\n")
writeStageTestFile(t, filepath.Join(runRoot, "trim", "outputs", "transcripts", "trimmed.json"), "{\"segments\":[]}\n")
writeStageTestFile(t, filepath.Join(runRoot, "analyze", "outputs", "artifacts", "session_recap.md"), "# recap\n")
writeStageTestFile(t, filepath.Join(runRoot, "polish", "reports", "audita.report.json"), "{}\n")
writeStageTestFile(t, filepath.Join(runRoot, "merge", "config", "seriatim.generated.yml"), "key: value\n")
writeStageTestFile(t, filepath.Join(runRoot, "logs", "audita.stderr.log"), "stderr\n")
writeStageTestFile(t, filepath.Join(runRoot, "audio", "speaker.flac"), "flac")
writeStageTestFile(t, filepath.Join(runRoot, "manifest.json"), "{}\n")
m := manifest.New(sessionID, time.Date(2026, 5, 16, 1, 2, 3, 0, time.UTC))
m.Campaign = campaign
m.RunID = runID
m.LocalWorkDir = workDir
m.LocalWorkDir = runRoot
m.S3Bucket = "my-dnd-archive"
m.S3SessionPrefix = artifacts.S3SessionPrefix("dnd", campaign, sessionID)
m.S3RunPrefix = artifacts.S3RunPrefix(m.S3SessionPrefix, runID)
@@ -301,8 +298,17 @@ func archiveFixture(t *testing.T) (*Env, *manifest.Manifest, string) {
Enabled: boolPtr(true),
UploadRun: boolPtr(true),
PromoteArtifacts: []config.ArchivePromotionRule{
{From: "transcripts/trimmed.json", To: "transcripts/trimmed.json", Required: boolPtr(true)},
{From: "artifacts/session_recap.md", To: "artifacts/session_recap.md", Required: boolPtr(true)},
{Source: "narratio.transcript.trimmed", Dest: "transcripts/trimmed.json", Required: boolPtr(true)},
{Source: "narratio.artifact.session_recap", Dest: "artifacts/session_recap.md", Required: boolPtr(true)},
},
},
Scriptorium: &config.ScriptoriumConfig{
Artifacts: map[string]config.ScriptoriumArtifactConfig{
"session_recap": {
Enabled: true,
PromptID: "dnd.session_recap",
OutputPath: "artifacts/session_recap.md",
},
},
},
},
@@ -313,7 +319,7 @@ func archiveFixture(t *testing.T) (*Env, *manifest.Manifest, string) {
},
ObjectStore: &storage.FakeBackend{},
}
return env, m, workDir
return env, m, runRoot
}
type promotionFailingStore struct {

View File

@@ -57,7 +57,11 @@ func (mergeStage) Run(ctx context.Context, env *Env, m *manifest.Manifest) (*Sta
return nil, fmt.Errorf("merge: session id is required")
}
paths := env.ArtifactStore.SessionPaths(sessionID)
paths := sessionPathsForEnv(env, sessionID)
runLayout, err := resolveRunStageLayout(env, m, paths, sessionID, "merge")
if err != nil {
return nil, fmt.Errorf("merge: resolve run-stage layout: %w", err)
}
inputs, err := discoverRawTranscripts(m, paths)
if err != nil {
@@ -81,13 +85,29 @@ func (mergeStage) Run(ctx context.Context, env *Env, m *manifest.Manifest) (*Sta
return nil, fmt.Errorf("merge: %w", err)
}
mergedPath := filepath.Join(paths.TranscriptsDir, "merged.json")
reportPath := filepath.Join(paths.ArtifactsDir, "seriatim.report.json")
canonicalMergedPath := filepath.Join(paths.TranscriptsDir, "merged.json")
mergedPath, err := runLocalPathForCanonical(runLayout, paths, canonicalMergedPath)
if err != nil {
return nil, fmt.Errorf("merge: resolve run-local merged transcript path: %w", err)
}
canonicalReportPath := filepath.Join(paths.ArtifactsDir, "seriatim.report.json")
reportPath := canonicalReportPath
if runLayout.Enabled {
reportPath, err = runLocalPathForCanonical(runLayout, paths, canonicalReportPath)
if err != nil {
return nil, fmt.Errorf("merge: resolve run-local report path: %w", err)
}
}
stdoutPath := filepath.Join(paths.LogsDir, "seriatim.stdout.log")
stderrPath := filepath.Join(paths.LogsDir, "seriatim.stderr.log")
genCfgPath := filepath.Join(paths.ConfigDir, "seriatim.generated.yml")
if runLayout.Enabled {
stdoutPath = filepath.Join(runLayout.LogsDir, "seriatim.stdout.log")
stderrPath = filepath.Join(runLayout.LogsDir, "seriatim.stderr.log")
genCfgPath = filepath.Join(runLayout.ConfigDir, "seriatim.generated.yml")
}
normalizedInputs, normalizeLogs, normalizeConfigs, normalizeMeta, err := normalizeMergeInputs(ctx, env, inputs, paths)
normalizedInputs, normalizeLogs, normalizeConfigs, normalizeMeta, err := normalizeMergeInputs(ctx, env, inputs, paths, runLayout)
if err != nil {
return nil, err
}
@@ -130,25 +150,35 @@ func (mergeStage) Run(ctx context.Context, env *Env, m *manifest.Manifest) (*Sta
}
}
outputs := []artifacts.Ref{{
Kind: "transcript_merged",
Category: "transcripts",
SessionID: sessionID,
AbsolutePath: finalMergedPath,
}}
promotedMerged, err := promoteRunLocalOutput(env.ArtifactStore, finalMergedPath, canonicalMergedPath, artifacts.Ref{
Kind: "transcript_merged",
Category: "transcripts",
SessionID: sessionID,
})
if err != nil {
return nil, fmt.Errorf("merge: promote merged transcript: %w", err)
}
outputs := []artifacts.Ref{promotedMerged}
if reportEnabled {
outputs = append(outputs, artifacts.Ref{
Kind: "seriatim_report",
Category: "artifacts",
SessionID: sessionID,
AbsolutePath: finalReportPath,
promotedReport, err := promoteRunLocalOutput(env.ArtifactStore, finalReportPath, canonicalReportPath, artifacts.Ref{
Kind: "seriatim_report",
Category: "artifacts",
SessionID: sessionID,
})
if err != nil {
return nil, fmt.Errorf("merge: promote report: %w", err)
}
outputs = append(outputs, promotedReport)
}
coalesceGap := any(nil)
if env.Config.Pipeline.Seriatim.CoalesceGap != nil {
coalesceGap = *env.Config.Pipeline.Seriatim.CoalesceGap
}
reportCanonicalPath := ""
if reportEnabled {
reportCanonicalPath = canonicalReportPath
}
meta := map[string]any{
"stage": "merge",
@@ -160,8 +190,10 @@ func (mergeStage) Run(ctx context.Context, env *Env, m *manifest.Manifest) (*Sta
"output_schema": env.Config.Pipeline.Seriatim.OutputSchema,
"coalesce_gap": coalesceGap,
"report_enabled": reportEnabled,
"output_path": finalMergedPath,
"report_path": finalReportPath,
"run_output_path": finalMergedPath,
"output_path": canonicalMergedPath,
"run_report_path": finalReportPath,
"report_path": reportCanonicalPath,
"timeout": env.Config.Pipeline.Seriatim.Timeout,
"binary": env.Config.Pipeline.Seriatim.Binary,
"adapter_duration_ms": res.Duration.Milliseconds(),
@@ -201,12 +233,21 @@ type normalizeMergeInputMeta struct {
AdapterOutputPath string `json:"adapter_output_path,omitempty"`
}
func normalizeMergeInputs(ctx context.Context, env *Env, rawInputs []string, paths artifacts.SessionPaths) ([]string, []string, []string, []normalizeMergeInputMeta, error) {
func normalizeMergeInputs(
ctx context.Context,
env *Env,
rawInputs []string,
paths artifacts.SessionPaths,
runLayout runStageLayout,
) ([]string, []string, []string, []normalizeMergeInputMeta, error) {
normalizedInputs := make([]string, 0, len(rawInputs))
logs := make([]string, 0, len(rawInputs)*2)
configs := make([]string, 0, len(rawInputs))
meta := make([]normalizeMergeInputMeta, 0, len(rawInputs))
normalizedDir := filepath.Join(paths.TranscriptsRawDir, "normalized")
if runLayout.Enabled {
normalizedDir = filepath.Join(runLayout.ScratchDir, "normalized")
}
if err := os.MkdirAll(normalizedDir, 0o755); err != nil {
return nil, nil, nil, nil, fmt.Errorf("merge: ensure normalized transcripts directory %q: %w", normalizedDir, err)
}
@@ -226,6 +267,11 @@ func normalizeMergeInputs(ctx context.Context, env *Env, rawInputs []string, pat
stdoutPath := filepath.Join(paths.LogsDir, "seriatim.normalize."+base+".stdout.log")
stderrPath := filepath.Join(paths.LogsDir, "seriatim.normalize."+base+".stderr.log")
cfgPath := filepath.Join(paths.ConfigDir, "seriatim.normalize."+base+".generated.yml")
if runLayout.Enabled {
stdoutPath = filepath.Join(runLayout.LogsDir, "seriatim.normalize."+base+".stdout.log")
stderrPath = filepath.Join(runLayout.LogsDir, "seriatim.normalize."+base+".stderr.log")
cfgPath = filepath.Join(runLayout.ConfigDir, "seriatim.normalize."+base+".generated.yml")
}
req := seriatim.NormalizeRequest{
Binary: env.Config.Pipeline.Seriatim.Binary,

Some files were not shown because too many files have changed in this diff Show More