eric/narratio
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 15 Wiki Activity

Document Scriptorium session recap integration

Browse Source
This commit is contained in:
Eric Rakestraw
2026-05-06 21:34:09 +00:00
parent 4aafeef830
commit 2c8a922cf7
4 changed files with 306 additions and 1015 deletions
Download Patch File Download Diff File Expand all files Collapse all files

245
README.md
View File

@@ -1,136 +1,157 @@
# narratio
`narratio` is a Go-based orchestration application for processing D&D session audio into transcripts and downstream artifacts.
`narratio` is a Go orchestration application for processing D&D session audio into transcripts and generated artifacts.
This repository currently contains a **working scaffold** with strict config loading, local workdir/manifest handling, resumable stage control, real WhisperX/Seriatim/Audita adapters, and real `prepare` + `transcribe` + `merge` + `polish` stages.
## Expected Config Files
`narratio` expects two YAML files:
- `pipeline.yml`: pipeline/workspace settings (`workspace`, `storage`, `whisperx`, `seriatim`, `audita`, optional `scriptorium`, `analyzer`, `notification`)
- `session.yml`: per-session settings (`session_id`, `inputs`, optional metadata)
WhisperX config contract in `pipeline.yml`:
- required: `whisperx.transcribe_url`
- defaulted when omitted: `whisperx.language` (`en`), `whisperx.timeout` (`30m`), `whisperx.retries` (`3`), `whisperx.retry_delay` (`2s`), `whisperx.concurrency` (`2`)
Seriatim config contract in `pipeline.yml`:
- required: `seriatim.binary` (name or path; existence is checked at execution time, not config validation time)
- defaulted when omitted: `seriatim.timeout` (`10m`), `seriatim.output_schema` (`seriatim-intermediate`), `seriatim.coalesce_gap` (`3.0`), `seriatim.report` (`true`)
- allowed `seriatim.output_schema` values: `seriatim-minimal`, `seriatim-intermediate`, `seriatim-full`
- optional tuning: `seriatim.env.*` (`overlap_word_run_gap`, `overlap_word_run_reorder_window`, `backchannel_max_duration`, `filler_max_duration`) must be `> 0` when provided
Audita config contract in `pipeline.yml`:
- required: `audita.binary` (name or path; existence is checked at execution time, not config validation time)
- optional: `audita.llm_api_key_env` (environment variable name holding the API key secret; no automatic default)
- defaulted when omitted: `audita.timeout` (`3h`), `audita.modules` (`glossary,homophones,glossary,spoken_word,grammar,homophones,glossary`), `audita.base_url` (`https://openrouter.ai/api/v1`), `audita.model` (`openrouter/google/gemma-4-31b-it`), `audita.llm_concurrency` (`1`), `audita.validation_model` (`""`), `audita.validation_llm_concurrency` (`1`), `audita.report` (`true`)
- allowed `audita.modules` values: `glossary`, `homophones`, `spoken_word`, `grammar` (order and repeats are allowed)
- `audita.base_url` must be a valid URL when provided
- `audita.llm_concurrency` and `audita.validation_llm_concurrency` must be `> 0`
Audita credentials note:
- store only the environment variable **name** in config (`audita.llm_api_key_env`), never the API key value itself
- API key values must not be written to pipeline config, generated configs, logs, or manifest metadata
- if `audita.llm_api_key_env` is configured and the named env var is not set (or is empty), Narratio fails before invocation with a redacted error
- if `audita.llm_api_key_env` is omitted/empty, Narratio does not require a credential and omits `AUDITA_LLM_API_KEY` from the subprocess overrides
Audita runtime note:
- Narratio currently passes primary LLM concurrency via `AUDITA_LLM_CONCURRENCY` subprocess environment override, not a `--llm-concurrency` flag.
Scriptorium config contract in `pipeline.yml` (optional until `analyze` is implemented):
- optional section: `scriptorium` (when omitted, current pipeline behavior is unchanged)
- required when `scriptorium` is present: `scriptorium.binary` (name on PATH or full path)
- optional: `scriptorium.config_path` (if set, it must be non-empty; file existence is validated at execution time, not basic config validation time)
- defaulted when omitted inside present `scriptorium` section: `scriptorium.timeout` (`10m`)
- optional: `scriptorium.render_debug` (`false` by default through YAML zero-value behavior)
- artifact definitions live under `scriptorium.artifacts.<artifact_name>` and support multiple configured artifacts
- enabled artifacts require `prompt_id` and `output_path`
- artifact `timeout` values must be valid Go durations when provided
- input definitions live under `scriptorium.artifacts.<artifact_name>.inputs.<input_name>` and require non-empty `source`
- `vars` values currently accept booleans and strings only
- Narratio does not store secrets in Scriptorium config; keep credentials in environment management, not YAML
Scriptorium initial artifact note:
- this pass does not auto-create `session_recap`; configure `scriptorium.artifacts.session_recap` explicitly in `pipeline.yml`
`speakers.yml` note:
- use Seriatim’s documented `match:` format (not the legacy direct mapping style used by older scripts/scaffolds)
- see [`examples/speakers.yml`](examples/speakers.yml) for a concrete `match:` example.
Prepared inputs note:
- `prepare` copies session inputs into the session workdir (`work/<session_id>/inputs`), and downstream stages consume those prepared copies, not the original source files.
- If you edit `speakers.yml`, `autocorrect.yml`, `glossary.yml`, or session input paths after `prepare` has run, rerun `prepare` with `--force` before rerunning downstream stages.
Decoding is strict (`KnownFields(true)`), so unknown YAML fields fail fast.
Example minimal files are available under `examples/`:
- `examples/pipeline.minimal.yml`
- `examples/session.minimal.yml`
## Current Scaffold Status
## Current Implementation
Implemented now:
- strict config load + validation
- local artifact/workdir creation and locking
- manifest create/load/save and stage status tracking
- stage framework with real `prepare`, `transcribe`, `merge`, and `polish` stages; placeholder downstream stages
- resumable run control (`run`, `resume`, `run-stage`, `plan` with run/skip decisions)
- real WhisperX HTTP adapter plus real Seriatim/Audita subprocess adapters (with fake/no-op adapters for test/scaffold usage)
- strict config loading/validation (`pipeline.yml` and `session.yml`)
- local workspace/session layout, locking, and manifest persistence
- resumable stage control (`run`, `plan`, `resume`, `run-stage`, `status`)
- real `prepare`, `transcribe`, `merge`, and `polish` stages
- real WhisperX, Seriatim, and Audita adapters
- real Scriptorium subprocess adapter
- real `analyze` stage for `session_recap`
- optional Scriptorium render diagnostics (`render_debug`)
Not implemented yet:
- real analyzer integration
- real remote archive/storage backend
- real notification backend
- real `normalize` behavior
- real `archive` behavior
- real `notify` behavior
- additional analyze artifacts beyond `session_recap`
- generic DAG orchestration
## Run Tests
## Config Files
Narratio expects two YAML files:
- `pipeline.yml`: pipeline/workspace settings
- `session.yml`: per-session settings
YAML decoding is strict (`KnownFields(true)`), so unknown fields fail fast.
## Scriptorium Configuration
`pipeline.scriptorium` is optional. When present, Narratio validates and uses it for analyze-stage artifact generation.
Key points:
- `scriptorium.binary` is required when section is present
- `scriptorium.config_path` is optional
- `scriptorium.timeout` defaults to `10m` when omitted
- `scriptorium.render_debug` enables render diagnostics globally
- artifacts are configured under `scriptorium.artifacts` (map shape supports multiple artifacts)
- enabled artifacts require `prompt_id` and `output_path`
- artifact `render_debug` may override global render setting
- `vars` currently support boolean and string values
Example `session_recap` artifact definition:
```yaml
scriptorium:
binary: "scriptorium"
config_path: "/etc/scriptorium/config.yml"
timeout: "10m"
render_debug: false
artifacts:
session_recap:
enabled: true
prompt_id: "dnd.session_recap"
profile_id: "local-quality" # optional
output_path: "artifacts/session_recap.md"
timeout: "10m"
# render_debug: true # optional per-artifact override
inputs:
transcript:
source: "processed_transcript"
required: true
previous_recap:
source: "previous_session_artifact"
artifact: "session_recap"
path: "" # optional; set when available
required: false
vars:
session_id: true
session_date: true
campaign_name: true
previous_session_id: true
output_kind: "session_recap"
```
Prompt IDs and profile IDs are configuration values. They are not hardcoded in analyze-stage logic.
Do not put secrets in `pipeline.yml`. If API-key behavior is configured, use env var names only.
## Scriptorium Runtime Behavior
Narratio integrates with Scriptorium through the public CLI subprocess contract:
- generation: `scriptorium run`
- diagnostics/testing: `scriptorium render --format json` when `render_debug` is enabled
For the initial implementation, only `session_recap` generation is supported.
Analyze-stage session recap behavior:
- uses processed transcript input (`transcripts/processed.json`)
- optionally includes `previous_recap` when configured and resolvable
- omits optional previous recap when unavailable
- fails if required inputs are missing
- validates output file exists and is non-empty
Expected session output paths:
- `artifacts/session_recap.md`
- `logs/scriptorium.session_recap.stdout.log`
- `logs/scriptorium.session_recap.stderr.log`
- `config/scriptorium.session_recap.generated.yml`
- `artifacts/session_recap.render.json` when render diagnostics are enabled
## Examples
Starter files:
- `examples/pipeline.minimal.yml`
- `examples/session.minimal.yml`
- `examples/speakers.yml`
## Commands
Run tests:
```bash
go test ./...
```
## Run Plan
Plan a run:
```bash
go run ./cmd/narratio plan --config examples/pipeline.minimal.yml --session examples/session.minimal.yml
```
## Run Pipeline (Current State)
The current `run` command executes `prepare` + real `transcribe` + real `merge` + real `polish` + placeholder downstream stages and records progress in `manifest.json`.
Default CLI wiring builds and uses:
- real WhisperX HTTP adapter from `pipeline.whisperx`
- real Seriatim subprocess adapter from `pipeline.seriatim`
- real Audita subprocess adapter from `pipeline.audita`
Subprocess runtime note:
- subprocesses inherit the parent environment by default, then apply Narratio override values (override values win).
- non-zero subprocess errors include stdout/stderr log paths and a short redacted stderr tail when available to speed diagnosis.
Real `polish` stage output paths:
- `transcripts/processed.json`
- `artifacts/audita.report.json` (when `audita.report: true`)
- `artifacts/audita-work`
- `logs/audita.stdout.log`
- `logs/audita.stderr.log`
- `config/audita.generated.yml`
Run full pipeline:
```bash
go run ./cmd/narratio run --config examples/pipeline.minimal.yml --session examples/session.minimal.yml
```
Run analyze only:
```bash
go run ./cmd/narratio run-stage --config examples/pipeline.minimal.yml --session examples/session.minimal.yml analyze
```
## Roadmap
Near-term roadmap:
- extend analyze to additional configured artifacts
- support workflows where later artifacts consume earlier generated artifacts
- keep orchestration explicit without a generic DAG engine
- implement archive and notify backends

1038
architecture.md
View File

File diff suppressed because it is too large Load Diff

35
docs/integrations/scriptorium.md
View File

@@ -217,23 +217,40 @@ Narratio should treat non-zero exit codes as failed stage execution, but may rec
## 17. Suggested Narratio Configuration Shape
Illustrative (not required schema):
Illustrative `pipeline.yml` shape:
```yaml
scriptorium:
binary: scriptorium
config_path: /etc/scriptorium/config.yml
stages:
timeout: 10m
render_debug: false
artifacts:
session_recap:
enabled: true
prompt_id: dnd.session_recap
profile_id: local-quality # optional
inputs: [transcript, glossary, previous_recap]
vars: [session_id, session_date, campaign_name]
output_path_template: artifacts/{session_id}/session_recap.md
timeout: 2m
render_debug: false
output_path: artifacts/session_recap.md
timeout: 10m
render_debug: false # optional artifact override
inputs:
transcript:
source: processed_transcript
required: true
previous_recap:
source: previous_session_artifact
artifact: session_recap
path: "" # optional
required: false
vars:
session_id: true
session_date: true
campaign_name: true
previous_session_id: true
output_kind: session_recap
```
The key idea: map Narratio stage/artifact names to prompt ID, optional profile, expected inputs, and output destination.
The key idea: map Narratio artifact names to prompt ID, optional profile, expected inputs, vars, and output destination.
## 18. Testing Strategy for Narratio Integration
@@ -317,6 +334,6 @@ Possible later extensions:
- HTTP API integration
- S3 artifact references if Scriptorium adds S3 reader support
- storing render diagnostics alongside generated artifacts
- richer render diagnostics and policy controls
- token budgeting/prompt-size checks
- batch execution if Scriptorium later adds batch support

3
examples/pipeline.minimal.yml
View File

@@ -55,6 +55,8 @@ scriptorium:
profile_id: "local-quality"
output_path: "artifacts/session_recap.md"
timeout: "10m"
# Optional per-artifact override of global scriptorium.render_debug.
# render_debug: true
inputs:
transcript:
source: "processed_transcript"
@@ -62,6 +64,7 @@ scriptorium:
previous_recap:
source: "previous_session_artifact"
artifact: "session_recap"
# Optional: set when previous recap is available.
path: ""
required: false
vars: