301 lines
10 KiB
Markdown
301 lines
10 KiB
Markdown
# narratio
|
||
|
||
`narratio` is a Go orchestration application for processing D&D session audio into transcripts and generated artifacts.
|
||
|
||
## Current Implementation
|
||
|
||
Implemented now:
|
||
|
||
- strict config loading/validation (`pipeline.yml` and `session.yml`)
|
||
- local workspace/session layout, locking, and manifest persistence
|
||
- resumable stage control (`run`, `plan`, `resume`, `run-stage`, `status`)
|
||
- real `prepare`, `transcribe`, `merge`, `polish`, `normalize`, `trim`, and `analyze` stages
|
||
- real WhisperX, Seriatim, and Audita adapters
|
||
- real Scriptorium subprocess adapter
|
||
- optional Scriptorium render diagnostics (`render_debug`)
|
||
|
||
Not implemented yet:
|
||
|
||
- `archive` stage behavior
|
||
- `notify` stage behavior
|
||
- additional analyze artifacts beyond `session_recap`
|
||
- generic DAG orchestration
|
||
|
||
## Config Files
|
||
|
||
Narratio expects two YAML files:
|
||
|
||
- `pipeline.yml`: pipeline/workspace settings
|
||
- `session.yml`: per-session settings
|
||
|
||
Pipeline config lookup for CLI commands:
|
||
|
||
- if `--config <path>` is provided, Narratio uses that path
|
||
- if `--config` is omitted, Narratio searches in this order:
|
||
- `/usr/local/etc/narratio/pipeline.yml`
|
||
- `/etc/narratio/pipeline.yml`
|
||
|
||
Optional secrets-from-files config:
|
||
|
||
- `pipeline.secrets.env_dir` may point to a directory of secret files
|
||
- each top-level file with an env-var-style name is loaded as an environment variable:
|
||
- file name = env var name
|
||
- file contents = env var value (trailing newline/CRLF trimmed)
|
||
- process environment wins: existing env vars are not overwritten
|
||
- if configured, Narratio fails fast when `env_dir` is missing/unreadable
|
||
- relative `env_dir` values resolve from Narratio’s current working directory
|
||
|
||
YAML decoding is strict (`KnownFields(true)`), so unknown fields fail fast.
|
||
|
||
## Canonical Stage Order
|
||
|
||
1. `prepare`
|
||
2. `transcribe`
|
||
3. `merge`
|
||
4. `polish`
|
||
5. `normalize`
|
||
6. `trim`
|
||
7. `analyze`
|
||
8. `archive`
|
||
9. `notify`
|
||
|
||
## Transcript Tiers
|
||
|
||
- `transcripts/merged.json`: canonical deterministic merged transcript from Seriatim merge
|
||
- `transcripts/processed.json`: full raw Audita-polished transcript output
|
||
- `transcripts/normalized.json`: Seriatim-normalized transcript from the normalize stage
|
||
- `transcripts/trimmed.json`: gameplay-only normalized polished transcript from trim stage
|
||
|
||
## Normalize Configuration
|
||
|
||
`pipeline.normalize` is optional. When omitted, Narratio defaults to:
|
||
|
||
- `output_path: transcripts/normalized.json`
|
||
- `output_schema: seriatim-intermediate`
|
||
- `report: true`
|
||
|
||
Allowed `normalize.output_schema` values:
|
||
|
||
- `seriatim-minimal`
|
||
- `seriatim-intermediate`
|
||
- `seriatim-full`
|
||
|
||
`normalize.output_path` is treated as session-workdir-relative when not absolute.
|
||
|
||
Normalize stage behavior summary:
|
||
|
||
- normalize runs after `polish` and before `trim`
|
||
- normalize resolves `transcripts/processed.json`
|
||
- normalize runs Seriatim `normalize` to produce `transcripts/normalized.json`
|
||
- normalize diagnostics are written to:
|
||
- `artifacts/seriatim.normalize.report.json` (when enabled)
|
||
- `logs/seriatim.normalize.stdout.log`
|
||
- `logs/seriatim.normalize.stderr.log`
|
||
- `config/seriatim.normalize.generated.yml`
|
||
|
||
## Trim Configuration
|
||
|
||
`pipeline.trim` is optional. If omitted, no trim config is loaded. If `trim.enabled` is omitted, it defaults to `false`.
|
||
|
||
When `trim.enabled: true`:
|
||
|
||
- `trim.output_path` is required
|
||
- `trim.bounds.prompt_id` is required
|
||
- `trim.bounds.transcript_input_name` is required
|
||
- `trim.bounds.output_path` is required
|
||
- `trim.bounds.timeout` must be a valid Go duration when provided
|
||
- `trim.bounds.render_debug: true` requires `trim.bounds.render_output_path`
|
||
- `trim.bounds.profile_id` may be empty to use the prompt default profile
|
||
|
||
Trim paths are treated as session-workdir-relative when not absolute.
|
||
|
||
Example trim config:
|
||
|
||
```yaml
|
||
trim:
|
||
enabled: true
|
||
output_path: "transcripts/trimmed.json"
|
||
bounds:
|
||
prompt_id: "dnd_session.bounds"
|
||
profile_id: ""
|
||
transcript_input_name: "transcript"
|
||
output_path: "artifacts/session_bounds.json"
|
||
timeout: "10m"
|
||
render_debug: false
|
||
render_output_path: "artifacts/session_bounds.render.json"
|
||
seriatim:
|
||
report: false
|
||
```
|
||
|
||
Trim behavior summary:
|
||
|
||
- trim discovers and validates `transcripts/normalized.json`
|
||
- trim uses Scriptorium bounds (`dnd_session.bounds` by example config) to produce `artifacts/session_bounds.json`
|
||
- bounds IDs are validated against the same normalized transcript ID space that Seriatim trim will consume
|
||
- trim converts bounds to Seriatim keep selector (for example `10-868`) and runs Seriatim trim
|
||
- if trim is disabled, Narratio copies normalized transcript to trimmed transcript and records `trim_action=copy_disabled`
|
||
|
||
Trim outputs and diagnostics:
|
||
|
||
- `artifacts/session_bounds.json`
|
||
- `transcripts/trimmed.json`
|
||
- `logs/scriptorium.bounds.stdout.log`
|
||
- `logs/scriptorium.bounds.stderr.log`
|
||
- `config/scriptorium.bounds.generated.yml`
|
||
- `logs/seriatim.trim.stdout.log`
|
||
- `logs/seriatim.trim.stderr.log`
|
||
- `config/seriatim.trim.generated.yml`
|
||
- optional bounds render-debug outputs:
|
||
- `artifacts/session_bounds.render.json`
|
||
- `logs/scriptorium.bounds.render.stdout.log`
|
||
- `logs/scriptorium.bounds.render.stderr.log`
|
||
- `config/scriptorium.bounds.render.generated.yml`
|
||
|
||
Render-debug files are diagnostics and are not treated as canonical stage output artifact refs.
|
||
|
||
## Scriptorium Configuration
|
||
|
||
`pipeline.scriptorium` is optional. When present, Narratio validates and uses it for analyze-stage artifact generation.
|
||
|
||
Key points:
|
||
|
||
- `scriptorium.binary` is required when section is present
|
||
- `scriptorium.config_path` is optional
|
||
- `scriptorium.timeout` defaults to `10m` when omitted
|
||
- `scriptorium.render_debug` enables render diagnostics globally
|
||
- artifacts are configured under `scriptorium.artifacts` (map shape supports multiple artifacts)
|
||
- enabled artifacts require `prompt_id` and `output_path`
|
||
- artifact `render_debug` may override global render setting
|
||
- `vars` currently support boolean and string values
|
||
|
||
Example `session_recap` artifact definition:
|
||
|
||
```yaml
|
||
scriptorium:
|
||
binary: "scriptorium"
|
||
config_path: "/etc/scriptorium/config.yml"
|
||
timeout: "10m"
|
||
render_debug: false
|
||
|
||
artifacts:
|
||
session_recap:
|
||
enabled: true
|
||
prompt_id: "dnd.session_recap"
|
||
profile_id: "local-quality" # optional
|
||
output_path: "artifacts/session_recap.md"
|
||
timeout: "10m"
|
||
# render_debug: true # optional per-artifact override
|
||
|
||
inputs:
|
||
transcript:
|
||
source: "trimmed_transcript"
|
||
required: true
|
||
|
||
previous_recap:
|
||
source: "previous_session_artifact"
|
||
artifact: "session_recap"
|
||
path: "" # optional; set when available
|
||
required: false
|
||
|
||
vars:
|
||
session_id: true
|
||
session_date: true
|
||
campaign_name: true
|
||
previous_session_id: true
|
||
output_kind: "session_recap"
|
||
```
|
||
|
||
Prompt IDs and profile IDs are configuration values. They are not hardcoded in analyze-stage logic.
|
||
|
||
Do not put secrets in `pipeline.yml`. If API-key behavior is configured, use env var names only.
|
||
|
||
If `pipeline.secrets.env_dir` is configured, keep only references and secret files there; secret values are still not written to manifests, generated configs, or Narratio-managed logs.
|
||
|
||
## Scriptorium Runtime Behavior
|
||
|
||
Narratio integrates with Scriptorium through the public CLI subprocess contract:
|
||
|
||
- generation: `scriptorium run`
|
||
- diagnostics/testing: `scriptorium render --format json` when `render_debug` is enabled
|
||
|
||
For the initial implementation, only `session_recap` generation is supported.
|
||
|
||
Analyze-stage session recap behavior:
|
||
|
||
- available transcript input sources for configured artifacts: `processed_transcript`, `normalized_transcript`, `trimmed_transcript`
|
||
- session recap should use gameplay-only transcript input (`source: trimmed_transcript`)
|
||
- Narratio resolves `trimmed_transcript` from trim manifest output (`transcript_trimmed`) or fallback `transcripts/trimmed.json`
|
||
- Narratio resolves `normalized_transcript` from normalize manifest output (`transcript_normalized`) or fallback `transcripts/normalized.json`
|
||
- missing trimmed transcript fails clearly and advises running trim stage first
|
||
- `normalized_transcript` is the preferred full-transcript source for future table/meta-analysis artifacts
|
||
- `processed_transcript` remains supported for advanced/debug use cases
|
||
- optionally includes `previous_recap` when configured and resolvable
|
||
- omits optional previous recap when unavailable
|
||
- fails if required inputs are missing
|
||
- validates output file exists and is non-empty
|
||
|
||
Expected session output paths:
|
||
|
||
- `artifacts/session_recap.md`
|
||
- `logs/scriptorium.session_recap.stdout.log`
|
||
- `logs/scriptorium.session_recap.stderr.log`
|
||
- `config/scriptorium.session_recap.generated.yml`
|
||
- `artifacts/session_recap.render.json` when render diagnostics are enabled
|
||
|
||
## Examples
|
||
|
||
Starter files:
|
||
|
||
- `examples/pipeline.minimal.yml`
|
||
- `examples/session.minimal.yml`
|
||
- `examples/speakers.yml`
|
||
|
||
## Commands
|
||
|
||
Run tests:
|
||
|
||
```bash
|
||
go test ./...
|
||
```
|
||
|
||
Plan a run:
|
||
|
||
```bash
|
||
go run ./cmd/narratio plan --session examples/session.minimal.yml
|
||
```
|
||
|
||
Use `--config <path>` to override default pipeline lookup when needed.
|
||
|
||
Run full pipeline:
|
||
|
||
```bash
|
||
go run ./cmd/narratio run --config examples/pipeline.minimal.yml --session examples/session.minimal.yml
|
||
```
|
||
|
||
Run analyze only:
|
||
|
||
```bash
|
||
go run ./cmd/narratio run-stage --config examples/pipeline.minimal.yml --session examples/session.minimal.yml analyze
|
||
```
|
||
|
||
## Operational Note
|
||
|
||
Checksum-based stale detection is not implemented yet.
|
||
|
||
If prepared inputs or prompt/runtime config change, rerun the appropriate upstream stages before relying on downstream artifacts.
|
||
|
||
Examples:
|
||
|
||
- glossary/autocorrect/speaker-context changes: rerun at least `merge`, `polish`, `normalize`, `trim`, and `analyze`
|
||
- trim bounds prompt/profile/config changes: rerun at least `normalize`, `trim`, and `analyze`
|
||
- session recap prompt/profile/input-source changes: rerun `analyze`
|
||
|
||
## Roadmap
|
||
|
||
Near-term roadmap:
|
||
|
||
- extend analyze to additional configured artifacts
|
||
- support workflows where later artifacts consume earlier generated artifacts
|
||
- keep orchestration explicit without a generic DAG engine
|
||
- implement archive and notify backends
|