Complete documentation rebuild
This commit is contained in:
471
README.md
471
README.md
@@ -1,467 +1,22 @@
|
||||
# narratio
|
||||
|
||||
`narratio` is a Go orchestration application for processing D&D session audio into transcripts and generated artifacts.
|
||||
Narratio is a Go orchestration application that turns D&D session audio into polished transcripts and generated session artifacts.
|
||||
|
||||
## Current Implementation
|
||||
|
||||
Implemented now:
|
||||
|
||||
- strict config loading/validation (`pipeline.yml` and `session.yml`)
|
||||
- local workspace/session layout, locking, and manifest persistence
|
||||
- resumable stage control (`run`, `plan`, `resume`, `run-stage`, `status`)
|
||||
- real `prepare`, `transcribe`, `merge`, `polish`, `normalize`, `trim`, and `analyze` stages
|
||||
- real WhisperX, Seriatim, and Audita adapters
|
||||
- real Scriptorium subprocess adapter
|
||||
- optional Scriptorium render diagnostics (`render_debug`)
|
||||
|
||||
Not implemented yet:
|
||||
|
||||
- `notify` stage behavior
|
||||
- additional analyze artifacts beyond `session_recap`
|
||||
- generic DAG orchestration
|
||||
|
||||
## Config Files
|
||||
|
||||
Narratio expects two YAML files:
|
||||
|
||||
- `pipeline.yml`: pipeline/workspace settings
|
||||
- `session.yml`: per-session settings
|
||||
|
||||
Pipeline config lookup for CLI commands:
|
||||
|
||||
- if `--config <path>` is provided, Narratio uses that path
|
||||
- if `--config` is omitted, Narratio searches in this order:
|
||||
- `/usr/local/etc/narratio/pipeline.yml`
|
||||
- `/etc/narratio/pipeline.yml`
|
||||
|
||||
Session config lookup for CLI commands:
|
||||
|
||||
- if `--session <path>` is provided, Narratio uses that path
|
||||
- if `--session` is omitted, Narratio searches in this order:
|
||||
- `./session.yml`
|
||||
- `/usr/local/etc/narratio/session.yml`
|
||||
- `/etc/narratio/session.yml`
|
||||
|
||||
Session template support:
|
||||
|
||||
- Narratio renders `session.yml` templates before strict YAML decode.
|
||||
- `--session-id <value>` provides the `session_id` template variable.
|
||||
- Supported placeholder forms:
|
||||
- `{{session_id}}`
|
||||
- `{{ session_id }}`
|
||||
- unresolved template placeholders fail with a clear error.
|
||||
- strict YAML validation still runs after rendering.
|
||||
- concrete `session.yml` files without templates remain fully supported.
|
||||
|
||||
Optional secrets-from-files config:
|
||||
|
||||
- `pipeline.secrets.env_dir` may point to a directory of secret files
|
||||
- each top-level file with an env-var-style name is loaded as an environment variable:
|
||||
- file name = env var name
|
||||
- file contents = env var value (trailing newline/CRLF trimmed)
|
||||
- process environment wins: existing env vars are not overwritten
|
||||
- if configured, Narratio fails fast when `env_dir` is missing/unreadable
|
||||
- relative `env_dir` values resolve from Narratio’s current working directory
|
||||
|
||||
YAML decoding is strict (`KnownFields(true)`), so unknown fields fail fast.
|
||||
|
||||
Maintainer note: application defaults are centralized in [`internal/config/defaults.go`](internal/config/defaults.go).
|
||||
|
||||
## Storage And Archive Foundations
|
||||
|
||||
Narratio now includes configuration and path-model foundations for archive support, plus implemented prepare-stage S3 audio input.
|
||||
|
||||
Implemented foundations:
|
||||
|
||||
- `pipeline.storage.s3` config shape (`bucket`, `root_prefix`, `region`, `endpoint`, `force_path_style`, `access_key_id_env`, `secret_access_key_env`)
|
||||
- `pipeline.spool` config shape (`root`, `delete_audio_after_archive`)
|
||||
- `pipeline.archive` config shape (`enabled`, `upload_run`, `promote_artifacts`)
|
||||
- promotion-rule validation (`from`/`to` required, relative-only paths, traversal rejected)
|
||||
- `session.campaign` requirement for campaign-aware path construction
|
||||
- optional `session.inputs.audio_s3.prefix` modeling and prepare-stage S3 audio download
|
||||
- run ID generation and S3/local path helper foundations
|
||||
- manifest run/path identity fields
|
||||
|
||||
Current defaults:
|
||||
|
||||
- `pipeline.storage.s3.root_prefix`: `dnd`
|
||||
- `pipeline.storage.s3.access_key_id_env`: `OBJECT_STORAGE_KEY_ID`
|
||||
- `pipeline.storage.s3.secret_access_key_env`: `OBJECT_STORAGE_KEY`
|
||||
- `pipeline.workspace.cleanup_after_archive`: `false`
|
||||
- `pipeline.spool.root`: `/var/spool/narratio`
|
||||
- `pipeline.spool.delete_audio_after_archive`: `false`
|
||||
- `pipeline.archive.enabled`: `true`
|
||||
- `pipeline.archive.upload_run`: `true`
|
||||
- default `pipeline.archive.promote_artifacts`:
|
||||
- `transcripts/trimmed.json` -> `transcripts/trimmed.json` (`required: true`)
|
||||
- `artifacts/session_recap.md` -> `artifacts/session_recap.md` (`required: true`)
|
||||
|
||||
Current boundaries:
|
||||
|
||||
- local development audio (`audio_dir` / `audio_files`) still works
|
||||
- `audio_dir`/`audio_files` and `audio_s3` are mutually exclusive
|
||||
- real S3-compatible backend now exists in the storage adapter package
|
||||
- storage backend tests use fake storage and do not require live S3
|
||||
- archive uploads successful run records under `runs/{run_id}/`
|
||||
- archive does not upload local audio by default
|
||||
- archive uploads promoted outputs to session-level keys using `archive.promote_artifacts`
|
||||
- archive uploads `current/manifest.json`
|
||||
- archive uploads `current/run_id.txt` last as the effective commit marker
|
||||
- required missing promotions fail archive
|
||||
- optional missing promotions are skipped and recorded
|
||||
- cleanup remains conservative and opt-in:
|
||||
- `pipeline.spool.delete_audio_after_archive: true` removes only the run-scoped spool audio directory after successful archive commit
|
||||
- `pipeline.workspace.cleanup_after_archive: true` removes only the run-scoped local workdir after successful archive commit
|
||||
- cleanup executes only after all selected stages for the command invocation succeed
|
||||
- cleanup does not run for failed, incomplete, skipped, or unarchived runs
|
||||
- local development `audio_dir`/`audio_files` source inputs are never deleted by spool cleanup
|
||||
- S3 credentials are resolved from configured env-var names when both are present; if either is missing, Narratio falls back to the AWS SDK default credential chain
|
||||
|
||||
S3 input details and current boundaries are documented in [docs/s3-audio-input.md](docs/s3-audio-input.md).
|
||||
|
||||
## Remote Storage Backend
|
||||
|
||||
Narratio includes an object-store backend layer for future prepare/archive work:
|
||||
|
||||
- `List(ctx, prefix)`
|
||||
- `Download(ctx, key, localPath)`
|
||||
- `Upload(ctx, localPath, key, opts)`
|
||||
- `Exists(ctx, key)`
|
||||
|
||||
Implemented backends:
|
||||
|
||||
- fake storage backend for deterministic tests
|
||||
- S3-compatible backend built from `pipeline.storage.s3`
|
||||
|
||||
Key invariant:
|
||||
|
||||
- callers pass full bucket-relative object keys
|
||||
- storage backends do not prepend `root_prefix` and do not infer session/campaign paths
|
||||
|
||||
Current boundary:
|
||||
|
||||
- `prepare` uses `List` + `Download` through the backend when `session.inputs.audio_s3` is configured
|
||||
- `archive` uses `Upload` through the backend for successful run-record uploads
|
||||
- `archive` also uses `Upload` for promotion writes and current pointers
|
||||
- no failed or incomplete runs are uploaded
|
||||
- local audio is not re-uploaded by default
|
||||
|
||||
Archive run-upload details and boundaries are documented in [docs/archive-storage.md](docs/archive-storage.md).
|
||||
|
||||
## Canonical Stage Order
|
||||
|
||||
1. `prepare`
|
||||
2. `transcribe`
|
||||
3. `merge`
|
||||
4. `polish`
|
||||
5. `normalize`
|
||||
6. `trim`
|
||||
7. `analyze`
|
||||
8. `archive`
|
||||
9. `notify`
|
||||
|
||||
## Transcript Tiers
|
||||
|
||||
- `transcripts/merged.json`: canonical deterministic merged transcript from Seriatim merge
|
||||
- `transcripts/processed.json`: full raw Audita-polished transcript output
|
||||
- `transcripts/normalized.json`: Seriatim-normalized transcript from the normalize stage
|
||||
- `transcripts/trimmed.json`: gameplay-only normalized polished transcript from trim stage
|
||||
|
||||
## Seriatim Configuration
|
||||
|
||||
`pipeline.seriatim` configures the Seriatim subprocess adapter used by `merge`, `normalize`, and `trim`.
|
||||
|
||||
Minimal behavior:
|
||||
|
||||
- `pipeline.seriatim` may be omitted entirely.
|
||||
- when omitted, Narratio defaults to:
|
||||
- `binary: seriatim`
|
||||
- `timeout: 10m`
|
||||
- `output_schema: seriatim-intermediate`
|
||||
- `coalesce_gap: 3.0`
|
||||
- `report: true`
|
||||
|
||||
Optional overrides in `pipeline.seriatim` continue to work, including explicit binary paths and advanced `env` tuning values.
|
||||
|
||||
## Audita Configuration
|
||||
|
||||
`pipeline.audita` configures the real Audita subprocess adapter used by `polish`.
|
||||
|
||||
Minimal behavior:
|
||||
|
||||
- `pipeline.audita` may be omitted entirely.
|
||||
- when omitted, Narratio defaults to:
|
||||
- `binary: audita`
|
||||
- `timeout: 3h`
|
||||
- `report: true`
|
||||
|
||||
Optional:
|
||||
|
||||
- `llm_api_key_env` (when set, Narratio requires that env var and passes it to Audita as `AUDITA_LLM_API_KEY`)
|
||||
- `modules` override list (when empty/omitted, Narratio does not pass `--modules`)
|
||||
- `base_url` (when omitted, Narratio does not pass `--base-url`; Audita runtime defaults/config may apply)
|
||||
- `model` (when omitted, Narratio does not pass `--model`; Audita runtime defaults/config may apply)
|
||||
- `transcript_description`
|
||||
- `config_path`
|
||||
- `output_schema` (`bare-segments` or `audita-v1`)
|
||||
- `work_dir_retention` (`always`, `auto`, or `never`)
|
||||
- `total_llm_concurrency` (> 0 when provided)
|
||||
- `proposal_llm_concurrency` (> 0 when provided)
|
||||
- `validation_model`
|
||||
- `validation_llm_concurrency` (> 0 when provided)
|
||||
- `report` (defaults to `true`)
|
||||
|
||||
Narratio passes only configured optional Audita flags. Omitted optional values are left to Audita runtime defaults/config.
|
||||
|
||||
## Normalize Configuration
|
||||
|
||||
`pipeline.normalize` is optional. When omitted, Narratio defaults to:
|
||||
|
||||
- `output_path: transcripts/normalized.json`
|
||||
- `output_schema: seriatim-intermediate`
|
||||
- `report: true`
|
||||
|
||||
Allowed `normalize.output_schema` values:
|
||||
|
||||
- `seriatim-minimal`
|
||||
- `seriatim-intermediate`
|
||||
- `seriatim-full`
|
||||
|
||||
`normalize.output_path` is treated as session-workdir-relative when not absolute.
|
||||
|
||||
Normalize stage behavior summary:
|
||||
|
||||
- normalize runs after `polish` and before `trim`
|
||||
- normalize resolves `transcripts/processed.json`
|
||||
- normalize runs Seriatim `normalize` to produce `transcripts/normalized.json`
|
||||
- normalize diagnostics are written to:
|
||||
- `artifacts/seriatim.normalize.report.json` (when enabled)
|
||||
- `logs/seriatim.normalize.stdout.log`
|
||||
- `logs/seriatim.normalize.stderr.log`
|
||||
- `config/seriatim.normalize.generated.yml`
|
||||
|
||||
## Trim Configuration
|
||||
|
||||
`pipeline.trim` is optional. If omitted, no trim config is loaded. If `trim.enabled` is omitted, it defaults to `false`.
|
||||
|
||||
When `trim.enabled: true`:
|
||||
|
||||
- `trim.output_path` is required
|
||||
- `trim.bounds.prompt_id` is required
|
||||
- `trim.bounds.transcript_input_name` is required
|
||||
- `trim.bounds.output_path` is required
|
||||
- `trim.bounds.timeout` must be a valid Go duration when provided
|
||||
- `trim.bounds.render_debug: true` requires `trim.bounds.render_output_path`
|
||||
- `trim.bounds.profile_id` may be empty to use the prompt default profile
|
||||
|
||||
Trim paths are treated as session-workdir-relative when not absolute.
|
||||
|
||||
Example trim config:
|
||||
|
||||
```yaml
|
||||
trim:
|
||||
enabled: true
|
||||
output_path: "transcripts/trimmed.json"
|
||||
bounds:
|
||||
prompt_id: "dnd_session.bounds"
|
||||
profile_id: ""
|
||||
transcript_input_name: "transcript"
|
||||
output_path: "artifacts/session_bounds.json"
|
||||
timeout: "10m"
|
||||
render_debug: false
|
||||
render_output_path: "artifacts/session_bounds.render.json"
|
||||
seriatim:
|
||||
report: false
|
||||
```
|
||||
|
||||
Trim behavior summary:
|
||||
|
||||
- trim discovers and validates `transcripts/normalized.json`
|
||||
- trim uses Scriptorium bounds (`dnd_session.bounds` by example config) to produce `artifacts/session_bounds.json`
|
||||
- bounds IDs are validated against the same normalized transcript ID space that Seriatim trim will consume
|
||||
- trim converts bounds to Seriatim keep selector (for example `10-868`) and runs Seriatim trim
|
||||
- if trim is disabled, Narratio copies normalized transcript to trimmed transcript and records `trim_action=copy_disabled`
|
||||
|
||||
Trim outputs and diagnostics:
|
||||
|
||||
- `artifacts/session_bounds.json`
|
||||
- `transcripts/trimmed.json`
|
||||
- `logs/scriptorium.bounds.stdout.log`
|
||||
- `logs/scriptorium.bounds.stderr.log`
|
||||
- `config/scriptorium.bounds.generated.yml`
|
||||
- `logs/seriatim.trim.stdout.log`
|
||||
- `logs/seriatim.trim.stderr.log`
|
||||
- `config/seriatim.trim.generated.yml`
|
||||
- optional bounds render-debug outputs:
|
||||
- `artifacts/session_bounds.render.json`
|
||||
- `logs/scriptorium.bounds.render.stdout.log`
|
||||
- `logs/scriptorium.bounds.render.stderr.log`
|
||||
- `config/scriptorium.bounds.render.generated.yml`
|
||||
|
||||
Render-debug files are diagnostics and are not treated as canonical stage output artifact refs.
|
||||
|
||||
## Scriptorium Configuration
|
||||
|
||||
`pipeline.scriptorium` is optional. When present, Narratio validates and uses it for analyze-stage artifact generation.
|
||||
|
||||
Key points:
|
||||
|
||||
- `scriptorium.binary` defaults to `scriptorium` when section is present
|
||||
- `scriptorium.config_path` is optional
|
||||
- `scriptorium.timeout` defaults to `10m` when omitted
|
||||
- `scriptorium.render_debug` enables render diagnostics globally
|
||||
- artifacts are configured under `scriptorium.artifacts` (map shape supports multiple artifacts)
|
||||
- enabled artifacts require `prompt_id` and `output_path`
|
||||
- artifact `render_debug` may override global render setting
|
||||
- `vars` currently support boolean and string values
|
||||
|
||||
Example `session_recap` artifact definition:
|
||||
|
||||
```yaml
|
||||
scriptorium:
|
||||
binary: "scriptorium"
|
||||
config_path: "/etc/scriptorium/config.yml"
|
||||
timeout: "10m"
|
||||
render_debug: false
|
||||
|
||||
artifacts:
|
||||
session_recap:
|
||||
enabled: true
|
||||
prompt_id: "dnd.session_recap"
|
||||
profile_id: "local-quality" # optional
|
||||
output_path: "artifacts/session_recap.md"
|
||||
timeout: "10m"
|
||||
# render_debug: true # optional per-artifact override
|
||||
|
||||
inputs:
|
||||
transcript:
|
||||
source: "trimmed_transcript"
|
||||
required: true
|
||||
|
||||
previous_recap:
|
||||
source: "previous_session_artifact"
|
||||
artifact: "session_recap"
|
||||
path: "" # optional; set when available
|
||||
required: false
|
||||
|
||||
vars:
|
||||
session_id: true
|
||||
session_date: true
|
||||
campaign_name: true
|
||||
previous_session_id: true
|
||||
output_kind: "session_recap"
|
||||
```
|
||||
|
||||
Prompt IDs and profile IDs are configuration values. They are not hardcoded in analyze-stage logic.
|
||||
|
||||
Do not put secrets in `pipeline.yml`. If API-key behavior is configured, use env var names only.
|
||||
|
||||
If `pipeline.secrets.env_dir` is configured, keep only references and secret files there; secret values are still not written to manifests, generated configs, or Narratio-managed logs.
|
||||
|
||||
## Scriptorium Runtime Behavior
|
||||
|
||||
Narratio integrates with Scriptorium through the public CLI subprocess contract:
|
||||
|
||||
- generation: `scriptorium run`
|
||||
- diagnostics/testing: `scriptorium render --format json` when `render_debug` is enabled
|
||||
|
||||
For the initial implementation, only `session_recap` generation is supported.
|
||||
|
||||
Analyze-stage session recap behavior:
|
||||
|
||||
- preferred transcript artifact source IDs:
|
||||
- `narratio.transcript.polished`
|
||||
- `narratio.transcript.full`
|
||||
- `narratio.transcript.trimmed`
|
||||
- backward-compatible aliases remain supported:
|
||||
- `processed_transcript`
|
||||
- `normalized_transcript`
|
||||
- `trimmed_transcript`
|
||||
- session recap should use gameplay-only transcript input (`source: trimmed_transcript`)
|
||||
- Narratio resolves transcript inputs from the artifact resolver (manifest producer outputs first, then canonical session paths)
|
||||
- missing trimmed transcript fails clearly and advises running trim stage first
|
||||
- `normalized_transcript` is the preferred full-transcript source for future table/meta-analysis artifacts
|
||||
- `processed_transcript` remains supported for advanced/debug use cases
|
||||
- optionally includes `previous_recap` when configured and resolvable
|
||||
- omits optional previous recap when unavailable
|
||||
- fails if required inputs are missing
|
||||
- validates output file exists and is non-empty
|
||||
|
||||
Expected session output paths:
|
||||
|
||||
- `artifacts/session_recap.md`
|
||||
- `logs/scriptorium.session_recap.stdout.log`
|
||||
- `logs/scriptorium.session_recap.stderr.log`
|
||||
- `config/scriptorium.session_recap.generated.yml`
|
||||
- `artifacts/session_recap.render.json` when render diagnostics are enabled
|
||||
|
||||
## Examples
|
||||
|
||||
Starter files:
|
||||
|
||||
- `examples/pipeline.minimal.yml`
|
||||
- `examples/pipeline.audita-overrides.yml`
|
||||
- `examples/session.minimal.yml`
|
||||
- `examples/session.template.yml`
|
||||
- `examples/speakers.yml`
|
||||
|
||||
## Commands
|
||||
|
||||
Run tests:
|
||||
It coordinates transcription, merge/polish/normalize/trim processing, artifact generation, archive publishing, and resumable run state in one operator workflow.
|
||||
|
||||
```bash
|
||||
go test ./...
|
||||
narratio run --session-id 2026-04-04
|
||||
```
|
||||
|
||||
Plan a run:
|
||||
This command requires discoverable `pipeline.yml` and `session.yml` files (or explicit `--config` and `--session` flags).
|
||||
|
||||
```bash
|
||||
go run ./cmd/narratio plan --session examples/session.minimal.yml
|
||||
```
|
||||
## Documentation
|
||||
|
||||
Use `--config <path>` to override default pipeline lookup when needed.
|
||||
|
||||
Run with a discoverable session template:
|
||||
|
||||
```bash
|
||||
go run ./cmd/narratio run --session-id 2026-04-04
|
||||
```
|
||||
|
||||
Run full pipeline:
|
||||
|
||||
```bash
|
||||
go run ./cmd/narratio run --config examples/pipeline.minimal.yml --session examples/session.minimal.yml
|
||||
```
|
||||
|
||||
Run analyze only:
|
||||
|
||||
```bash
|
||||
go run ./cmd/narratio run-stage --config examples/pipeline.minimal.yml --session examples/session.minimal.yml analyze
|
||||
```
|
||||
|
||||
Resume with a template session ID:
|
||||
|
||||
```bash
|
||||
go run ./cmd/narratio resume --config examples/pipeline.minimal.yml --session examples/session.template.yml --session-id 2026-04-04
|
||||
```
|
||||
|
||||
## Operational Note
|
||||
|
||||
Checksum-based stale detection is not implemented yet.
|
||||
|
||||
If prepared inputs or prompt/runtime config change, rerun the appropriate upstream stages before relying on downstream artifacts.
|
||||
|
||||
Examples:
|
||||
|
||||
- glossary/autocorrect/speaker-context changes: rerun at least `merge`, `polish`, `normalize`, `trim`, and `analyze`
|
||||
- trim bounds prompt/profile/config changes: rerun at least `normalize`, `trim`, and `analyze`
|
||||
- session recap prompt/profile/input-source changes: rerun `analyze`
|
||||
|
||||
## Roadmap
|
||||
|
||||
Near-term roadmap:
|
||||
|
||||
- extend analyze to additional configured artifacts
|
||||
- support workflows where later artifacts consume earlier generated artifacts
|
||||
- keep orchestration explicit without a generic DAG engine
|
||||
- implement archive and notify backends
|
||||
- [Configuration](docs/config.md)
|
||||
- [CLI Reference](docs/cli.md)
|
||||
- [Operations and Recovery](docs/operations.md)
|
||||
- [Troubleshooting](docs/troubleshooting.md)
|
||||
- [Development Guide](docs/development.md)
|
||||
- [Architecture Principles](docs/architecture.md)
|
||||
- [Internal Component Contracts](docs/internal/README.md)
|
||||
- [Config Examples](docs/examples/)
|
||||
|
||||
502
architecture.md
502
architecture.md
@@ -1,502 +0,0 @@
|
||||
# Narratio Architecture
|
||||
|
||||
## 1. Purpose
|
||||
|
||||
`narratio` is a Go orchestrator for D&D session processing. It runs a stage-based local pipeline from audio input through transcript processing and artifact generation, with manifest-based skip/force/resume behavior.
|
||||
|
||||
Narratio integrates with Scriptorium through the **public CLI** (`scriptorium run` and `scriptorium render`) via synchronous subprocess execution.
|
||||
|
||||
## 2. Current Status
|
||||
|
||||
Implemented:
|
||||
|
||||
- strict `pipeline.yml` + `session.yml` loading with strict YAML field checking (`KnownFields(true)`)
|
||||
- local workspace/session layout, lock file handling, artifact path helpers, checksums, and atomic writes
|
||||
- manifest store and stage status transitions for resumable runs
|
||||
- real `prepare`, `transcribe`, `merge`, and `polish` stages
|
||||
- real WhisperX HTTP adapter
|
||||
- real Seriatim subprocess adapter
|
||||
- real Audita subprocess adapter
|
||||
- real Scriptorium subprocess adapter
|
||||
- real `normalize` stage producing `transcripts/normalized.json`
|
||||
- real `trim` stage producing `transcripts/trimmed.json`
|
||||
- real `analyze` stage for initial `session_recap` generation
|
||||
- optional Scriptorium render diagnostics (`render_debug`) before production run
|
||||
- storage/archive configuration and validation foundations for:
|
||||
- `pipeline.storage.s3`
|
||||
- `pipeline.spool`
|
||||
- `pipeline.archive` promotion rules
|
||||
- `session.inputs.audio_s3`
|
||||
- run identity and path-model foundations:
|
||||
- run ID generation (`YYYYMMDDTHHMMSSZ-xxxxxxxx`)
|
||||
- S3 session/run/current key builders
|
||||
- campaign/session/run local work/spool path helpers
|
||||
- manifest run/path identity fields (`campaign`, `run_id`, local and S3 prefixes)
|
||||
- remote storage backend layer:
|
||||
- narrow object-store interface (`List`, `Download`, `Upload`, `Exists`)
|
||||
- fake storage backend for deterministic tests (no network dependency)
|
||||
- S3-compatible backend using AWS SDK v2
|
||||
- config-based object-store construction helper
|
||||
|
||||
Still placeholder/future:
|
||||
|
||||
- `notify` stage behavior
|
||||
- additional Scriptorium artifact types beyond `session_recap`
|
||||
- artifact-to-artifact workflows beyond the initial single-artifact implementation
|
||||
- generic stale detection based on input/config checksums
|
||||
|
||||
## 3. Pipeline and Stage Boundaries
|
||||
|
||||
Canonical stage order:
|
||||
|
||||
1. `prepare`
|
||||
2. `transcribe`
|
||||
3. `merge`
|
||||
4. `polish`
|
||||
5. `normalize`
|
||||
6. `trim`
|
||||
7. `analyze`
|
||||
8. `archive`
|
||||
9. `notify`
|
||||
|
||||
Boundary rules:
|
||||
|
||||
- orchestration logic lives in `internal/app`
|
||||
- stage business logic lives in `internal/stage`
|
||||
- external-tool CLI construction lives in adapter packages
|
||||
- Scriptorium CLI details stay in `internal/adapters/scriptorium`
|
||||
|
||||
## 4. Scriptorium Integration Model
|
||||
|
||||
Integration mode:
|
||||
|
||||
- public CLI subprocesses only (no Scriptorium internal Go packages, no HTTP API)
|
||||
- production generation uses `scriptorium run`
|
||||
- diagnostics/testing render uses `scriptorium render --format json`
|
||||
|
||||
Run invocation shape used by adapter:
|
||||
|
||||
```bash
|
||||
scriptorium run --prompt <prompt_id> --input name=path --out <output_path>
|
||||
```
|
||||
|
||||
Optional flags passed when configured:
|
||||
|
||||
- `--config <path>`
|
||||
- `--profile <profile_id>`
|
||||
- repeated `--var name=value`
|
||||
- repeated `--input name=path`
|
||||
- `--timeout <duration>`
|
||||
- `--api-key-env <ENV_NAME>` when configured
|
||||
|
||||
Render invocation shape used by adapter:
|
||||
|
||||
```bash
|
||||
scriptorium render --prompt <prompt_id> --input name=path --format json --out <render_output_path>
|
||||
```
|
||||
|
||||
Adapter behavior:
|
||||
|
||||
- always passes `--out`
|
||||
- captures stdout/stderr separately
|
||||
- writes generated invocation metadata YAML (redacted, no secrets)
|
||||
- treats exit code `0` as success
|
||||
- treats exit code `1` as failure
|
||||
- treats exit code `2` as failure with `validation_failed=true` and preserves output metadata when available
|
||||
- validates successful output files exist and are non-empty
|
||||
- does not treat non-empty stderr as failure by itself
|
||||
|
||||
## 5. Configuration Contract
|
||||
|
||||
CLI pipeline config path resolution:
|
||||
|
||||
- when `--config <path>` is provided, that path is used
|
||||
- when `--config` is omitted, Narratio searches defaults in order:
|
||||
- `/usr/local/etc/narratio/pipeline.yml`
|
||||
- `/etc/narratio/pipeline.yml`
|
||||
- default values are centralized in `internal/config/defaults.go`
|
||||
|
||||
CLI session config path resolution:
|
||||
|
||||
- when `--session <path>` is provided, that path is used
|
||||
- when `--session` is omitted, Narratio searches defaults in order:
|
||||
- `./session.yml`
|
||||
- `/usr/local/etc/narratio/session.yml`
|
||||
- `/etc/narratio/session.yml`
|
||||
|
||||
Session template rendering:
|
||||
|
||||
- session templates are rendered before strict YAML decode
|
||||
- `--session-id <value>` provides the `session_id` template variable
|
||||
- supported placeholders:
|
||||
- `{{session_id}}`
|
||||
- `{{ session_id }}`
|
||||
- unresolved placeholders fail clearly
|
||||
- strict `KnownFields(true)` YAML validation still applies after rendering
|
||||
- if rendered `session.session_id` conflicts with `--session-id`, load fails clearly
|
||||
|
||||
Optional pipeline secrets directory:
|
||||
|
||||
- `pipeline.secrets.env_dir` enables loading environment variables from local files before command execution
|
||||
- file name = env var name; file contents = env var value (trailing newline/CRLF trimmed)
|
||||
- only env-var-style file names are considered; other entries are ignored
|
||||
- existing process environment values are preserved (not overwritten)
|
||||
- if configured, unreadable/missing `env_dir` fails command execution early
|
||||
- relative `env_dir` values are resolved from current working directory
|
||||
|
||||
Storage and archive foundations:
|
||||
|
||||
- `pipeline.storage.s3` is available for modeling S3 coordinates:
|
||||
- `bucket`
|
||||
- `root_prefix` (default `dnd`)
|
||||
- `region`
|
||||
- `endpoint`
|
||||
- `force_path_style` (default `false`)
|
||||
- `access_key_id_env` (default `OBJECT_STORAGE_KEY_ID`)
|
||||
- `secret_access_key_env` (default `OBJECT_STORAGE_KEY`)
|
||||
- `pipeline.spool.root` defaults to `/var/spool/narratio`
|
||||
- `pipeline.workspace.cleanup_after_archive` defaults to `false`
|
||||
- `pipeline.spool.delete_audio_after_archive` defaults to `false`
|
||||
- `pipeline.archive` is optional and defaults to:
|
||||
- `enabled: true`
|
||||
- `upload_run: true`
|
||||
- default `promote_artifacts`:
|
||||
- `transcripts/trimmed.json`
|
||||
- `artifacts/session_recap.md`
|
||||
- archive promotion rules enforce safe relative paths:
|
||||
- `from` and `to` are required
|
||||
- absolute paths are rejected
|
||||
- traversal segments such as `..` are rejected
|
||||
|
||||
Session input foundations:
|
||||
|
||||
- `session.campaign` is required
|
||||
- local audio remains supported through `session.inputs.audio_dir` or `session.inputs.audio_files`
|
||||
- optional S3 audio input shape is `session.inputs.audio_s3.prefix`
|
||||
- `audio_dir`/`audio_files` and `audio_s3` are mutually exclusive
|
||||
- when `audio_s3` is configured, `prepare` lists and downloads `.flac` objects through the object-store backend
|
||||
|
||||
Cross-config validation scope:
|
||||
|
||||
- `pipeline.storage.s3.bucket` is required only when an S3-dependent feature is explicitly configured (for current foundations, that includes `session.inputs.audio_s3`, and archive upload intent when using `storage.backend: s3`)
|
||||
- no AWS credential values are stored in Narratio config; only env-var names are configured
|
||||
- when both configured credential env vars resolve to non-empty values, the S3 backend uses them as static credentials
|
||||
- when either configured credential value is missing, the S3 backend falls back to the AWS SDK default credential chain
|
||||
|
||||
Remote object-store backend scope:
|
||||
|
||||
- remote storage APIs are isolated to `internal/adapters/storage`
|
||||
- AWS SDK types remain contained within the S3 backend implementation package
|
||||
- S3 key/session path semantics remain outside the backend, with this invariant:
|
||||
- callers pass full bucket-relative object keys
|
||||
- backend methods do not prepend `root_prefix` or infer campaign/session/run paths
|
||||
- `prepare` now uses object-store `List` and `Download` for S3 audio input
|
||||
- `archive` now uses object-store `Upload` for successful run-record upload under the run prefix
|
||||
- `archive` now uses object-store `Upload` for promoted outputs and current pointers
|
||||
|
||||
Prepare S3 audio behavior (implemented):
|
||||
|
||||
- compute session prefix as `{root_prefix}/campaigns/{campaign}/sessions/{session_id}/`
|
||||
- resolve `session.inputs.audio_s3.prefix` under that session prefix
|
||||
- list objects under the computed audio prefix and filter `.flac` keys
|
||||
- fail clearly when no `.flac` objects are found
|
||||
- download selected objects to spool audio path:
|
||||
- `{spool.root}/{campaign}/{session_id}/{run_id}/audio/`
|
||||
- materialize audio files into workdir audio path:
|
||||
- `{workspace.root}/work/{campaign}/{session_id}/{run_id}/audio/`
|
||||
- record S3 provenance in manifest input records (bucket/key/metadata/local paths/checksum)
|
||||
- no AWS SDK types are used in stage code; storage implementation details stay in storage adapter packages
|
||||
|
||||
Archive publishing behavior (implemented):
|
||||
|
||||
- `archive` verifies prerequisite stage success before upload:
|
||||
- `prepare`, `transcribe`, `merge`, `polish`, `normalize`, `trim`, `analyze`
|
||||
- only successful/completed runs are uploaded
|
||||
- uploaded run record destination is:
|
||||
- `{root_prefix}/campaigns/{campaign}/sessions/{session_id}/runs/{run_id}/`
|
||||
- uploaded existing local paths include:
|
||||
- `inputs/`, `transcripts/`, `artifacts/`, optional `reports/`, `config/`, `logs/`, and `manifest.json`
|
||||
- local `audio/` is intentionally excluded from upload by default
|
||||
- file upload order is deterministic (sorted relative paths)
|
||||
- `archive.enabled: false` and `archive.upload_run: false` skip upload cleanly
|
||||
- stage metadata records non-secret upload context:
|
||||
- run upload details, promoted output details, current manifest key, current pointer key
|
||||
- no secrets, transcript contents, prompt contents, or environment dumps
|
||||
- promotion rules:
|
||||
- `from` resolves from local workdir
|
||||
- `to` resolves under session-level S3 root
|
||||
- missing required source fails archive
|
||||
- missing optional source is skipped and recorded
|
||||
- default promoted outputs:
|
||||
- `transcripts/trimmed.json`
|
||||
- `artifacts/session_recap.md`
|
||||
- current pointers:
|
||||
- `current/manifest.json` uploaded after run upload and promotions
|
||||
- `current/run_id.txt` uploaded last with `{run_id}\n`
|
||||
- `current/run_id.txt` is the effective commit marker
|
||||
- if promotion or current-manifest upload fails, archive returns failure and does not write `current/run_id.txt`
|
||||
- failed/incomplete runs remain local and are not uploaded
|
||||
- post-archive local cleanup (implemented, opt-in):
|
||||
- cleanup runs only after archive succeeded and wrote `current/run_id.txt`
|
||||
- cleanup is executed after all selected stages in the command invocation succeed (for example, a later `notify` failure leaves local files intact)
|
||||
- `pipeline.spool.delete_audio_after_archive: true` removes only `{spool.root}/{campaign}/{session_id}/{run_id}/audio/`
|
||||
- `pipeline.workspace.cleanup_after_archive: true` removes only `{workspace.root}/work/{campaign}/{session_id}/{run_id}/`
|
||||
- cleanup does not run when archive is skipped/disabled/fails or when run upload is disabled
|
||||
- local development `audio_dir`/`audio_files` inputs are never removed by spool cleanup
|
||||
|
||||
`pipeline.scriptorium` is optional. Existing pipelines without Scriptorium continue to work.
|
||||
|
||||
`pipeline.trim` is optional. Existing pipelines without trim config continue to work.
|
||||
|
||||
`pipeline.normalize` is optional. Existing pipelines without normalize config continue to work.
|
||||
|
||||
`pipeline.audita` drives the real Audita subprocess adapter for the `polish` stage.
|
||||
|
||||
Audita defaulted fields:
|
||||
|
||||
- `binary` defaults to `audita`
|
||||
- `timeout` defaults to `3h`
|
||||
- `report` defaults to `true`
|
||||
|
||||
Audita optional fields:
|
||||
|
||||
- `llm_api_key_env` (enforced only when configured)
|
||||
- `modules` override list (when omitted/empty, Narratio does not pass `--modules`)
|
||||
- `base_url` (when omitted, Narratio does not pass `--base-url`)
|
||||
- `model` (when omitted, Narratio does not pass `--model`)
|
||||
- `transcript_description`
|
||||
- `config_path`
|
||||
- `output_schema` (`bare-segments` or `audita-v1`)
|
||||
- `work_dir_retention` (`always`, `auto`, `never`)
|
||||
- `total_llm_concurrency` (> 0 when provided)
|
||||
- `proposal_llm_concurrency` (> 0 when provided)
|
||||
- `validation_model`
|
||||
- `validation_llm_concurrency` (> 0 when provided)
|
||||
- `report` override
|
||||
|
||||
Narratio passes only configured optional Audita flags; omitted optional values defer to Audita runtime defaults/config.
|
||||
|
||||
Seriatim defaults:
|
||||
|
||||
- `pipeline.seriatim` may be omitted
|
||||
- `binary` defaults to `seriatim`
|
||||
- `timeout` defaults to `10m`
|
||||
- `output_schema` defaults to `seriatim-intermediate`
|
||||
- `coalesce_gap` defaults to `3.0`
|
||||
- `report` defaults to `true`
|
||||
|
||||
When `pipeline.normalize` is omitted, defaults are applied:
|
||||
|
||||
- `output_path: transcripts/normalized.json`
|
||||
- `output_schema: seriatim-intermediate`
|
||||
- `report: true`
|
||||
|
||||
When `pipeline.normalize` is present:
|
||||
|
||||
- `output_path` must be non-empty
|
||||
- `output_schema` must be one of `seriatim-minimal`, `seriatim-intermediate`, or `seriatim-full`
|
||||
- relative `output_path` values are session-workdir-relative paths
|
||||
- Seriatim binary settings still come from `pipeline.seriatim`
|
||||
|
||||
When `pipeline.trim` is present:
|
||||
|
||||
- `enabled` is optional and defaults to `false` when omitted
|
||||
- relative `output_path`, `bounds.output_path`, and `bounds.render_output_path` values are session-workdir-relative paths
|
||||
- do not store secrets in trim config values
|
||||
|
||||
When `pipeline.trim.enabled: true`:
|
||||
|
||||
- `output_path` is required and non-empty
|
||||
- `bounds.prompt_id` is required and non-empty
|
||||
- `bounds.transcript_input_name` is required and non-empty
|
||||
- `bounds.output_path` is required and non-empty
|
||||
- `bounds.timeout` must parse as a Go duration when provided
|
||||
- `bounds.render_debug: true` requires non-empty `bounds.render_output_path`
|
||||
- `bounds.profile_id` may be empty to use the prompt default profile
|
||||
- prompt IDs are config values, not hardcoded stage logic
|
||||
|
||||
When `pipeline.scriptorium` is present:
|
||||
|
||||
- `binary` defaults to `scriptorium` when omitted
|
||||
- `config_path` is optional; when provided it must be non-empty
|
||||
- `timeout` is optional; when provided it must parse as a Go duration
|
||||
- default `timeout` is `10m`
|
||||
- unknown YAML fields fail strict decode
|
||||
|
||||
Artifacts are configured as a map under `pipeline.scriptorium.artifacts` so multiple artifacts are possible in the config shape.
|
||||
|
||||
For each artifact definition:
|
||||
|
||||
- `enabled: true` requires non-empty `prompt_id`
|
||||
- `enabled: true` requires non-empty `output_path`
|
||||
- `timeout` must parse as Go duration when present
|
||||
- optional per-artifact `render_debug` may override global `scriptorium.render_debug`
|
||||
- `inputs` are named and each input requires non-empty `source`
|
||||
- inputs may be optional (`required: false`)
|
||||
- `vars` values currently support `string` and `bool`
|
||||
|
||||
Prompt IDs and profile IDs are configuration values, not hardcoded stage logic.
|
||||
|
||||
Trim config shape:
|
||||
|
||||
```yaml
|
||||
trim:
|
||||
enabled: true
|
||||
output_path: "transcripts/trimmed.json"
|
||||
bounds:
|
||||
prompt_id: "dnd_session.bounds"
|
||||
profile_id: ""
|
||||
transcript_input_name: "transcript"
|
||||
output_path: "artifacts/session_bounds.json"
|
||||
timeout: "10m"
|
||||
render_debug: false
|
||||
render_output_path: "artifacts/session_bounds.render.json"
|
||||
seriatim:
|
||||
report: false
|
||||
```
|
||||
|
||||
## 6. Transcript Tiers
|
||||
|
||||
Narratio currently produces and uses four transcript tiers:
|
||||
|
||||
- `transcripts/merged.json`: canonical deterministic merged transcript from Seriatim merge
|
||||
- `transcripts/processed.json`: full raw Audita-polished transcript output (includes pre/post-game content)
|
||||
- `transcripts/normalized.json`: normalized transcript generated by Seriatim normalize
|
||||
- `transcripts/trimmed.json`: gameplay-only normalized polished transcript from trim stage
|
||||
|
||||
Trim reads `transcripts/normalized.json`, validates bounds IDs against that same transcript ID space, and writes `transcripts/trimmed.json`.
|
||||
|
||||
## 7. Normalize Stage (Current Implementation)
|
||||
|
||||
Normalize stage behavior:
|
||||
|
||||
- stage order position: after `polish` and before `trim`
|
||||
- discovers processed transcript from manifest polish outputs (`transcript_processed`) when present, else `work/<session_id>/transcripts/processed.json`
|
||||
- validates processed transcript JSON shape (`segments` array required)
|
||||
- runs Seriatim `normalize` to produce normalized transcript
|
||||
- validates normalized transcript JSON shape (`segments` array required)
|
||||
- validates normalize report JSON when enabled
|
||||
|
||||
Expected normalize outputs and diagnostics:
|
||||
|
||||
- `transcripts/normalized.json`
|
||||
- `artifacts/seriatim.normalize.report.json` (when normalize report is enabled)
|
||||
- `logs/seriatim.normalize.stdout.log`
|
||||
- `logs/seriatim.normalize.stderr.log`
|
||||
- `config/seriatim.normalize.generated.yml`
|
||||
|
||||
## 8. Trim Stage (Current Implementation)
|
||||
|
||||
Trim stage behavior:
|
||||
|
||||
- stage order position: after `normalize` and before `analyze`
|
||||
- discovers normalized transcript from manifest normalize outputs (`transcript_normalized`) when present, else `work/<session_id>/transcripts/normalized.json`
|
||||
- validates normalized transcript JSON shape (`segments` array required)
|
||||
- when `trim.enabled: false` (or trim config omitted), deterministically copies normalized transcript to `transcripts/trimmed.json` and records `trim_action=copy_disabled`
|
||||
- when `trim.enabled: true`:
|
||||
- runs Scriptorium bounds prompt using configured `trim.bounds.prompt_id`
|
||||
- writes bounds output to configured path (typically `artifacts/session_bounds.json`)
|
||||
- parses and validates bounds output against the same normalized transcript being trimmed
|
||||
- converts bounds range to Seriatim keep selector (for example `10-868`)
|
||||
- runs Seriatim `trim` to produce `transcripts/trimmed.json`
|
||||
- supports no-trim bounds actions (`none`/`copy`) by copying normalized transcript unchanged
|
||||
- validates trimmed transcript JSON shape (`segments` array required)
|
||||
|
||||
Expected trim outputs and diagnostics:
|
||||
|
||||
- `artifacts/session_bounds.json`
|
||||
- `transcripts/trimmed.json`
|
||||
- `logs/scriptorium.bounds.stdout.log`
|
||||
- `logs/scriptorium.bounds.stderr.log`
|
||||
- `config/scriptorium.bounds.generated.yml`
|
||||
- `logs/seriatim.trim.stdout.log`
|
||||
- `logs/seriatim.trim.stderr.log`
|
||||
- `config/seriatim.trim.generated.yml`
|
||||
- optional bounds render-debug outputs when enabled:
|
||||
- `artifacts/session_bounds.render.json`
|
||||
- `logs/scriptorium.bounds.render.stdout.log`
|
||||
- `logs/scriptorium.bounds.render.stderr.log`
|
||||
- `config/scriptorium.bounds.render.generated.yml`
|
||||
|
||||
Render-debug files are diagnostics. They are recorded in stage metadata/log/config refs and are not treated as canonical stage output artifact refs.
|
||||
|
||||
## 9. Analyze Stage (Current Implementation)
|
||||
|
||||
The current real analyze implementation supports only `scriptorium.artifacts.session_recap`.
|
||||
|
||||
Behavior:
|
||||
|
||||
- if `pipeline.scriptorium` is missing, analyze returns a skipped result with metadata
|
||||
- if no Scriptorium artifacts are enabled, analyze returns a skipped result with metadata
|
||||
- if enabled artifacts exist but `session_recap` is not enabled, analyze fails clearly
|
||||
- available transcript input sources for configured artifacts: `processed_transcript`, `normalized_transcript`, `trimmed_transcript`
|
||||
- `session_recap` should use `trimmed_transcript` input (`transcripts/trimmed.json`) for in-universe recap generation
|
||||
- `trimmed_transcript` input is resolved from manifest (`trim` output kind `transcript_trimmed`) when available, otherwise fallback path `work/<session_id>/transcripts/trimmed.json`
|
||||
- `normalized_transcript` input is resolved from manifest (`normalize` output kind `transcript_normalized`) when available, otherwise fallback path `work/<session_id>/transcripts/normalized.json`
|
||||
- `processed_transcript` input is resolved from manifest (`polish` output kind `transcript_processed`) when available, otherwise fallback path `work/<session_id>/transcripts/processed.json`
|
||||
- `normalized_transcript` is the preferred full-transcript source for future table/meta-analysis artifacts
|
||||
- `processed_transcript` remains available for advanced/debug use cases
|
||||
- transcript inputs are validated as JSON with top-level `segments` array
|
||||
- configured inputs are resolved by source
|
||||
- optional `previous_recap` is omitted when unavailable
|
||||
- required `previous_recap` fails before invocation when unavailable
|
||||
- vars are built from config + session metadata
|
||||
- `render_debug` controls pre-run `scriptorium render` diagnostics
|
||||
- render failure stops stage before production run
|
||||
- render output is validated as JSON
|
||||
- production call uses Scriptorium adapter `RunArtifact`
|
||||
- successful run output must exist and be non-empty
|
||||
- missing `trimmed_transcript` input for configured `trimmed_transcript` source fails clearly with guidance to run trim stage first
|
||||
- manifest records output refs, logs, generated config paths, and non-secret provenance metadata
|
||||
|
||||
## 10. Session Recap Paths
|
||||
|
||||
Current expected paths for `session_recap`:
|
||||
|
||||
- artifact output: `artifacts/session_recap.md`
|
||||
- run stdout log: `logs/scriptorium.session_recap.stdout.log`
|
||||
- run stderr log: `logs/scriptorium.session_recap.stderr.log`
|
||||
- run generated invocation/config: `config/scriptorium.session_recap.generated.yml`
|
||||
- render output (when enabled): `artifacts/session_recap.render.json`
|
||||
- render stdout log: `logs/scriptorium.session_recap.render.stdout.log`
|
||||
- render stderr log: `logs/scriptorium.session_recap.render.stderr.log`
|
||||
- render generated invocation/config: `config/scriptorium.session_recap.render.generated.yml`
|
||||
|
||||
## 11. Security and Privacy
|
||||
|
||||
- do not store secrets in pipeline YAML, generated invocation YAML, logs, or manifest metadata
|
||||
- if API-key integration is configured, pass env var names only (never raw key values)
|
||||
- with `pipeline.secrets.env_dir`, secret file values are loaded into process env only and are not persisted in manifest metadata or generated configs
|
||||
- avoid logging transcript content or rendered prompt content by default
|
||||
- treat generated artifacts and logs as potentially sensitive session material
|
||||
|
||||
## 12. Operational Caveat (Pre-Stale-Detection)
|
||||
|
||||
Checksum-based stale detection is not implemented yet.
|
||||
|
||||
If prepared inputs or prompt/runtime configuration change (for example glossary files, prompt IDs, profile IDs, or relevant pipeline settings), rerun the appropriate prior stages to refresh downstream artifacts.
|
||||
|
||||
Examples:
|
||||
|
||||
- glossary or autocorrect changes usually require rerunning at least `merge`, `polish`, `normalize`, `trim`, and `analyze`
|
||||
- trim prompt/profile changes require rerunning at least `normalize`, `trim`, and `analyze`
|
||||
- session recap prompt/profile/input-source changes require rerunning `analyze`
|
||||
|
||||
## 13. Roadmap
|
||||
|
||||
Planned next steps:
|
||||
|
||||
- extend analyze beyond `session_recap` to additional configured artifacts
|
||||
- support artifact inputs that consume prior generated artifacts
|
||||
- keep this composable without adding a generic DAG engine in the near term
|
||||
- implement real `archive` backend behavior
|
||||
- implement real `notify` backend behavior
|
||||
- add checksum-based stale detection and stale transitions
|
||||
|
||||
Architectural invariants remain:
|
||||
|
||||
- strict config decoding/validation
|
||||
- manifest-driven run control
|
||||
- clear stage/adapter separation
|
||||
- configuration-driven prompt/profile/input/vars/output mapping
|
||||
- Scriptorium integration through public CLI subprocess contract
|
||||
202
docs/architecture.md
Normal file
202
docs/architecture.md
Normal file
@@ -0,0 +1,202 @@
|
||||
# Narratio Architecture
|
||||
|
||||
## Purpose
|
||||
|
||||
`narratio` is a Go orchestration application for processing D&D session audio into polished transcripts and generated session artifacts.
|
||||
|
||||
This document defines the development principles for the project. It is inward-facing: its audience is developers and LLM coding agents. It should guide future changes, not serve as a complete implementation reference.
|
||||
|
||||
Implemented component details belong under `docs/internal/`.
|
||||
|
||||
## Project Shape
|
||||
|
||||
Narratio is a modular, stage-driven orchestrator.
|
||||
|
||||
It coordinates specialized downstream systems rather than reimplementing their domains:
|
||||
|
||||
- WhisperX handles transcription.
|
||||
- Seriatim handles deterministic transcript merge/normalization/trim behavior.
|
||||
- Audita handles transcript correction and polishing.
|
||||
- Scriptorium handles prompt execution and generated artifacts.
|
||||
|
||||
Narratio owns orchestration, configuration loading, session/run state, local and remote path modeling, manifest persistence, stage sequencing, resume behavior, and archive semantics.
|
||||
|
||||
Narratio should remain explicit and comprehensible. It is not intended to become a generic workflow engine.
|
||||
|
||||
## Core Principles
|
||||
|
||||
### Modular and composable
|
||||
|
||||
Code should be organized around clear responsibilities. Stages, adapters, config loading, manifest persistence, path construction, and storage behavior should remain separable and independently testable.
|
||||
|
||||
### Hexagonal boundaries
|
||||
|
||||
External systems should be isolated behind narrow adapters. Stage logic should depend on Narratio-level interfaces and data structures, not on external SDK types, subprocess argument construction, or transport-specific details.
|
||||
|
||||
### Standard library preference
|
||||
|
||||
Prefer the Go standard library. Add dependencies only when they provide substantial value, are necessary for an external integration, or are a widely used de facto standard.
|
||||
|
||||
Accepted examples include a YAML library for configuration and the AWS SDK for S3-compatible storage.
|
||||
|
||||
### Explicit orchestration
|
||||
|
||||
The pipeline should remain stage-driven and explicit. New behavior should be added through clear stage, adapter, config, or manifest contracts rather than implicit side effects or generic workflow abstraction.
|
||||
|
||||
## Stage Design
|
||||
|
||||
Each stage should have a clear scope of responsibility.
|
||||
|
||||
A stage should define:
|
||||
|
||||
- its purpose;
|
||||
- required input state;
|
||||
- produced output state;
|
||||
- config fields it consumes;
|
||||
- external adapters it uses;
|
||||
- manifest refs it reads or writes;
|
||||
- skip, force, and resume behavior;
|
||||
- failure behavior;
|
||||
- tests that protect its contract.
|
||||
|
||||
Stages should avoid reaching across boundaries. If shared behavior is needed, prefer a helper or service with a narrow interface over duplicating ad hoc logic between stages.
|
||||
|
||||
## Transactionality and Resume
|
||||
|
||||
A stage should behave transactionally.
|
||||
|
||||
A stage is complete only when its outputs have been written, validated, and recorded in the manifest. If a stage fails, Narratio should preserve enough local state for inspection, recovery, and resume.
|
||||
|
||||
A failed or incomplete run must not be treated as successful. Later stages should depend on manifest-recorded success, not merely on incidental files existing on disk.
|
||||
|
||||
## Manifest Model
|
||||
|
||||
The manifest is the durable local ledger for a run.
|
||||
|
||||
It should record:
|
||||
|
||||
- run identity;
|
||||
- stage status;
|
||||
- input and output refs;
|
||||
- logs and generated config refs;
|
||||
- checksums or provenance where useful;
|
||||
- non-secret adapter and archive metadata.
|
||||
|
||||
Resume behavior should be manifest-driven. Filesystem state may be inspected and validated, but it should not replace manifest stage state as the source of run progress.
|
||||
|
||||
## Adapter Boundaries
|
||||
|
||||
Adapters own external integration details.
|
||||
|
||||
Expected boundaries:
|
||||
|
||||
- WhisperX HTTP details stay in the WhisperX adapter.
|
||||
- Seriatim CLI construction stays in the Seriatim adapter.
|
||||
- Audita CLI construction stays in the Audita adapter.
|
||||
- Scriptorium CLI construction stays in the Scriptorium adapter.
|
||||
- Object-storage details stay behind the storage adapter interface.
|
||||
- AWS SDK types stay inside the S3 storage implementation.
|
||||
|
||||
Stage code should express intent in Narratio terms and call adapters through narrow contracts.
|
||||
|
||||
## Configuration Philosophy
|
||||
|
||||
Configuration should be strict, explicit, and operator-friendly.
|
||||
|
||||
Principles:
|
||||
|
||||
- YAML decoding should reject unknown fields.
|
||||
- Defaults should be centralized and testable.
|
||||
- Empty configured values should not silently override meaningful defaults.
|
||||
- Session templating should remain narrow and deterministic.
|
||||
- Template support should serve operator convenience, not become a general configuration language.
|
||||
|
||||
Narratio should not become a secondary configuration system for downstream tools. Seriatim, Audita, and Scriptorium should own their runtime defaults wherever practical. Narratio should pass required stage-contract paths and explicit operator overrides.
|
||||
|
||||
## Path and Storage Discipline
|
||||
|
||||
Local and remote paths are part of Narratio’s application contract.
|
||||
|
||||
Code should use centralized path helpers for workspace, spool, session, run, artifact, log, config, and archive paths. Stages should avoid reconstructing canonical paths through scattered string concatenation.
|
||||
|
||||
Storage backends should receive explicit bucket-relative keys. Storage implementations should not infer campaign, session, run, or root-prefix semantics.
|
||||
|
||||
## Archive Invariants
|
||||
|
||||
Archive behavior must preserve a clear commit boundary.
|
||||
|
||||
A remote run is current only after the archive stage has successfully uploaded the run record, required promoted outputs, `current/manifest.json`, and finally `current/run_id.txt`.
|
||||
|
||||
`current/run_id.txt` is the final remote commit marker and must be written last.
|
||||
|
||||
Failed, incomplete, skipped, or uncommitted archive attempts must not be presented as current remote state. Local cleanup is permitted only after successful archive commit and only when explicitly configured.
|
||||
|
||||
## Security and Privacy
|
||||
|
||||
Narratio handles private campaign material.
|
||||
|
||||
Rules:
|
||||
|
||||
- Do not store raw secrets in pipeline or session YAML.
|
||||
- Use environment variable names or secret-file references for secret handling.
|
||||
- Do not write raw secret values to manifests, logs, generated configs, or archive metadata.
|
||||
- Treat transcripts, generated artifacts, prompts, reports, and logs as potentially sensitive.
|
||||
- Avoid logging transcript or prompt content unless there is a deliberate diagnostic reason.
|
||||
|
||||
## Diagnostics
|
||||
|
||||
Diagnostics should be durable and discoverable, but distinct from canonical outputs.
|
||||
|
||||
Logs, reports, generated invocation/config files, and render-debug files support debugging. Transcript tiers and configured artifacts are pipeline products.
|
||||
|
||||
Manifest refs should preserve that distinction.
|
||||
|
||||
## Determinism
|
||||
|
||||
Where practical, Narratio should prefer deterministic behavior:
|
||||
|
||||
- stable local path layout;
|
||||
- stable remote key layout;
|
||||
- sorted upload order;
|
||||
- predictable generated config files;
|
||||
- repeatable command construction;
|
||||
- tests that do not depend on live external services.
|
||||
|
||||
Run IDs and timestamps may be intentionally variable, but surrounding behavior should remain testable.
|
||||
|
||||
## Testing Expectations
|
||||
|
||||
Core behavior should be testable without live external services.
|
||||
|
||||
Tests should cover:
|
||||
|
||||
- config loading, defaults, and validation;
|
||||
- CLI parsing and command construction;
|
||||
- path helpers;
|
||||
- manifest transitions;
|
||||
- stage success, failure, skip, and resume behavior;
|
||||
- adapter command construction;
|
||||
- fake storage behavior;
|
||||
- archive commit ordering;
|
||||
- example config validity where practical.
|
||||
|
||||
Live S3, WhisperX, LLM, or subprocess integration tests should be explicit integration tests, not required for ordinary unit test runs.
|
||||
|
||||
## Documentation Expectations
|
||||
|
||||
Documentation must follow `docs/documentation/policy.md`.
|
||||
|
||||
Current behavior belongs in user-facing docs and `docs/internal/`. Future, planned, aspirational, experimental, or unimplemented work belongs only under `docs/roadmap/`.
|
||||
|
||||
`docs/architecture.md` should remain concise and principle-focused. It should not duplicate the full config reference, CLI reference, operations guide, or internal stage documentation.
|
||||
|
||||
## Non-Goals
|
||||
|
||||
Narratio is not:
|
||||
|
||||
- a generic DAG or workflow engine;
|
||||
- a replacement configuration layer for Seriatim, Audita, or Scriptorium;
|
||||
- a storage backend abstraction beyond the needs of this pipeline;
|
||||
- a place to embed raw secrets;
|
||||
- a place for stage logic to depend directly on AWS SDK types or downstream tool internals;
|
||||
- a prompt-authoring system.
|
||||
230
docs/cli.md
Normal file
230
docs/cli.md
Normal file
@@ -0,0 +1,230 @@
|
||||
# CLI
|
||||
|
||||
## Shortest Useful Command
|
||||
|
||||
```bash
|
||||
narratio run --session-id 2026-04-04
|
||||
```
|
||||
|
||||
This uses default config discovery for `pipeline.yml` and `session.yml`; both files must be discoverable for this command to run.
|
||||
|
||||
## Command Overview
|
||||
|
||||
Implemented commands:
|
||||
|
||||
- `run`: execute the full stage plan and persist manifest state.
|
||||
- `plan`: validate config, prepare workdir, and print run/skip decisions.
|
||||
- `resume`: continue from the first non-succeeded stage in the manifest.
|
||||
- `status`: read and print stage statuses from an existing manifest file.
|
||||
- `run-stage`: execute exactly one selected stage.
|
||||
|
||||
Unknown commands print usage (`Usage: narratio <run|plan|status|resume|run-stage>`) and exit non-zero.
|
||||
|
||||
For configuration field details, see [docs/config.md](./config.md). For operational lifecycle details, see [docs/operations.md](./operations.md).
|
||||
|
||||
## Complete Flag Reference
|
||||
|
||||
### `run`
|
||||
|
||||
- `--config <path>`: optional explicit `pipeline.yml` path; if omitted, default locations are searched.
|
||||
- `--session <path>`: optional explicit `session.yml` path; if omitted, default locations are searched.
|
||||
- `--session-id <value>`: session template variable value for `session.yml` rendering.
|
||||
- `--force`: force stage execution (prevents skip of already-succeeded stages).
|
||||
|
||||
### `plan`
|
||||
|
||||
- `--config <path>`
|
||||
- `--session <path>`
|
||||
- `--session-id <value>`
|
||||
- `--force`: show forced run decisions instead of normal skip behavior.
|
||||
|
||||
### `resume`
|
||||
|
||||
- `--config <path>`
|
||||
- `--session <path>`
|
||||
- `--session-id <value>`
|
||||
- `--force`: run full stage order rather than starting at first non-succeeded stage.
|
||||
|
||||
### `run-stage`
|
||||
|
||||
- `--config <path>`
|
||||
- `--session <path>`
|
||||
- `--session-id <value>`
|
||||
- `--force`
|
||||
- positional `<stage>`: required stage name.
|
||||
|
||||
Valid stage names:
|
||||
|
||||
- `prepare`
|
||||
- `transcribe`
|
||||
- `merge`
|
||||
- `polish`
|
||||
- `normalize`
|
||||
- `trim`
|
||||
- `analyze`
|
||||
- `archive`
|
||||
- `notify`
|
||||
|
||||
### `status`
|
||||
|
||||
- `--manifest <path>`: required manifest path.
|
||||
|
||||
## Command Reference
|
||||
|
||||
### `run`
|
||||
|
||||
Purpose:
|
||||
|
||||
- Validate configuration and execute all stages in canonical order.
|
||||
|
||||
Syntax:
|
||||
|
||||
```bash
|
||||
narratio run [--config <pipeline.yml>] [--session <session.yml>] [--session-id <id>] [--force]
|
||||
```
|
||||
|
||||
Success output:
|
||||
|
||||
- `narratio run: session <session_id>; executed=<n> skipped=<n>; manifest=<path>`
|
||||
|
||||
Common failure cases:
|
||||
|
||||
- no pipeline config found in default search paths when `--config` is omitted.
|
||||
- no session config found in default search paths when `--session` is omitted.
|
||||
- invalid flags or unexpected positional arguments.
|
||||
- config/template/validation errors.
|
||||
|
||||
### `plan`
|
||||
|
||||
Purpose:
|
||||
|
||||
- Validate config, load secrets (if configured), prepare workspace layout, and print per-stage run/skip decisions.
|
||||
|
||||
Syntax:
|
||||
|
||||
```bash
|
||||
narratio plan [--config <pipeline.yml>] [--session <session.yml>] [--session-id <id>] [--force]
|
||||
```
|
||||
|
||||
Success output includes:
|
||||
|
||||
- `narratio plan: workdir prepared at <path>`
|
||||
- one line per stage (`<stage>: run|skip`)
|
||||
- `totals: run=<n> skip=<n>`
|
||||
|
||||
Common failure cases:
|
||||
|
||||
- same discovery, template, and validation failures as `run`.
|
||||
- secrets directory read failures when `pipeline.secrets.env_dir` is configured.
|
||||
|
||||
### `resume`
|
||||
|
||||
Purpose:
|
||||
|
||||
- Continue execution from manifest state for the same session.
|
||||
|
||||
Syntax:
|
||||
|
||||
```bash
|
||||
narratio resume [--config <pipeline.yml>] [--session <session.yml>] [--session-id <id>] [--force]
|
||||
```
|
||||
|
||||
Success output:
|
||||
|
||||
- either `narratio resume: session <session_id> has no remaining stages`
|
||||
- or `narratio resume: session <session_id>; executed=<n> skipped=<n>; manifest=<path>`
|
||||
|
||||
Common failure cases:
|
||||
|
||||
- same discovery/template/validation failures as `run`.
|
||||
- manifest load errors when an existing manifest is unreadable.
|
||||
|
||||
### `status`
|
||||
|
||||
Purpose:
|
||||
|
||||
- Inspect an existing manifest file without running stages.
|
||||
|
||||
Syntax:
|
||||
|
||||
```bash
|
||||
narratio status --manifest <manifest.json>
|
||||
```
|
||||
|
||||
Success output includes:
|
||||
|
||||
- `session_id: <id>`
|
||||
- `updated_at: <timestamp>`
|
||||
- `stages:` section with `- <stage>: <status>` entries.
|
||||
|
||||
Common failure cases:
|
||||
|
||||
- missing `--manifest`.
|
||||
- manifest path unreadable or invalid JSON shape.
|
||||
|
||||
### `run-stage`
|
||||
|
||||
Purpose:
|
||||
|
||||
- Execute exactly one stage from the supported stage set.
|
||||
|
||||
Syntax:
|
||||
|
||||
```bash
|
||||
narratio run-stage [--config <pipeline.yml>] [--session <session.yml>] [--session-id <id>] [--force] <stage>
|
||||
```
|
||||
|
||||
Success output:
|
||||
|
||||
- `narratio run-stage: stage=<name> executed=<n> skipped=<n> force=<true|false>; manifest=<path>`
|
||||
|
||||
Common failure cases:
|
||||
|
||||
- missing stage positional argument.
|
||||
- unknown stage name.
|
||||
- same discovery/template/validation failures as `run`.
|
||||
|
||||
## Common Workflows
|
||||
|
||||
Default-discovery run:
|
||||
|
||||
```bash
|
||||
narratio run --session-id 2026-04-04
|
||||
```
|
||||
|
||||
Explicit config/session run:
|
||||
|
||||
```bash
|
||||
narratio run --config /etc/narratio/pipeline.yml --session ./session.yml --session-id 2026-04-04
|
||||
```
|
||||
|
||||
Plan before run:
|
||||
|
||||
```bash
|
||||
narratio plan --config /etc/narratio/pipeline.yml --session ./session.yml --session-id 2026-04-04
|
||||
```
|
||||
|
||||
Resume interrupted work:
|
||||
|
||||
```bash
|
||||
narratio resume --config /etc/narratio/pipeline.yml --session ./session.yml --session-id 2026-04-04
|
||||
```
|
||||
|
||||
Run one stage:
|
||||
|
||||
```bash
|
||||
narratio run-stage --config /etc/narratio/pipeline.yml --session ./session.yml --session-id 2026-04-04 polish
|
||||
```
|
||||
|
||||
## Diagnostic / Recovery Commands
|
||||
|
||||
Read stage status from a manifest:
|
||||
|
||||
```bash
|
||||
narratio status --manifest <manifest.json>
|
||||
```
|
||||
|
||||
How to get manifest path:
|
||||
|
||||
- `run`, `resume`, and `run-stage` success output includes `manifest=<path>`.
|
||||
- use that path with `status` for direct inspection.
|
||||
293
docs/config.md
Normal file
293
docs/config.md
Normal file
@@ -0,0 +1,293 @@
|
||||
# Configuration
|
||||
|
||||
## 1. Overview
|
||||
|
||||
Narratio loads two YAML files:
|
||||
|
||||
- `pipeline.yml`: pipeline-level runtime configuration.
|
||||
- `session.yml`: per-session metadata and input selection.
|
||||
|
||||
These commands load and validate both files before running:
|
||||
|
||||
- `narratio run`
|
||||
- `narratio plan`
|
||||
- `narratio resume`
|
||||
- `narratio run-stage`
|
||||
|
||||
Configuration behavior:
|
||||
|
||||
- strict YAML decode is enabled (`KnownFields(true)`): unknown fields fail.
|
||||
- session templates are rendered before session YAML decode.
|
||||
- defaults are applied for many optional pipeline fields.
|
||||
- validation enforces required fields, value formats, and cross-field constraints.
|
||||
|
||||
## 2. Config file discovery
|
||||
|
||||
Pipeline config lookup for `run`, `plan`, `resume`, and `run-stage`:
|
||||
|
||||
- If `--config <path>` is provided, that explicit path is used.
|
||||
- If `--config` is omitted, Narratio searches in order:
|
||||
1. `/usr/local/etc/narratio/pipeline.yml`
|
||||
2. `/etc/narratio/pipeline.yml`
|
||||
- The first existing file wins.
|
||||
- If none exist, the command fails with a searched-paths error.
|
||||
|
||||
## 3. Session file discovery and templating
|
||||
|
||||
Session config lookup for `run`, `plan`, `resume`, and `run-stage`:
|
||||
|
||||
- If `--session <path>` is provided, that explicit path is used.
|
||||
- If `--session` is omitted, Narratio searches in order:
|
||||
1. `./session.yml`
|
||||
2. `/usr/local/etc/narratio/session.yml`
|
||||
3. `/etc/narratio/session.yml`
|
||||
- The first existing file wins.
|
||||
- If none exist, the command fails and asks you to pass `--session`.
|
||||
|
||||
Session templating:
|
||||
|
||||
- Supported placeholders:
|
||||
- `{{session_id}}`
|
||||
- `{{ session_id }}`
|
||||
- `--session-id <value>` supplies the template value.
|
||||
- Unresolved placeholders fail with a template-rendering error.
|
||||
- If `--session-id` is provided and rendered `session_id` differs, load fails with a mismatch error.
|
||||
- Strict YAML decode still applies after template rendering.
|
||||
|
||||
## 4. Minimal pipeline config
|
||||
|
||||
```yaml
|
||||
workspace:
|
||||
root: ./tmp/narratio-workspace
|
||||
|
||||
whisperx:
|
||||
transcribe_url: "https://transcription.example.com/transcribe"
|
||||
```
|
||||
|
||||
Why this is sufficient:
|
||||
|
||||
- `workspace.root` and `whisperx.transcribe_url` are the core required pipeline fields.
|
||||
- Seriatim and Audita sections may be omitted; defaults are applied.
|
||||
- Archive, storage, spool, normalize, and other optional sections get defaults when omitted.
|
||||
|
||||
## 5. Minimal session template
|
||||
|
||||
```yaml
|
||||
session_id: "{{ session_id }}"
|
||||
campaign: sample-campaign
|
||||
inputs:
|
||||
audio_dir: ./audio
|
||||
speakers_file: ./examples/speakers.yml
|
||||
autocorrect_file: ./examples/autocorrect.yml
|
||||
glossary_file: ./examples/glossary.yml
|
||||
```
|
||||
|
||||
Usage:
|
||||
|
||||
```bash
|
||||
narratio run --config /path/to/pipeline.yml --session ./session.yml --session-id 2026-05-03
|
||||
```
|
||||
|
||||
## 6. Production-oriented config
|
||||
|
||||
```yaml
|
||||
workspace:
|
||||
root: /var/lib/narratio/workspace
|
||||
cleanup_after_archive: true
|
||||
|
||||
storage:
|
||||
backend: s3
|
||||
s3:
|
||||
bucket: my-dnd-archive
|
||||
root_prefix: dnd
|
||||
region: us-east-1
|
||||
access_key_id_env: OBJECT_STORAGE_KEY_ID
|
||||
secret_access_key_env: OBJECT_STORAGE_KEY
|
||||
|
||||
spool:
|
||||
root: /var/spool/narratio
|
||||
delete_audio_after_archive: true
|
||||
|
||||
archive:
|
||||
enabled: true
|
||||
upload_run: true
|
||||
promote_artifacts:
|
||||
- from: transcripts/trimmed.json
|
||||
to: transcripts/trimmed.json
|
||||
required: true
|
||||
- from: artifacts/session_recap.md
|
||||
to: artifacts/session_recap.md
|
||||
required: true
|
||||
|
||||
whisperx:
|
||||
transcribe_url: "https://transcription.example.com/transcribe"
|
||||
```
|
||||
|
||||
Operational notes:
|
||||
|
||||
- `workspace.cleanup_after_archive` controls run-scoped workspace cleanup after successful archive commit.
|
||||
- `spool.delete_audio_after_archive` controls run-scoped spool-audio cleanup after successful archive commit.
|
||||
- S3 archive/session-audio workflows require `storage.s3.bucket`.
|
||||
|
||||
## 7. Full pipeline reference
|
||||
|
||||
Defaults listed here are effective runtime defaults after load.
|
||||
|
||||
| Path | Type | Required | Default | Constraints / Notes |
|
||||
| --- | --- | --- | --- | --- |
|
||||
| `pipeline.workspace.root` | string | Yes | none | Must be non-empty. |
|
||||
| `pipeline.workspace.cleanup_after_archive` | bool | No | `false` | Run-scoped workdir cleanup after successful archive commit. |
|
||||
| `pipeline.secrets.env_dir` | string | Conditional | none | If `pipeline.secrets` is set, `env_dir` must be non-empty. |
|
||||
| `pipeline.storage.backend` | string | No | empty | `s3` enables S3 archive decision path checks. |
|
||||
| `pipeline.storage.bucket` | string | No | empty | Accepted by schema; compatibility field. |
|
||||
| `pipeline.storage.prefix` | string | No | empty | Accepted by schema; compatibility field. |
|
||||
| `pipeline.storage.s3.bucket` | string | Conditional | empty | Required when S3 session audio is used or S3 archive upload is enabled. |
|
||||
| `pipeline.storage.s3.root_prefix` | string | No | `dnd` | Must be relative, non-empty, no `..`. |
|
||||
| `pipeline.storage.s3.region` | string | No | empty | Optional region hint for backend wiring. |
|
||||
| `pipeline.storage.s3.endpoint` | string | No | empty | If provided, must not be all-whitespace. |
|
||||
| `pipeline.storage.s3.force_path_style` | bool | No | `false` | S3-compatible endpoint toggle. |
|
||||
| `pipeline.storage.s3.access_key_id_env` | string | No | `OBJECT_STORAGE_KEY_ID` | Must be a valid env var name. |
|
||||
| `pipeline.storage.s3.secret_access_key_env` | string | No | `OBJECT_STORAGE_KEY` | Must be a valid env var name. |
|
||||
| `pipeline.spool.root` | string | No | `/var/spool/narratio` | Local spool root. |
|
||||
| `pipeline.spool.delete_audio_after_archive` | bool | No | `false` | Cleanup toggle for run-scoped spool audio. |
|
||||
| `pipeline.archive.enabled` | bool | No | `true` | Archive stage enablement. |
|
||||
| `pipeline.archive.upload_run` | bool | No | `true` | Run-record upload toggle when archive enabled. |
|
||||
| `pipeline.archive.promote_artifacts[]` | list | No | two default rules | Defaults: `transcripts/trimmed.json` and `artifacts/session_recap.md`. |
|
||||
| `pipeline.archive.promote_artifacts[].from` | string | Yes (per rule) | none | Must be relative, non-empty, no `..`. |
|
||||
| `pipeline.archive.promote_artifacts[].to` | string | Yes (per rule) | none | Must be relative, non-empty, no `..`. |
|
||||
| `pipeline.archive.promote_artifacts[].required` | bool | No | `true` | Defaults per rule if omitted. |
|
||||
| `pipeline.whisperx.transcribe_url` | string | Yes | none | Must be a valid URL. |
|
||||
| `pipeline.whisperx.language` | string | No | `en` | Passed to WhisperX adapter. |
|
||||
| `pipeline.whisperx.timeout` | duration string | No | `30m` | Must parse as duration. |
|
||||
| `pipeline.whisperx.retries` | int | No | `3` | Must be `>= 0`. |
|
||||
| `pipeline.whisperx.retry_delay` | duration string | No | `2s` | Must parse as duration. |
|
||||
| `pipeline.whisperx.concurrency` | int | No | `2` | Must be `> 0`. |
|
||||
| `pipeline.seriatim.binary` | string | No | `seriatim` | Must be non-empty after defaults. |
|
||||
| `pipeline.seriatim.timeout` | duration string | No | `10m` | Must parse as duration. |
|
||||
| `pipeline.seriatim.output_schema` | string | No | `seriatim-intermediate` | Allowed: `seriatim-minimal`, `seriatim-intermediate`, `seriatim-full`. |
|
||||
| `pipeline.seriatim.coalesce_gap` | float | No | `3.0` | Must be `>= 0`. |
|
||||
| `pipeline.seriatim.report` | bool | No | `true` | Enables report output in Seriatim calls. |
|
||||
| `pipeline.seriatim.env.overlap_word_run_gap` | float | No | unset | If set, must be `> 0`. |
|
||||
| `pipeline.seriatim.env.overlap_word_run_reorder_window` | float | No | unset | If set, must be `> 0`. |
|
||||
| `pipeline.seriatim.env.backchannel_max_duration` | float | No | unset | If set, must be `> 0`. |
|
||||
| `pipeline.seriatim.env.filler_max_duration` | float | No | unset | If set, must be `> 0`. |
|
||||
| `pipeline.audita.binary` | string | No | `audita` | Must be non-empty after defaults. |
|
||||
| `pipeline.audita.timeout` | duration string | No | `3h` | Must parse as duration. |
|
||||
| `pipeline.audita.llm_api_key_env` | string | No | empty | Name of env var to forward to Audita. |
|
||||
| `pipeline.audita.modules[]` | list[string] | No | empty | If set, each must be one of `glossary`, `homophones`, `spoken_word`, `grammar`. |
|
||||
| `pipeline.audita.base_url` | string | No | empty | If non-empty, must be a valid URL. |
|
||||
| `pipeline.audita.model` | string | No | empty | Optional model override passed to Audita. |
|
||||
| `pipeline.audita.total_llm_concurrency` | int | No | unset | If set, must be `> 0`. |
|
||||
| `pipeline.audita.proposal_llm_concurrency` | int | No | unset | If set, must be `> 0`. |
|
||||
| `pipeline.audita.validation_model` | string | No | empty | Optional validation model override. |
|
||||
| `pipeline.audita.validation_llm_concurrency` | int | No | unset | If set, must be `> 0`. |
|
||||
| `pipeline.audita.transcript_description` | string | No | empty | If provided, must not be all-whitespace. |
|
||||
| `pipeline.audita.config_path` | string | No | empty | If provided, must not be all-whitespace. |
|
||||
| `pipeline.audita.output_schema` | string | No | empty | Allowed: empty, `bare-segments`, `audita-v1`. |
|
||||
| `pipeline.audita.work_dir_retention` | string | No | empty | Allowed: empty, `always`, `auto`, `never`. |
|
||||
| `pipeline.audita.report` | bool | No | `true` | Enables Audita report output. |
|
||||
| `pipeline.normalize.output_path` | string | No | `transcripts/normalized.json` | Must be non-empty after defaults. |
|
||||
| `pipeline.normalize.output_schema` | string | No | `seriatim-intermediate` | Allowed: `seriatim-minimal`, `seriatim-intermediate`, `seriatim-full`. |
|
||||
| `pipeline.normalize.report` | bool | No | `true` | Enables normalize report output. |
|
||||
| `pipeline.trim.enabled` | bool | No | `false` | When `false`, trim bounds fields are not required. |
|
||||
| `pipeline.trim.output_path` | string | Conditional | none | Required when `pipeline.trim.enabled=true`. |
|
||||
| `pipeline.trim.bounds.prompt_id` | string | Conditional | none | Required when `pipeline.trim.enabled=true`. |
|
||||
| `pipeline.trim.bounds.profile_id` | string | No | empty | Optional profile override. |
|
||||
| `pipeline.trim.bounds.transcript_input_name` | string | Conditional | none | Required when `pipeline.trim.enabled=true`. |
|
||||
| `pipeline.trim.bounds.output_path` | string | Conditional | none | Required when `pipeline.trim.enabled=true`. |
|
||||
| `pipeline.trim.bounds.timeout` | duration string | No | `10m` | Must parse as duration when set. |
|
||||
| `pipeline.trim.bounds.render_debug` | bool | No | `false` | Enables render-debug output for bounds prompt. |
|
||||
| `pipeline.trim.bounds.render_output_path` | string | Conditional | none | Required when `render_debug=true`. |
|
||||
| `pipeline.trim.seriatim.report` | bool | No | `false` | Trim-stage Seriatim report toggle. |
|
||||
| `pipeline.scriptorium.binary` | string | No | `scriptorium` | Required only when `pipeline.scriptorium` is configured. |
|
||||
| `pipeline.scriptorium.config_path` | string | No | empty | If provided, must not be all-whitespace. |
|
||||
| `pipeline.scriptorium.timeout` | duration string | No | `10m` | Must parse as duration when set. |
|
||||
| `pipeline.scriptorium.render_debug` | bool | No | `false` | Global render-debug toggle for Scriptorium adapter usage. |
|
||||
| `pipeline.scriptorium.artifacts` | map | No | empty | Artifact-generation map keyed by artifact name. |
|
||||
| `pipeline.scriptorium.artifacts.<name>.enabled` | bool | No | `false` | If `true`, `prompt_id` and `output_path` are required. |
|
||||
| `pipeline.scriptorium.artifacts.<name>.render_debug` | bool | No | unset | Per-artifact render-debug override. |
|
||||
| `pipeline.scriptorium.artifacts.<name>.prompt_id` | string | Conditional | none | Required when artifact is enabled. |
|
||||
| `pipeline.scriptorium.artifacts.<name>.profile_id` | string | No | empty | Optional profile override. |
|
||||
| `pipeline.scriptorium.artifacts.<name>.output_path` | string | Conditional | none | Required when artifact is enabled. |
|
||||
| `pipeline.scriptorium.artifacts.<name>.timeout` | duration string | No | empty | If set, must parse as duration. |
|
||||
| `pipeline.scriptorium.artifacts.<name>.inputs.<key>.source` | string | Conditional | none | Required when input is present; allowed values listed below. |
|
||||
| `pipeline.scriptorium.artifacts.<name>.inputs.<key>.artifact` | string | No | empty | Used by `previous_session_artifact` source. |
|
||||
| `pipeline.scriptorium.artifacts.<name>.inputs.<key>.path` | string | No | empty | Optional explicit path metadata. |
|
||||
| `pipeline.scriptorium.artifacts.<name>.inputs.<key>.required` | bool | No | `false` | Input requirement flag for artifact generation. |
|
||||
| `pipeline.scriptorium.artifacts.<name>.vars.<key>` | map value | No | empty | Value must be string or boolean. |
|
||||
| `pipeline.analyzer.binary_path` | string | No | empty | Optional analyzer binary override. |
|
||||
| `pipeline.analyzer.timeout` | duration string | No | empty | If set, must parse as duration. |
|
||||
| `pipeline.analyzer.artifacts.output_dir` | string | No | empty | Optional analyzer output directory hint. |
|
||||
| `pipeline.analyzer.artifacts.types[]` | list[string] | No | empty | Optional analyzer artifact type list. |
|
||||
| `pipeline.notification.backend` | string | No | empty | Optional notifier backend selector. |
|
||||
| `pipeline.notification.recipient` | string | No | empty | Optional notification recipient target. |
|
||||
| `pipeline.notification.timeout` | duration string | No | empty | If set, must parse as duration. |
|
||||
|
||||
Allowed `pipeline.scriptorium.artifacts.<name>.inputs.<key>.source` values:
|
||||
|
||||
- `previous_session_artifact`
|
||||
- `processed_transcript`
|
||||
- `normalized_transcript`
|
||||
- `trimmed_transcript`
|
||||
- `narratio.transcript.merged`
|
||||
- `narratio.transcript.polished`
|
||||
- `narratio.transcript.full`
|
||||
- `narratio.transcript.trimmed`
|
||||
- `narratio.bounds.session`
|
||||
- `narratio.artifact.session_recap`
|
||||
|
||||
## 8. Full session reference
|
||||
|
||||
| Path | Type | Required | Default | Constraints / Notes |
|
||||
| --- | --- | --- | --- | --- |
|
||||
| `session.session_id` | string | Yes | none | Must be non-empty after template rendering. |
|
||||
| `session.campaign` | string | Yes | none | Must be non-empty. Used in local/remote path modeling. |
|
||||
| `session.date` | string | No | empty | Optional session metadata for prompts/artifacts. |
|
||||
| `session.title` | string | No | empty | Optional session metadata for prompts/artifacts. |
|
||||
| `session.inputs.audio_dir` | string | Conditional | empty | One audio-source option. Mutually exclusive with `audio_s3`. |
|
||||
| `session.inputs.audio_files[]` | list[string] | Conditional | empty | One audio-source option. At least one entry can satisfy audio-source requirement. Mutually exclusive with `audio_s3`. |
|
||||
| `session.inputs.audio_s3.prefix` | string | Conditional | none | Required if `audio_s3` object is present; must be relative, non-empty, no `..`. Mutually exclusive with `audio_dir` and `audio_files`. |
|
||||
| `session.inputs.speakers_file` | string | Yes | none | Must be non-empty. |
|
||||
| `session.inputs.autocorrect_file` | string | Yes | none | Must be non-empty. |
|
||||
| `session.inputs.glossary_file` | string | Yes | none | Must be non-empty. |
|
||||
|
||||
Audio-source rule:
|
||||
|
||||
- You must configure exactly one audio source mode:
|
||||
- `audio_dir`, or
|
||||
- `audio_files` (at least one), or
|
||||
- `audio_s3.prefix`
|
||||
- `audio_s3` cannot be combined with `audio_dir` or `audio_files`.
|
||||
|
||||
## 9. Secrets
|
||||
|
||||
Narratio supports filesystem-based secret injection via `pipeline.secrets.env_dir`.
|
||||
|
||||
Behavior:
|
||||
|
||||
- `env_dir` may be absolute or relative.
|
||||
- Relative `env_dir` is resolved from Narratio’s current working directory.
|
||||
- Each top-level file with a valid env-var filename (`[A-Za-z_][A-Za-z0-9_]*`) is loaded.
|
||||
- File contents become env-var values, with trailing `\n` / `\r\n` trimmed.
|
||||
- Existing process environment variables are preserved and not overwritten.
|
||||
- Invalid names and directories inside `env_dir` are skipped.
|
||||
- Missing/unreadable `env_dir` fails command execution.
|
||||
|
||||
Guidance:
|
||||
|
||||
- Store secret values in secret files or pre-set environment variables.
|
||||
- Do not put secret values directly in `pipeline.yml` or `session.yml`.
|
||||
- Use config fields like `llm_api_key_env` and S3 credential env names to reference secret variable names, not secret data.
|
||||
|
||||
## 10. Examples
|
||||
|
||||
Maintained config examples:
|
||||
|
||||
- `docs/examples/pipeline.minimal.yml`
|
||||
- `docs/examples/pipeline.production.yml`
|
||||
- `docs/examples/pipeline.full.annotated.yml`
|
||||
- `docs/examples/session.template.yml`
|
||||
- `docs/examples/session.local-audio.yml`
|
||||
- `docs/examples/session.s3-audio.yml`
|
||||
|
||||
These examples are covered by configuration load/validate tests in `internal/config`.
|
||||
92
docs/development.md
Normal file
92
docs/development.md
Normal file
@@ -0,0 +1,92 @@
|
||||
# Development Guide
|
||||
|
||||
## Purpose
|
||||
Canonical contributor workflow and engineering conventions for implemented Narratio behavior.
|
||||
|
||||
## Repository layout
|
||||
|
||||
- `cmd/narratio/`: CLI entrypoint.
|
||||
- `internal/app/`: command handlers, plan/run/resume orchestration, cleanup gates, secrets loading.
|
||||
- `internal/config/`: strict YAML loading, defaults, and validation.
|
||||
- `internal/stage/`: stage implementations and stage registry/order.
|
||||
- `internal/adapters/`: external boundary adapters (WhisperX, Seriatim, Audita, Scriptorium, storage, notify).
|
||||
- `internal/manifest/`: session/run manifest types and persistence.
|
||||
- `internal/artifacts/`: canonical local/remote path helpers and local artifact store.
|
||||
- `docs/`: canonical documentation set.
|
||||
- `docs/examples/`: maintained config examples used by tests.
|
||||
|
||||
## Build and test commands
|
||||
|
||||
- Run focused CLI behavior checks:
|
||||
|
||||
```bash
|
||||
go test ./internal/app -run TestExecute -v
|
||||
```
|
||||
|
||||
- Run config example load/validate checks:
|
||||
|
||||
```bash
|
||||
go test ./internal/config -run TestExamplesLoadAndValidate -v
|
||||
```
|
||||
|
||||
- Run full test suite:
|
||||
|
||||
```bash
|
||||
go test ./...
|
||||
```
|
||||
|
||||
## Coding conventions
|
||||
|
||||
- Keep orchestration explicit and stage-driven; do not introduce generic workflow/DAG abstractions.
|
||||
- Keep external-system details inside adapter packages; stages should consume Narratio-level contracts only.
|
||||
- Use centralized path helpers from `internal/artifacts` rather than ad hoc path concatenation.
|
||||
- Preserve manifest-driven state transitions (`running`, `succeeded`, `failed`, `skipped`, `stale`) as the source of run progress.
|
||||
- Keep user/operator docs implementation-accurate; planned work belongs only under `docs/roadmap/`.
|
||||
|
||||
For design principles and invariants, see [docs/architecture.md](./architecture.md). For stage/adapter contracts, see [docs/internal/README.md](./internal/README.md).
|
||||
|
||||
## Dependency policy
|
||||
|
||||
- Prefer Go standard library where practical.
|
||||
- Add third-party dependencies only when they provide clear value for required behavior.
|
||||
- Keep dependency additions narrow to the boundary package that needs them.
|
||||
|
||||
## Change playbooks
|
||||
|
||||
### Add config fields
|
||||
|
||||
1. Add fields to config structs in `internal/config`.
|
||||
2. Set defaults in `internal/config/defaults.go` when appropriate.
|
||||
3. Add validation rules in `internal/config/validate.go`.
|
||||
4. Add or update load/validate tests in `internal/config/*_test.go`.
|
||||
5. Update canonical config docs and examples:
|
||||
- [docs/config.md](./config.md)
|
||||
- relevant files under `docs/examples/`
|
||||
|
||||
### Add CLI flags or commands
|
||||
|
||||
1. Update command parsing and behavior in `internal/app`.
|
||||
2. Add or update command tests (`TestExecute` and command-specific tests).
|
||||
3. Update [docs/cli.md](./cli.md) and, if operator workflow changes, [docs/operations.md](./operations.md).
|
||||
|
||||
### Add or modify stages/adapters
|
||||
|
||||
1. Implement stage behavior in `internal/stage` with clear input/output boundaries.
|
||||
2. Keep external transport/subprocess details in `internal/adapters`.
|
||||
3. Preserve manifest and promotion semantics expected by runner and archive logic.
|
||||
4. Add/update stage and adapter tests.
|
||||
5. Update internal component contracts in `docs/internal/`.
|
||||
|
||||
### Update examples
|
||||
|
||||
1. Keep canonical examples only in `docs/examples/`.
|
||||
2. Ensure examples load and validate through runtime config paths.
|
||||
3. Update `internal/config/load_validate_test.go` as needed.
|
||||
4. Update links in `docs/config.md` if example filenames change.
|
||||
|
||||
### Update docs and roadmap
|
||||
|
||||
1. Keep implemented behavior in canonical docs (`README`, `docs/*.md`, `docs/internal/`).
|
||||
2. Keep planned/unimplemented behavior only in `docs/roadmap/`.
|
||||
3. After completing roadmap items, remove or mark them complete in `docs/roadmap/documentation.md`.
|
||||
4. Run a link/path sweep before finalizing changes.
|
||||
@@ -1,130 +0,0 @@
|
||||
# Workspace Architecture Implementation Plan (Status)
|
||||
|
||||
This document tracks the implemented workspace architecture and remaining work for v1.0.
|
||||
|
||||
## Current Architecture (Implemented)
|
||||
|
||||
Narratio now uses a canonical campaign-aware local layout:
|
||||
|
||||
```text
|
||||
{workspace.root}/work/{campaign_id}/{session_id}/
|
||||
manifest.json
|
||||
current/
|
||||
manifest.json
|
||||
run_id.txt
|
||||
inputs/
|
||||
transcripts/
|
||||
artifacts/
|
||||
reports/
|
||||
logs/
|
||||
config/
|
||||
runs/
|
||||
{run_id}/
|
||||
manifest.json
|
||||
{stage}/
|
||||
outputs/
|
||||
logs/
|
||||
reports/
|
||||
config/
|
||||
scratch/
|
||||
```
|
||||
|
||||
Core behavior:
|
||||
|
||||
- Session manifest remains the skip/resume source of truth.
|
||||
- Each invocation creates a run manifest at `runs/{run_id}/manifest.json`.
|
||||
- Stage execution writes run-local artifacts and promotes durable outputs to canonical session paths.
|
||||
- Archive uploads run records under `runs/{run_id}/`, applies promotion rules, then publishes `current/manifest.json` and `current/run_id.txt`.
|
||||
- Analyze input resolution uses centralized artifact IDs with alias support.
|
||||
- Forced upstream reruns mark downstream succeeded stages `stale` so later runs do not skip stale outputs.
|
||||
|
||||
## Section 4 Sequence Status
|
||||
|
||||
### Step 1: Campaign-aware session path model
|
||||
|
||||
Status: complete.
|
||||
|
||||
Implemented:
|
||||
|
||||
- Campaign-aware session and run path helpers.
|
||||
- Campaign-aware artifact-store layout APIs.
|
||||
- Canonical session manifest pathing under `work/{campaign}/{session}`.
|
||||
|
||||
### Step 2: Session manifest + run manifest scaffolding
|
||||
|
||||
Status: complete.
|
||||
|
||||
Implemented:
|
||||
|
||||
- Invocation-scoped run manifest type and store methods.
|
||||
- Runner creates/saves run manifests per invocation.
|
||||
- Session manifest remains authoritative for idempotent stage skipping.
|
||||
|
||||
### Step 3: Run-local stage execution + promotion
|
||||
|
||||
Status: complete.
|
||||
|
||||
Implemented:
|
||||
|
||||
- Run-local stage directory layout under `runs/{run_id}/{stage}`.
|
||||
- Shared helpers for run-local output mapping and promotion to canonical durable paths.
|
||||
- Producer run provenance recorded on durable artifact outputs.
|
||||
|
||||
### Step 4: Archive alignment
|
||||
|
||||
Status: complete.
|
||||
|
||||
Implemented:
|
||||
|
||||
- Canonical run-root/session-root resolution.
|
||||
- Deterministic run-file collection and promotion source resolution.
|
||||
- Current-pointer publication ordering retained (`current/manifest.json` then `current/run_id.txt`).
|
||||
|
||||
### Step 5: Artifact registry/resolver (analyze first consumer)
|
||||
|
||||
Status: complete.
|
||||
|
||||
Implemented:
|
||||
|
||||
- Central artifact resolver with canonical IDs:
|
||||
- `narratio.transcript.merged`
|
||||
- `narratio.transcript.polished`
|
||||
- `narratio.transcript.full`
|
||||
- `narratio.transcript.trimmed`
|
||||
- `narratio.bounds.session`
|
||||
- `narratio.artifact.session_recap`
|
||||
- Backward-compatible aliases:
|
||||
- `processed_transcript`
|
||||
- `normalized_transcript`
|
||||
- `trimmed_transcript`
|
||||
- Analyze stage switched to resolver-based source resolution.
|
||||
|
||||
### Step 6: Minimal downstream invalidation for forced reruns
|
||||
|
||||
Status: complete.
|
||||
|
||||
Implemented:
|
||||
|
||||
- Deterministic downstream invalidation based on canonical stage order.
|
||||
- On forced successful rerun of stage `X`, downstream succeeded stages are marked `stale`.
|
||||
- Resume and non-forced runs naturally re-execute stale stages.
|
||||
|
||||
### Step 7: Legacy layout migration strategy
|
||||
|
||||
Status: intentionally skipped.
|
||||
|
||||
Decision:
|
||||
|
||||
- Automatic migration and legacy fallback compatibility are intentionally not implemented.
|
||||
- The codebase targets canonical-only local layout behavior.
|
||||
- Legacy local workspace state, if present, should be recreated or migrated manually outside Narratio.
|
||||
|
||||
## Remaining Work (v1.0)
|
||||
|
||||
No required workspace/run-history migration steps remain from Section 4.
|
||||
|
||||
Possible future enhancements (non-blocking):
|
||||
|
||||
- Full checksum/input-graph stale detection.
|
||||
- Optional retention-policy expansion for run-history cleanup.
|
||||
- Broader artifact-resolver adoption across additional stage consumers.
|
||||
@@ -1,744 +0,0 @@
|
||||
# Narratio Workspace, Run History, and Artifact Resolution Architecture
|
||||
|
||||
## 1. Purpose
|
||||
|
||||
This document defines the intended v1.0 architecture for Narratio's local workspace layout, run history model, durable session outputs, manifest responsibilities, and artifact resolution contract.
|
||||
|
||||
Narratio is an idempotent session orchestrator. The command:
|
||||
|
||||
```bash
|
||||
narratio run --session-id 2026-05-07
|
||||
```
|
||||
|
||||
means "bring the identified session to its desired completed state." It does **not** mean "always create an entirely new independent output tree and ignore prior session state."
|
||||
|
||||
This distinction drives the architecture:
|
||||
|
||||
* A **session** is the durable domain object and idempotency boundary.
|
||||
* A **run** is an execution attempt that may update the session's durable state.
|
||||
* Durable outputs live at the session level.
|
||||
* Run-specific outputs, logs, generated configs, scratch files, and diagnostics live under `runs/{run_id}/`.
|
||||
* Successful stage outputs are promoted from run-local locations into canonical session-level locations.
|
||||
* The session manifest records current durable state.
|
||||
* Run manifests record execution history and debugging/provenance details.
|
||||
|
||||
This model intentionally mirrors the S3 archive model: session-level current artifacts are distinct from run-record history.
|
||||
|
||||
## 2. Core Concepts
|
||||
|
||||
### 2.1 Session
|
||||
|
||||
A session is the stable unit of work identified by `campaign_id` and `session_id`.
|
||||
|
||||
Examples:
|
||||
|
||||
```text
|
||||
campaign_id = dilfs
|
||||
session_id = 2026-05-07
|
||||
```
|
||||
|
||||
The session directory represents the current durable local state for that session. Re-running Narratio for the same session should consult this state, skip already-completed stages by default, and produce no changes unless work is incomplete, stale, forced, or explicitly selected.
|
||||
|
||||
### 2.2 Run
|
||||
|
||||
A run is a particular execution attempt identified by a generated `run_id`, for example:
|
||||
|
||||
```text
|
||||
20260517T174748Z-abcd1234
|
||||
```
|
||||
|
||||
A run may execute all stages or only a sparse subset of stages. Sparse runs are expected and desirable when the user invokes `--force`, `run-stage`, or a stage-limited command.
|
||||
|
||||
Run directories are provenance/debug records. They should reflect what actually happened during that invocation, not a synthetic complete pipeline layout.
|
||||
|
||||
### 2.3 Durable Output
|
||||
|
||||
A durable output is a canonical session-level artifact intended for later stages, user consumption, archive promotion, or future idempotency decisions.
|
||||
|
||||
Examples:
|
||||
|
||||
```text
|
||||
transcripts/merged.json
|
||||
transcripts/processed.json
|
||||
transcripts/normalized.json
|
||||
transcripts/trimmed.json
|
||||
artifacts/session_recap.md
|
||||
```
|
||||
|
||||
Durable outputs live directly under the session directory, not under a particular run directory.
|
||||
|
||||
### 2.4 Run-Local Output
|
||||
|
||||
A run-local output is the file initially produced by a stage during a specific run. After validation, durable outputs are promoted from run-local paths to session-level canonical paths.
|
||||
|
||||
Run-local outputs, logs, generated configs, reports, and scratch files should remain under:
|
||||
|
||||
```text
|
||||
runs/{run_id}/{stage}/...
|
||||
```
|
||||
|
||||
## 3. Local Workspace Layout
|
||||
|
||||
The canonical local workspace layout is:
|
||||
|
||||
```text
|
||||
{workspace.root}/work/{campaign_id}/{session_id}/
|
||||
manifest.json
|
||||
current/
|
||||
manifest.json
|
||||
run_id.txt
|
||||
inputs/
|
||||
transcripts/
|
||||
artifacts/
|
||||
reports/
|
||||
logs/
|
||||
config/
|
||||
runs/
|
||||
{run_id}/
|
||||
manifest.json
|
||||
prepare/
|
||||
transcribe/
|
||||
merge/
|
||||
polish/
|
||||
normalize/
|
||||
trim/
|
||||
analyze/
|
||||
archive/
|
||||
notify/
|
||||
```
|
||||
|
||||
Not every directory must exist at all times. Directories should be created idempotently when needed.
|
||||
|
||||
### 3.1 Session Root
|
||||
|
||||
The session root is:
|
||||
|
||||
```text
|
||||
{workspace.root}/work/{campaign_id}/{session_id}/
|
||||
```
|
||||
|
||||
The session root is the stable local home for the session. It is the default base for resolving canonical artifact paths.
|
||||
|
||||
The only files that should live directly in the session root are core session-state files, primarily:
|
||||
|
||||
```text
|
||||
manifest.json
|
||||
```
|
||||
|
||||
Lock files may also be session-root scoped if the implementation uses file locks there, but transient locks should not be treated as durable artifacts.
|
||||
|
||||
### 3.2 Session-Level Canonical Directories
|
||||
|
||||
The following directories contain current durable session state:
|
||||
|
||||
```text
|
||||
inputs/
|
||||
transcripts/
|
||||
artifacts/
|
||||
reports/
|
||||
logs/
|
||||
config/
|
||||
current/
|
||||
```
|
||||
|
||||
Recommended meanings:
|
||||
|
||||
| Directory | Purpose |
|
||||
| -------------- | ----------------------------------------------------------------------------- |
|
||||
| `inputs/` | Materialized or copied input files used by the current durable session state. |
|
||||
| `transcripts/` | Canonical transcript tiers. |
|
||||
| `artifacts/` | User-facing and machine-readable generated artifacts. |
|
||||
| `reports/` | Canonical stage reports worth preserving at the session level. |
|
||||
| `logs/` | Optional session-level logs or promoted/latest logs. |
|
||||
| `config/` | Optional session-level generated config snapshots or promoted/latest configs. |
|
||||
| `current/` | Current published session pointers, mirroring the archive backend. |
|
||||
|
||||
Canonical durable outputs should use stable paths under these directories.
|
||||
|
||||
### 3.3 Run History Directory
|
||||
|
||||
Run history lives under:
|
||||
|
||||
```text
|
||||
{workspace.root}/work/{campaign_id}/{session_id}/runs/{run_id}/
|
||||
```
|
||||
|
||||
Each run directory records what happened during that invocation. A run may contain all stage directories or only a sparse subset.
|
||||
|
||||
Example full run:
|
||||
|
||||
```text
|
||||
runs/20260517T174748Z-abcd1234/
|
||||
manifest.json
|
||||
prepare/
|
||||
transcribe/
|
||||
merge/
|
||||
polish/
|
||||
normalize/
|
||||
trim/
|
||||
analyze/
|
||||
archive/
|
||||
notify/
|
||||
```
|
||||
|
||||
Example sparse forced analyze run:
|
||||
|
||||
```text
|
||||
runs/20260518T030000Z-efgh5678/
|
||||
manifest.json
|
||||
analyze/
|
||||
```
|
||||
|
||||
Example sparse polish-through-analyze rerun:
|
||||
|
||||
```text
|
||||
runs/20260518T041500Z-a1b2c3d4/
|
||||
manifest.json
|
||||
polish/
|
||||
normalize/
|
||||
trim/
|
||||
analyze/
|
||||
```
|
||||
|
||||
Run directories should not create stage folders for stages that were not selected, executed, skipped, or otherwise considered during that run unless there is a clear diagnostic reason to do so.
|
||||
|
||||
### 3.4 Stage Run-Local Directories
|
||||
|
||||
Each stage receives a run-local directory:
|
||||
|
||||
```text
|
||||
runs/{run_id}/{stage}/
|
||||
```
|
||||
|
||||
Within that stage directory, the stage may use subdirectories such as:
|
||||
|
||||
```text
|
||||
outputs/
|
||||
logs/
|
||||
reports/
|
||||
config/
|
||||
scratch/
|
||||
```
|
||||
|
||||
For example:
|
||||
|
||||
```text
|
||||
runs/{run_id}/polish/
|
||||
outputs/transcripts/processed.json
|
||||
reports/audita.polish.report.json
|
||||
logs/stdout.log
|
||||
logs/stderr.log
|
||||
config/audita.polish.generated.yml
|
||||
scratch/
|
||||
```
|
||||
|
||||
The exact internal layout of a stage directory may vary by stage, but it should be deterministic, documented, and generated through centralized path helpers rather than ad hoc path joins.
|
||||
|
||||
## 4. Promotion Model
|
||||
|
||||
Narratio uses stage-level promotion with immediate promotion after successful validation.
|
||||
|
||||
The stage lifecycle is:
|
||||
|
||||
1. Resolve required inputs from the current session state and/or run-local context.
|
||||
2. Create the run-local stage directory.
|
||||
3. Execute the stage, writing outputs under `runs/{run_id}/{stage}/...`.
|
||||
4. Validate run-local outputs.
|
||||
5. Promote durable outputs into session-level canonical paths.
|
||||
6. Update the session manifest.
|
||||
7. Update the run manifest.
|
||||
|
||||
Promotion means an atomic or effectively atomic copy/rename from a run-local path to a session-level canonical path.
|
||||
|
||||
Example:
|
||||
|
||||
```text
|
||||
runs/{run_id}/polish/outputs/transcripts/processed.json
|
||||
```
|
||||
|
||||
is promoted to:
|
||||
|
||||
```text
|
||||
transcripts/processed.json
|
||||
```
|
||||
|
||||
Promotion should be safe and deterministic:
|
||||
|
||||
* Validate before promotion.
|
||||
* Write promoted files atomically where possible.
|
||||
* Never leave partially written durable outputs.
|
||||
* Record the producing `run_id` in the session manifest.
|
||||
* Preserve run-local files for debugging unless retention policy deletes them.
|
||||
|
||||
## 5. Promotion Policy: Option A
|
||||
|
||||
Narratio uses immediate stage-level promotion.
|
||||
|
||||
If a selected stage succeeds, its durable outputs are promoted immediately, even if a later selected stage fails.
|
||||
|
||||
Example:
|
||||
|
||||
```bash
|
||||
narratio run --session-id 2026-05-07 --force --stages polish,normalize,trim,analyze
|
||||
```
|
||||
|
||||
If `polish` succeeds and `normalize` fails:
|
||||
|
||||
* `transcripts/processed.json` may be updated from the new run.
|
||||
* `normalize`, `trim`, and `analyze` should not be marked succeeded for the new input state.
|
||||
* Downstream outputs may now be stale relative to the newly promoted polished transcript.
|
||||
|
||||
This policy is simpler, transparent, and consistent with stage-level resumability. It does require explicit stale/invalidation handling.
|
||||
|
||||
## 6. Stale and Invalidation Semantics
|
||||
|
||||
Full checksum-based stale detection may be implemented later. Before that exists, Narratio should still use a simple deterministic invalidation rule for forced or explicit upstream reruns.
|
||||
|
||||
When a stage is successfully re-executed and promoted, downstream stages should be marked stale unless they are also re-executed successfully in the same command invocation.
|
||||
|
||||
Example stage order:
|
||||
|
||||
```text
|
||||
prepare -> transcribe -> merge -> polish -> normalize -> trim -> analyze -> archive -> notify
|
||||
```
|
||||
|
||||
If `polish` is forced and promoted, then the following downstream stages should be invalidated unless rerun successfully:
|
||||
|
||||
```text
|
||||
normalize
|
||||
trim
|
||||
analyze
|
||||
archive
|
||||
notify
|
||||
```
|
||||
|
||||
A stale stage is not equivalent to a failed stage. It means its current durable outputs may no longer correspond to current upstream inputs or configuration.
|
||||
|
||||
Minimum manifest state model:
|
||||
|
||||
```text
|
||||
pending
|
||||
running
|
||||
succeeded
|
||||
failed
|
||||
skipped
|
||||
stale
|
||||
```
|
||||
|
||||
If adding a new `stale` state is too invasive for v1.0, the implementation should at least record stale metadata or clear downstream success markers in a way that prevents accidental idempotent skips based on obsolete outputs.
|
||||
|
||||
## 7. Manifest Responsibilities
|
||||
|
||||
Narratio should distinguish between session manifests and run manifests.
|
||||
|
||||
The same underlying Go types may be reused where practical, but the concepts should remain separate.
|
||||
|
||||
### 7.1 Session Manifest
|
||||
|
||||
Path:
|
||||
|
||||
```text
|
||||
{workspace.root}/work/{campaign_id}/{session_id}/manifest.json
|
||||
```
|
||||
|
||||
The session manifest answers:
|
||||
|
||||
```text
|
||||
What is the current durable state of this session?
|
||||
```
|
||||
|
||||
It should record:
|
||||
|
||||
* campaign ID
|
||||
* session ID
|
||||
* current or latest run ID
|
||||
* current stage states
|
||||
* canonical durable output refs
|
||||
* artifact IDs and paths
|
||||
* producing run ID for each current stage output
|
||||
* relevant input/config checksums when available
|
||||
* stale/invalidated stage information
|
||||
* archive/current publication metadata
|
||||
|
||||
A session's durable state may be a composite of multiple runs.
|
||||
|
||||
For example:
|
||||
|
||||
```text
|
||||
transcripts/merged.json produced by run A
|
||||
transcripts/processed.json produced by run B
|
||||
transcripts/normalized.json produced by run B
|
||||
transcripts/trimmed.json produced by run B
|
||||
artifacts/session_recap.md produced by run C
|
||||
```
|
||||
|
||||
This is valid and expected.
|
||||
|
||||
### 7.2 Run Manifest
|
||||
|
||||
Path:
|
||||
|
||||
```text
|
||||
{workspace.root}/work/{campaign_id}/{session_id}/runs/{run_id}/manifest.json
|
||||
```
|
||||
|
||||
The run manifest answers:
|
||||
|
||||
```text
|
||||
What happened during this specific execution attempt?
|
||||
```
|
||||
|
||||
It should record:
|
||||
|
||||
* run ID
|
||||
* campaign ID
|
||||
* session ID
|
||||
* command mode and selected stages
|
||||
* force flags or stage selection flags
|
||||
* stages considered during this run
|
||||
* stages executed during this run
|
||||
* stages skipped during this run and reasons
|
||||
* run-local output paths
|
||||
* promoted output paths
|
||||
* logs
|
||||
* reports
|
||||
* generated configs
|
||||
* timings
|
||||
* errors
|
||||
* non-secret subprocess invocation metadata
|
||||
|
||||
Run manifests are primarily for debugging, auditability, and archive history.
|
||||
|
||||
## 8. Idempotency and Resume Behavior
|
||||
|
||||
The idempotency boundary is the session, not the run.
|
||||
|
||||
By default:
|
||||
|
||||
```bash
|
||||
narratio run --session-id 2026-05-07
|
||||
```
|
||||
|
||||
should consult the session manifest and skip stages that are already succeeded and not stale.
|
||||
|
||||
If all stages are already complete, the command should execute zero stages and report that the session is already complete.
|
||||
|
||||
Forced execution creates a new run record but updates session-level durable state only for stages that actually succeed and promote outputs.
|
||||
|
||||
Examples:
|
||||
|
||||
```bash
|
||||
narratio run --session-id 2026-05-07 --force
|
||||
```
|
||||
|
||||
Creates a new run and attempts to re-execute the selected/default stage set.
|
||||
|
||||
```bash
|
||||
narratio run-stage --session-id 2026-05-07 analyze --force
|
||||
```
|
||||
|
||||
Creates a sparse run that executes only `analyze`, then promotes updated analysis artifacts if successful.
|
||||
|
||||
```bash
|
||||
narratio resume --session-id 2026-05-07
|
||||
```
|
||||
|
||||
Uses the session manifest to determine what remains incomplete or stale. Resume does not need to resume the same `run_id` unless the implementation explicitly supports resuming an interrupted active run.
|
||||
|
||||
## 9. Artifact Resolution Contract
|
||||
|
||||
Narratio should provide a first-class artifact registry and resolver.
|
||||
|
||||
The resolver maps symbolic artifact source names to canonical session-level paths and manifest output kinds.
|
||||
|
||||
Stages and adapters should not hardcode path fragments when resolving cross-stage inputs. They should ask the artifact resolver for the current durable artifact by ID.
|
||||
|
||||
### 9.1 Canonical Artifact IDs
|
||||
|
||||
Preferred artifact IDs should be namespaced:
|
||||
|
||||
```text
|
||||
narratio.transcript.merged
|
||||
narratio.transcript.polished
|
||||
narratio.transcript.full
|
||||
narratio.transcript.trimmed
|
||||
narratio.bounds.session
|
||||
narratio.artifact.session_recap
|
||||
```
|
||||
|
||||
Recommended initial registry:
|
||||
|
||||
| Artifact ID | Canonical Path | Producer Stage | Output Kind | Meaning |
|
||||
| --------------------------------- | ------------------------------- | -------------- | ------------------------ | ------------------------------------- |
|
||||
| `narratio.transcript.merged` | `transcripts/merged.json` | `merge` | `transcript_merged` | Deterministic Seriatim merge. |
|
||||
| `narratio.transcript.polished` | `transcripts/processed.json` | `polish` | `transcript_processed` | Full Audita-polished transcript. |
|
||||
| `narratio.transcript.full` | `transcripts/normalized.json` | `normalize` | `transcript_normalized` | Preferred full normalized transcript. |
|
||||
| `narratio.transcript.trimmed` | `transcripts/trimmed.json` | `trim` | `transcript_trimmed` | Gameplay-only transcript. |
|
||||
| `narratio.bounds.session` | `artifacts/session_bounds.json` | `trim` | `session_bounds` | Trim bounds selected for the session. |
|
||||
| `narratio.artifact.session_recap` | `artifacts/session_recap.md` | `analyze` | `artifact_session_recap` | Generated session recap. |
|
||||
|
||||
### 9.2 Backward-Compatible Aliases
|
||||
|
||||
Existing source names should remain supported:
|
||||
|
||||
| Legacy Source | Preferred Artifact ID |
|
||||
| ----------------------- | ------------------------------ |
|
||||
| `processed_transcript` | `narratio.transcript.polished` |
|
||||
| `normalized_transcript` | `narratio.transcript.full` |
|
||||
| `trimmed_transcript` | `narratio.transcript.trimmed` |
|
||||
|
||||
These aliases may be supported silently for v1.0. Documentation should prefer namespaced IDs.
|
||||
|
||||
### 9.3 Resolver Behavior
|
||||
|
||||
Artifact resolution should follow this order:
|
||||
|
||||
1. Normalize aliases to canonical artifact IDs.
|
||||
2. Look for a current output reference in the session manifest.
|
||||
3. Fall back to the canonical session-level path.
|
||||
4. If the artifact is required, fail clearly if missing.
|
||||
5. If the artifact is optional and missing, omit it from the downstream invocation.
|
||||
6. Validate the artifact using the expected content validator.
|
||||
7. Return a resolved artifact record containing ID, path, producer stage, output kind, and provenance.
|
||||
|
||||
Example conceptual result:
|
||||
|
||||
```json
|
||||
{
|
||||
"id": "narratio.transcript.trimmed",
|
||||
"path": "/var/lib/narratio/work/dilfs/2026-05-07/transcripts/trimmed.json",
|
||||
"producer_stage": "trim",
|
||||
"producer_run_id": "20260517T174748Z-abcd1234",
|
||||
"output_kind": "transcript_trimmed",
|
||||
"content_type": "application/json"
|
||||
}
|
||||
```
|
||||
|
||||
### 9.4 Artifact Validation
|
||||
|
||||
Transcript artifacts must be valid JSON with a top-level `segments` array.
|
||||
|
||||
Markdown/text artifacts must exist and be non-empty when required.
|
||||
|
||||
Bounds artifacts must match the expected bounds schema and refer to segment IDs in the same transcript ID space used by the trim stage.
|
||||
|
||||
Validation should happen before a resolved artifact is passed to another stage or external subprocess.
|
||||
|
||||
## 10. Analyze Stage Implications
|
||||
|
||||
The analyze stage should consume artifacts through the artifact resolver.
|
||||
|
||||
Preferred Scriptorium config shape:
|
||||
|
||||
```yaml
|
||||
scriptorium:
|
||||
artifacts:
|
||||
session_recap:
|
||||
enabled: true
|
||||
prompt_id: "dnd.session_recap"
|
||||
output_path: "artifacts/session_recap.md"
|
||||
inputs:
|
||||
transcript:
|
||||
source: "narratio.transcript.trimmed"
|
||||
required: true
|
||||
```
|
||||
|
||||
Additional artifacts can choose different transcript tiers:
|
||||
|
||||
```yaml
|
||||
scriptorium:
|
||||
artifacts:
|
||||
table_summary:
|
||||
enabled: true
|
||||
prompt_id: "dnd.table_summary"
|
||||
output_path: "artifacts/table_summary.md"
|
||||
inputs:
|
||||
transcript:
|
||||
source: "narratio.transcript.full"
|
||||
required: true
|
||||
```
|
||||
|
||||
For v1.0, Narratio does not need a generic DAG engine. It may execute configured analyze artifacts in deterministic order and allow later artifacts to consume earlier artifacts only when that relationship is explicit and unambiguous.
|
||||
|
||||
Rules:
|
||||
|
||||
* Artifact inputs resolve from current session-level durable state.
|
||||
* Outputs are first written run-locally.
|
||||
* Successful analyze outputs are promoted to session-level `artifacts/` paths.
|
||||
* Manifest output refs record the producing run ID.
|
||||
* Optional inputs are omitted when unavailable.
|
||||
* Required missing inputs fail before invoking Scriptorium.
|
||||
|
||||
## 11. Archive Alignment
|
||||
|
||||
Local workspace semantics should mirror archive semantics.
|
||||
|
||||
Local session-level durable paths:
|
||||
|
||||
```text
|
||||
work/{campaign}/{session}/transcripts/trimmed.json
|
||||
work/{campaign}/{session}/artifacts/session_recap.md
|
||||
work/{campaign}/{session}/current/manifest.json
|
||||
work/{campaign}/{session}/current/run_id.txt
|
||||
work/{campaign}/{session}/runs/{run_id}/...
|
||||
```
|
||||
|
||||
should map naturally to remote archive paths:
|
||||
|
||||
```text
|
||||
{root_prefix}/campaigns/{campaign}/sessions/{session}/transcripts/trimmed.json
|
||||
{root_prefix}/campaigns/{campaign}/sessions/{session}/artifacts/session_recap.md
|
||||
{root_prefix}/campaigns/{campaign}/sessions/{session}/current/manifest.json
|
||||
{root_prefix}/campaigns/{campaign}/sessions/{session}/current/run_id.txt
|
||||
{root_prefix}/campaigns/{campaign}/sessions/{session}/runs/{run_id}/...
|
||||
```
|
||||
|
||||
The archive stage should publish run records and promoted current artifacts consistently with the local model.
|
||||
|
||||
`current/run_id.txt` remains the effective commit marker for the archived current session state.
|
||||
|
||||
## 12. Path Helper Requirements
|
||||
|
||||
All code should use centralized path helpers for workspace paths.
|
||||
|
||||
Stage code should not manually assemble durable cross-stage paths using raw string joins except through the path model.
|
||||
|
||||
Recommended helper surface:
|
||||
|
||||
```text
|
||||
SessionRoot(campaignID, sessionID)
|
||||
SessionManifestPath(campaignID, sessionID)
|
||||
SessionCurrentDir(campaignID, sessionID)
|
||||
SessionTranscriptsDir(campaignID, sessionID)
|
||||
SessionArtifactsDir(campaignID, sessionID)
|
||||
SessionReportsDir(campaignID, sessionID)
|
||||
SessionLogsDir(campaignID, sessionID)
|
||||
SessionConfigDir(campaignID, sessionID)
|
||||
RunsDir(campaignID, sessionID)
|
||||
RunRoot(campaignID, sessionID, runID)
|
||||
RunManifestPath(campaignID, sessionID, runID)
|
||||
RunStageDir(campaignID, sessionID, runID, stage)
|
||||
RunStageOutputsDir(campaignID, sessionID, runID, stage)
|
||||
RunStageLogsDir(campaignID, sessionID, runID, stage)
|
||||
RunStageReportsDir(campaignID, sessionID, runID, stage)
|
||||
RunStageConfigDir(campaignID, sessionID, runID, stage)
|
||||
CanonicalArtifactPath(campaignID, sessionID, artifactID)
|
||||
```
|
||||
|
||||
Path helpers should enforce safe relative paths for configured output paths:
|
||||
|
||||
* reject absolute paths unless explicitly allowed for a particular config field
|
||||
* reject `..` traversal
|
||||
* normalize separators
|
||||
* preserve deterministic output paths
|
||||
|
||||
## 13. Directory Creation Policy
|
||||
|
||||
Directory creation should be centralized and idempotent.
|
||||
|
||||
Recommended policy:
|
||||
|
||||
* `prepare` ensures the baseline session directory structure exists.
|
||||
* Every stage also calls shared layout helpers to ensure its required run-local directories exist before writing.
|
||||
* `run-stage` should not depend on a prior `prepare` invocation merely to create folders.
|
||||
* Missing directories should be created with appropriate permissions.
|
||||
* Directory creation should not imply stage success.
|
||||
|
||||
This provides consistent layout while keeping direct stage execution robust.
|
||||
|
||||
## 14. Cleanup and Retention
|
||||
|
||||
Cleanup must preserve the distinction between durable session state and run history.
|
||||
|
||||
Workspace cleanup after successful archive may remove selected local directories only according to explicit configuration.
|
||||
|
||||
Potential retention policies:
|
||||
|
||||
```text
|
||||
keep_all_runs
|
||||
keep_failed_runs
|
||||
keep_last_n_runs
|
||||
delete_run_after_success
|
||||
```
|
||||
|
||||
For v1.0, conservative retention is preferred:
|
||||
|
||||
* Do not delete durable session-level outputs unless explicitly requested.
|
||||
* Do not delete failed run directories by default.
|
||||
* If cleanup is enabled, remove only documented run-scoped or spool-scoped paths.
|
||||
* Local development audio inputs must never be deleted by workspace cleanup.
|
||||
|
||||
## 15. Canonical-Only Layout Policy
|
||||
|
||||
Narratio now supports only the canonical campaign-aware layout:
|
||||
|
||||
```text
|
||||
{workspace.root}/work/{campaign_id}/{session_id}/manifest.json
|
||||
{workspace.root}/work/{campaign_id}/{session_id}/runs/{run_id}/...
|
||||
```
|
||||
|
||||
Legacy session-only layout compatibility is intentionally not implemented.
|
||||
|
||||
If legacy workspace data exists, operators should recreate or manually migrate that data outside Narratio before running v1.0 commands.
|
||||
|
||||
## 16. Documentation Updates Required
|
||||
|
||||
The following documentation should be updated to reflect this architecture:
|
||||
|
||||
* `README.md`
|
||||
* `docs/architecture.md`
|
||||
* a dedicated workspace/run-history document, such as this file
|
||||
* S3/archive documentation
|
||||
* analyze/artifact configuration documentation
|
||||
* example pipeline files
|
||||
|
||||
Documentation should consistently use the following terms:
|
||||
|
||||
| Term | Meaning |
|
||||
| ---------------- | ---------------------------------------------------------------------- |
|
||||
| Session | Durable domain object and idempotency boundary. |
|
||||
| Run | Execution attempt that may update session state. |
|
||||
| Durable output | Canonical current session-level output. |
|
||||
| Run-local output | Output produced inside a specific run directory before promotion. |
|
||||
| Promotion | Validated copy/rename from run-local output to durable session output. |
|
||||
| Session manifest | Current durable state of the session. |
|
||||
| Run manifest | Execution record for a particular run. |
|
||||
| Artifact ID | Symbolic source name resolved by the artifact registry. |
|
||||
|
||||
## 17. Architectural Invariants
|
||||
|
||||
The following invariants should hold after implementation:
|
||||
|
||||
1. `session_id` remains the idempotency boundary for normal operator commands.
|
||||
2. `run_id` identifies an execution attempt, not the primary durable workspace.
|
||||
3. Session-level canonical artifacts are the default inputs for downstream stages.
|
||||
4. Run-local outputs are promoted only after validation.
|
||||
5. A session's current durable state may be composed of outputs from multiple runs.
|
||||
6. Sparse run directories are valid and expected.
|
||||
7. The session manifest records current stage/artifact state and producer run IDs.
|
||||
8. The run manifest records what happened during one invocation.
|
||||
9. Artifact consumers resolve symbolic artifact IDs through a registry/resolver.
|
||||
10. Local workspace semantics mirror S3 archive semantics.
|
||||
11. Directory creation is centralized and idempotent.
|
||||
12. Stage code uses path helpers rather than ad hoc path construction.
|
||||
13. Forced upstream reruns invalidate downstream stage success unless downstream stages are rerun successfully.
|
||||
14. Cleanup never removes durable session outputs or local development inputs unless explicitly configured to do so.
|
||||
|
||||
## 18. Implementation Guidance
|
||||
|
||||
A practical implementation sequence is:
|
||||
|
||||
1. Add this architecture document.
|
||||
2. Add or revise path model helpers for session roots, run roots, stage directories, and canonical artifact paths.
|
||||
3. Introduce session manifest versus run manifest concepts.
|
||||
4. Route stage outputs through run-local directories.
|
||||
5. Add promotion helpers with validation and atomic writes.
|
||||
6. Update existing stages to promote durable outputs to session-level canonical paths.
|
||||
7. Add artifact registry and resolver.
|
||||
8. Update analyze to use artifact IDs and aliases.
|
||||
9. Add simple downstream stale invalidation for forced upstream reruns.
|
||||
10. Align archive/local path behavior and documentation.
|
||||
11. Update examples and README.
|
||||
12. Add tests for idempotency, sparse forced runs, promotion, manifest provenance, and artifact resolution.
|
||||
|
||||
This sequence intentionally avoids introducing a generic DAG engine. The v1.0 goal is a clear, deterministic, stage-oriented orchestrator with stable session-level outputs and inspectable run history.
|
||||
346
docs/documentation/policy.md
Normal file
346
docs/documentation/policy.md
Normal file
@@ -0,0 +1,346 @@
|
||||
# Go Project Documentation Policy
|
||||
|
||||
## Purpose
|
||||
|
||||
Project documentation must help four audiences:
|
||||
|
||||
1. users who need to run the application;
|
||||
2. administrators/operators who need to configure and operate it;
|
||||
3. developers who need to understand and change it safely;
|
||||
4. LLM coding agents that need clear scope, boundaries, and invariants.
|
||||
|
||||
Docs should be accurate, concise, task-oriented, and organized by audience. Prefer links to canonical docs over repetition.
|
||||
|
||||
## Core Rules
|
||||
|
||||
### 1. Keep docs concise
|
||||
|
||||
Each document should cover a defined scope and only the essentials for that scope.
|
||||
|
||||
Avoid:
|
||||
- long background explanations;
|
||||
- repeated reference material;
|
||||
- implementation detail in user-facing docs;
|
||||
- aspirational language outside roadmap docs;
|
||||
- verbose examples where one minimal example is clearer.
|
||||
|
||||
### 2. Document only implemented behavior outside roadmap files
|
||||
|
||||
Unimplemented, planned, aspirational, experimental, or future work may be described only under:
|
||||
|
||||
- `docs/roadmap/`
|
||||
|
||||
No other documentation file, including `README.md`, should describe code, features, modules, stages, commands, config fields, or behaviors that do not currently exist.
|
||||
|
||||
If a feature is partial, non-roadmap docs may describe only the implemented portion and its current boundary.
|
||||
|
||||
### 3. Use canonical homes
|
||||
|
||||
Each type of information should have one canonical location.
|
||||
|
||||
Canonical homes:
|
||||
|
||||
- project purpose and quickstart: `README.md`
|
||||
- development principles: `docs/architecture.md`
|
||||
- configuration reference: `docs/config.md`
|
||||
- CLI reference: `docs/cli.md`
|
||||
- operations and recovery: `docs/operations.md`
|
||||
- troubleshooting: `docs/troubleshooting.md`
|
||||
- implemented internals: `docs/internal/`
|
||||
- future work: `docs/roadmap/`
|
||||
- contributor workflow: `docs/development.md`
|
||||
- copyable examples: `examples/`
|
||||
|
||||
Other files should summarize briefly and link to the canonical source.
|
||||
|
||||
### 4. Keep examples real
|
||||
|
||||
Examples should be valid, maintained, and free of secrets.
|
||||
|
||||
Where practical:
|
||||
- example configs should load successfully;
|
||||
- example commands should match real CLI syntax;
|
||||
- important examples should be covered by tests.
|
||||
|
||||
## Documentation Profiles
|
||||
|
||||
All projects require:
|
||||
|
||||
- `README.md`
|
||||
- `docs/architecture.md`
|
||||
|
||||
Additional docs depend on the project.
|
||||
|
||||
### Small library
|
||||
|
||||
Recommended:
|
||||
- `docs/development.md`, if contributor conventions are non-obvious
|
||||
|
||||
### Simple CLI
|
||||
|
||||
Required:
|
||||
- `docs/cli.md`
|
||||
|
||||
Recommended:
|
||||
- `docs/development.md`
|
||||
|
||||
### Config-driven CLI
|
||||
|
||||
Required:
|
||||
- `docs/cli.md`
|
||||
- `docs/config.md`
|
||||
|
||||
Recommended:
|
||||
- `examples/`
|
||||
- `docs/development.md`
|
||||
|
||||
### Stateful or operator-facing application
|
||||
|
||||
Required:
|
||||
- `docs/cli.md`, if CLI-based
|
||||
- `docs/config.md`, if config-driven
|
||||
- `docs/operations.md`
|
||||
|
||||
Recommended:
|
||||
- `docs/troubleshooting.md`
|
||||
- `examples/`
|
||||
- `docs/development.md`
|
||||
|
||||
### Modular, staged, service-oriented, or orchestration application
|
||||
|
||||
Required:
|
||||
- `docs/cli.md`, if CLI-based
|
||||
- `docs/config.md`, if config-driven
|
||||
- `docs/operations.md`
|
||||
- `docs/internal/`
|
||||
- `docs/development.md`
|
||||
|
||||
Recommended:
|
||||
- `docs/troubleshooting.md`
|
||||
- validated examples under `examples/`
|
||||
|
||||
## Required Documents
|
||||
|
||||
### README.md
|
||||
|
||||
**Audience:** users, administrators, operators
|
||||
|
||||
The README is the outward-facing project orientation page.
|
||||
|
||||
It should include, in order:
|
||||
|
||||
1. concise description;
|
||||
2. elevator pitch;
|
||||
3. shortest useful command or usage example;
|
||||
4. links to targeted docs.
|
||||
|
||||
The README should be short. It is not a manual.
|
||||
|
||||
The “shortest useful command” means the simplest command that performs the project’s core use case. (It does not mean `app --help`.)
|
||||
|
||||
### docs/architecture.md
|
||||
|
||||
**Audience:** developers, LLM coding agents
|
||||
|
||||
`docs/architecture.md` is required for every project.
|
||||
|
||||
It is an inward-facing development policy document. It should describe how the project is intended to be built and changed.
|
||||
|
||||
It should include:
|
||||
|
||||
- project shape;
|
||||
- core design principles;
|
||||
- package and boundary philosophy;
|
||||
- state/persistence philosophy, if applicable;
|
||||
- external integration philosophy, if applicable;
|
||||
- error-handling and logging principles;
|
||||
- testing expectations;
|
||||
- documentation expectations;
|
||||
- architectural invariants;
|
||||
- explicit non-goals, if useful.
|
||||
|
||||
For small projects, this file may be brief. It may simply state that the project is intentionally narrow, monolithic, and dependency-light.
|
||||
|
||||
### docs/config.md
|
||||
|
||||
**Audience:** administrators, operators, advanced users
|
||||
|
||||
Required for applications with configuration files.
|
||||
|
||||
It should include, in order:
|
||||
|
||||
1. config file locations and discovery precedence;
|
||||
2. minimal working config;
|
||||
3. production-oriented config;
|
||||
4. full configuration reference;
|
||||
5. secrets handling, if applicable;
|
||||
6. links to maintained examples.
|
||||
|
||||
The full configuration reference should be canonical.
|
||||
|
||||
### docs/cli.md
|
||||
|
||||
**Audience:** users, administrators, operators
|
||||
|
||||
Required for CLI applications.
|
||||
|
||||
It should include, in order:
|
||||
|
||||
1. shortest useful command;
|
||||
2. command overview;
|
||||
3. complete flag reference;
|
||||
4. common workflows;
|
||||
5. diagnostic or recovery commands, if applicable.
|
||||
|
||||
Explain when commands are useful, not just their syntax.
|
||||
|
||||
### docs/operations.md
|
||||
|
||||
**Audience:** administrators, operators
|
||||
|
||||
Required for applications that maintain state, support resume behavior, run multiple stages, write durable artifacts, use remote storage, or require recovery procedures.
|
||||
|
||||
It should cover:
|
||||
|
||||
- normal workflow;
|
||||
- filesystem layout;
|
||||
- remote storage layout, if applicable;
|
||||
- logs and manifests;
|
||||
- resume/retry behavior;
|
||||
- cleanup behavior;
|
||||
- archive/backup behavior;
|
||||
- safe recovery procedures;
|
||||
- operational caveats.
|
||||
|
||||
### docs/troubleshooting.md
|
||||
|
||||
**Audience:** administrators, operators
|
||||
|
||||
Recommended once recurring failure modes exist.
|
||||
|
||||
Each entry should include:
|
||||
|
||||
- symptom;
|
||||
- likely cause;
|
||||
- diagnostic command or inspection step;
|
||||
- safe fix;
|
||||
- relevant links.
|
||||
|
||||
### docs/development.md
|
||||
|
||||
**Audience:** developers, LLM coding agents
|
||||
|
||||
Required for projects maintained by humans and LLM coding agents.
|
||||
|
||||
It should include:
|
||||
|
||||
- repository layout;
|
||||
- build/test commands;
|
||||
- coding conventions;
|
||||
- dependency policy;
|
||||
- how to add config fields;
|
||||
- how to add CLI flags;
|
||||
- how to add stages/modules/adapters, if applicable;
|
||||
- how to update examples;
|
||||
- documentation update expectations.
|
||||
|
||||
### docs/internal/
|
||||
|
||||
**Audience:** developers, LLM coding agents
|
||||
|
||||
Required for modular, staged, service-oriented, or orchestration projects.
|
||||
|
||||
This directory describes implemented internal components. It is not the roadmap.
|
||||
|
||||
Use one file per major component where useful.
|
||||
|
||||
Each component doc should include:
|
||||
|
||||
1. purpose;
|
||||
2. inputs and outputs;
|
||||
3. boundaries;
|
||||
4. config fields used;
|
||||
5. external adapters used;
|
||||
6. state or manifest behavior, if applicable;
|
||||
7. skip/resume behavior, if applicable;
|
||||
8. failure behavior;
|
||||
9. tests to inspect before changing;
|
||||
10. architectural invariants.
|
||||
|
||||
### docs/roadmap/
|
||||
|
||||
**Audience:** maintainers, developers, LLM coding agents
|
||||
|
||||
This is the only place for planned, future, aspirational, experimental, or unimplemented work.
|
||||
|
||||
Roadmap docs should clearly distinguish:
|
||||
|
||||
- proposed work;
|
||||
- accepted plans;
|
||||
- deferred ideas;
|
||||
- rejected ideas;
|
||||
- implementation prompts or task breakdowns, if useful.
|
||||
|
||||
Roadmap docs should not be confused with current behavior.
|
||||
|
||||
## Examples Directory
|
||||
|
||||
Projects with non-trivial configuration or workflows should include `examples/`.
|
||||
|
||||
Useful examples include:
|
||||
|
||||
- minimal working config;
|
||||
- production-oriented config;
|
||||
- full annotated config;
|
||||
- local development config;
|
||||
- remote/object-storage config;
|
||||
- minimal session/input file.
|
||||
|
||||
Examples should be valid, maintained, tested when practical, and linked from relevant docs.
|
||||
|
||||
## Security and Privacy
|
||||
|
||||
Docs and examples must not include:
|
||||
|
||||
- real API keys;
|
||||
- tokens;
|
||||
- passwords;
|
||||
- private keys;
|
||||
- private environment dumps;
|
||||
- sensitive user data;
|
||||
- raw private transcripts;
|
||||
- private infrastructure details unless intentionally public.
|
||||
|
||||
Document secret-handling mechanisms, not actual secret values.
|
||||
|
||||
## Maintenance Rules
|
||||
|
||||
When docs change, verify the affected behavior.
|
||||
|
||||
Where practical:
|
||||
|
||||
- load example config files in tests;
|
||||
- test CLI examples or command parser behavior;
|
||||
- validate documented flags against real flags;
|
||||
- remove stale references;
|
||||
- update links after renames;
|
||||
- keep roadmap content out of non-roadmap docs.
|
||||
|
||||
If documentation and code disagree, fix the documentation and/or open a roadmap item; do not leave aspirational behavior in current-behavior docs.
|
||||
|
||||
Documentation is complete only when it matches the current code.
|
||||
|
||||
## Documentation Change Checklist
|
||||
|
||||
Before merging documentation changes, verify:
|
||||
|
||||
- README is concise and orientation-focused.
|
||||
- `docs/architecture.md` describes development principles.
|
||||
- Future work appears only under `docs/roadmap/`.
|
||||
- User-facing docs avoid unnecessary internals.
|
||||
- Developer-facing docs preserve boundaries and invariants.
|
||||
- Config examples match the schema.
|
||||
- CLI examples match real commands and flags.
|
||||
- Defaults appear in the canonical config reference.
|
||||
- No secrets or private data are included.
|
||||
- Links are accurate.
|
||||
155
docs/examples/pipeline.full.annotated.yml
Normal file
155
docs/examples/pipeline.full.annotated.yml
Normal file
@@ -0,0 +1,155 @@
|
||||
# Full annotated pipeline example for implemented Narratio config fields.
|
||||
# Values are safe placeholders and must be adapted per environment.
|
||||
|
||||
workspace:
|
||||
# Required: local workspace root.
|
||||
root: ./tmp/narratio-workspace
|
||||
# Optional: remove run-scoped workdir after successful archive commit.
|
||||
cleanup_after_archive: false
|
||||
|
||||
# Optional: local secret file loader (directory of ENV_VAR_NAME files).
|
||||
# secrets:
|
||||
# env_dir: ./secrets
|
||||
|
||||
storage:
|
||||
# Optional storage backend selector; use "s3" for archive + S3 audio workflows.
|
||||
backend: s3
|
||||
# Legacy fields retained in schema for compatibility.
|
||||
bucket: ""
|
||||
prefix: ""
|
||||
s3:
|
||||
# Required when using S3 audio or S3 archive uploads.
|
||||
bucket: my-dnd-archive
|
||||
# Optional; defaults to "dnd".
|
||||
root_prefix: dnd
|
||||
# Optional region/endpoint settings.
|
||||
region: us-east-1
|
||||
endpoint: ""
|
||||
force_path_style: false
|
||||
# Optional; defaults shown explicitly.
|
||||
access_key_id_env: OBJECT_STORAGE_KEY_ID
|
||||
secret_access_key_env: OBJECT_STORAGE_KEY
|
||||
|
||||
spool:
|
||||
# Optional; defaults to /var/spool/narratio.
|
||||
root: /var/spool/narratio
|
||||
# Optional cleanup of run-scoped spool audio after successful archive commit.
|
||||
delete_audio_after_archive: false
|
||||
|
||||
archive:
|
||||
# Optional booleans; defaults are true.
|
||||
enabled: true
|
||||
upload_run: true
|
||||
# Optional promotions; defaults shown explicitly.
|
||||
promote_artifacts:
|
||||
- from: transcripts/trimmed.json
|
||||
to: transcripts/trimmed.json
|
||||
required: true
|
||||
- from: artifacts/session_recap.md
|
||||
to: artifacts/session_recap.md
|
||||
required: true
|
||||
|
||||
whisperx:
|
||||
# Required.
|
||||
transcribe_url: "https://transcription.example.com/transcribe"
|
||||
# Optional overrides; defaults shown explicitly.
|
||||
language: en
|
||||
timeout: 30m
|
||||
retries: 3
|
||||
retry_delay: 2s
|
||||
concurrency: 2
|
||||
|
||||
seriatim:
|
||||
# Optional overrides; defaults shown explicitly.
|
||||
binary: seriatim
|
||||
timeout: 10m
|
||||
output_schema: seriatim-intermediate
|
||||
coalesce_gap: 3.0
|
||||
report: true
|
||||
env:
|
||||
# Optional advanced tuning; set only when needed.
|
||||
overlap_word_run_gap: 1.0
|
||||
overlap_word_run_reorder_window: 1.0
|
||||
backchannel_max_duration: 2.0
|
||||
filler_max_duration: 1.25
|
||||
|
||||
audita:
|
||||
# Optional overrides; defaults shown explicitly where applicable.
|
||||
binary: audita
|
||||
timeout: 3h
|
||||
llm_api_key_env: AUDITA_LLM_API_KEY
|
||||
modules: [glossary, homophones, spoken_word, grammar]
|
||||
base_url: ""
|
||||
model: ""
|
||||
total_llm_concurrency: 2
|
||||
proposal_llm_concurrency: 1
|
||||
validation_model: ""
|
||||
validation_llm_concurrency: 1
|
||||
transcript_description: ""
|
||||
config_path: /usr/local/etc/audita/config.yml
|
||||
output_schema: audita-v1
|
||||
work_dir_retention: auto
|
||||
report: true
|
||||
|
||||
normalize:
|
||||
# Optional; defaults shown explicitly.
|
||||
output_path: transcripts/normalized.json
|
||||
output_schema: seriatim-intermediate
|
||||
report: true
|
||||
|
||||
trim:
|
||||
# Keep disabled unless bounds prompt integration is configured.
|
||||
enabled: false
|
||||
output_path: transcripts/trimmed.json
|
||||
bounds:
|
||||
prompt_id: dnd.session_bounds
|
||||
profile_id: local-fast
|
||||
transcript_input_name: transcript
|
||||
output_path: reports/session_bounds.json
|
||||
timeout: 10m
|
||||
render_debug: false
|
||||
render_output_path: reports/session_bounds.render.json
|
||||
seriatim:
|
||||
report: false
|
||||
|
||||
scriptorium:
|
||||
binary: scriptorium
|
||||
config_path: /usr/local/etc/scriptorium/config.yml
|
||||
timeout: 10m
|
||||
render_debug: false
|
||||
artifacts:
|
||||
session_recap:
|
||||
enabled: true
|
||||
prompt_id: dnd.session_recap
|
||||
profile_id: local-fast
|
||||
output_path: artifacts/session_recap.md
|
||||
timeout: 10m
|
||||
inputs:
|
||||
transcript:
|
||||
source: narratio.transcript.trimmed
|
||||
required: true
|
||||
previous_recap:
|
||||
source: previous_session_artifact
|
||||
artifact: session_recap
|
||||
path: ""
|
||||
required: false
|
||||
vars:
|
||||
session_id: true
|
||||
session_date: true
|
||||
campaign_name: true
|
||||
previous_session_id: true
|
||||
output_kind: session_recap
|
||||
|
||||
analyzer:
|
||||
# Optional adapter settings.
|
||||
binary_path: ""
|
||||
timeout: 2m
|
||||
artifacts:
|
||||
output_dir: ""
|
||||
types: []
|
||||
|
||||
notification:
|
||||
# Optional notification settings.
|
||||
backend: ""
|
||||
recipient: ""
|
||||
timeout: 30s
|
||||
5
docs/examples/pipeline.minimal.yml
Normal file
5
docs/examples/pipeline.minimal.yml
Normal file
@@ -0,0 +1,5 @@
|
||||
workspace:
|
||||
root: ./tmp/narratio-workspace
|
||||
|
||||
whisperx:
|
||||
transcribe_url: "https://transcription.example.com/transcribe"
|
||||
95
docs/examples/pipeline.production.yml
Normal file
95
docs/examples/pipeline.production.yml
Normal file
@@ -0,0 +1,95 @@
|
||||
workspace:
|
||||
root: /var/lib/narratio/workspace
|
||||
cleanup_after_archive: true
|
||||
|
||||
storage:
|
||||
backend: s3
|
||||
s3:
|
||||
bucket: my-dnd-archive
|
||||
root_prefix: dnd
|
||||
region: us-east-1
|
||||
access_key_id_env: OBJECT_STORAGE_KEY_ID
|
||||
secret_access_key_env: OBJECT_STORAGE_KEY
|
||||
|
||||
spool:
|
||||
root: /var/spool/narratio
|
||||
delete_audio_after_archive: true
|
||||
|
||||
archive:
|
||||
enabled: true
|
||||
upload_run: true
|
||||
promote_artifacts:
|
||||
- from: transcripts/trimmed.json
|
||||
to: transcripts/trimmed.json
|
||||
required: true
|
||||
- from: artifacts/session_recap.md
|
||||
to: artifacts/session_recap.md
|
||||
required: true
|
||||
|
||||
whisperx:
|
||||
transcribe_url: "https://transcription.example.com/transcribe"
|
||||
language: en
|
||||
timeout: 45m
|
||||
retries: 3
|
||||
retry_delay: 3s
|
||||
concurrency: 2
|
||||
|
||||
seriatim:
|
||||
binary: seriatim
|
||||
timeout: 10m
|
||||
output_schema: seriatim-intermediate
|
||||
coalesce_gap: 3.0
|
||||
report: true
|
||||
|
||||
audita:
|
||||
binary: audita
|
||||
timeout: 3h
|
||||
llm_api_key_env: AUDITA_LLM_API_KEY
|
||||
modules: [glossary, homophones, spoken_word, grammar]
|
||||
output_schema: audita-v1
|
||||
work_dir_retention: auto
|
||||
total_llm_concurrency: 2
|
||||
proposal_llm_concurrency: 1
|
||||
validation_llm_concurrency: 1
|
||||
report: true
|
||||
|
||||
normalize:
|
||||
output_path: transcripts/normalized.json
|
||||
output_schema: seriatim-intermediate
|
||||
report: true
|
||||
|
||||
trim:
|
||||
enabled: false
|
||||
|
||||
scriptorium:
|
||||
binary: scriptorium
|
||||
config_path: /usr/local/etc/scriptorium/config.yml
|
||||
timeout: 10m
|
||||
render_debug: false
|
||||
artifacts:
|
||||
session_recap:
|
||||
enabled: true
|
||||
prompt_id: dnd.session_recap
|
||||
profile_id: local-fast
|
||||
output_path: artifacts/session_recap.md
|
||||
timeout: 10m
|
||||
inputs:
|
||||
transcript:
|
||||
source: narratio.transcript.trimmed
|
||||
required: true
|
||||
previous_recap:
|
||||
source: previous_session_artifact
|
||||
artifact: session_recap
|
||||
required: false
|
||||
vars:
|
||||
session_id: true
|
||||
session_date: true
|
||||
campaign_name: true
|
||||
previous_session_id: true
|
||||
output_kind: session_recap
|
||||
|
||||
analyzer:
|
||||
timeout: 2m
|
||||
|
||||
notification:
|
||||
timeout: 30s
|
||||
9
docs/examples/session.local-audio.yml
Normal file
9
docs/examples/session.local-audio.yml
Normal file
@@ -0,0 +1,9 @@
|
||||
session_id: 2026-05-03
|
||||
campaign: sample-campaign
|
||||
date: 2026-05-03
|
||||
title: Sample Session
|
||||
inputs:
|
||||
audio_dir: ./audio
|
||||
speakers_file: ./examples/speakers.yml
|
||||
autocorrect_file: ./examples/autocorrect.yml
|
||||
glossary_file: ./examples/glossary.yml
|
||||
10
docs/examples/session.s3-audio.yml
Normal file
10
docs/examples/session.s3-audio.yml
Normal file
@@ -0,0 +1,10 @@
|
||||
session_id: 2026-05-03
|
||||
campaign: sample-campaign
|
||||
date: 2026-05-03
|
||||
title: Sample Session
|
||||
inputs:
|
||||
audio_s3:
|
||||
prefix: audio/
|
||||
speakers_file: ./examples/speakers.yml
|
||||
autocorrect_file: ./examples/autocorrect.yml
|
||||
glossary_file: ./examples/glossary.yml
|
||||
7
docs/examples/session.template.yml
Normal file
7
docs/examples/session.template.yml
Normal file
@@ -0,0 +1,7 @@
|
||||
session_id: "{{ session_id }}"
|
||||
campaign: sample-campaign
|
||||
inputs:
|
||||
audio_dir: ./audio
|
||||
speakers_file: ./examples/speakers.yml
|
||||
autocorrect_file: ./examples/autocorrect.yml
|
||||
glossary_file: ./examples/glossary.yml
|
||||
@@ -1,96 +0,0 @@
|
||||
# Audita Subprocess Operations
|
||||
|
||||
This document describes how parent processes should invoke `audita process` safely in production orchestration.
|
||||
|
||||
## Recommended command form
|
||||
|
||||
Use explicit file outputs for orchestrated runs:
|
||||
|
||||
```sh
|
||||
audita process <transcript.json> \
|
||||
--transcript-description "Brief context that may help resolve ambiguous terms." \
|
||||
--glossary <glossary.yaml> \
|
||||
--output <output-transcript.json> \
|
||||
--report-json <report.json>
|
||||
```
|
||||
|
||||
Additional flags that may be situationally appropriate:
|
||||
- `--config <path>` to select an explicit versioned config file.
|
||||
- `--output-schema <bare-segments|audita-v1>` to select transcript output shape.
|
||||
- `--work-dir <dir>` to control diagnostics location.
|
||||
- `--work-dir-retention <always|auto|never>` to control retained run directories.
|
||||
- `--total-llm-concurrency`, `--proposal-llm-concurrency`, and `--validation-llm-concurrency` when orchestration needs to set explicit LLM throughput controls.
|
||||
- `--modules ...` only when intentionally overriding the default sequence.
|
||||
|
||||
For config-driven orchestration, validate config files in CI/preflight:
|
||||
|
||||
```sh
|
||||
audita config validate --config <path>
|
||||
```
|
||||
|
||||
## Stdout behavior
|
||||
|
||||
- With `--output`: stdout is expected to be empty on success.
|
||||
- Without `--output`: stdout contains transcript JSON only on success.
|
||||
- Report JSON is never written to stdout.
|
||||
|
||||
## Stderr behavior
|
||||
|
||||
- Success path should be quiet or minimal human-readable logs.
|
||||
- Failure path writes concise human-readable errors.
|
||||
- When a diagnostics run directory exists, failure stderr includes its path.
|
||||
- Prompt/response diagnostic payloads are not streamed to stderr.
|
||||
|
||||
## Output file behavior
|
||||
|
||||
- `--output` writes transcript JSON in the selected output schema to the provided path.
|
||||
- Output write failures return nonzero and surface actionable errors.
|
||||
- The command does not silently ignore output write errors.
|
||||
|
||||
## Report JSON behavior
|
||||
|
||||
- `--report-json` writes a machine-readable process report to the requested path.
|
||||
- Run-directory `report.json` is written independently under diagnostics.
|
||||
- Best-effort failure reports are emitted when possible without masking the primary failure.
|
||||
- Report write failures return nonzero with clear stderr messaging.
|
||||
- Report diagnostics metadata references run-directory artifacts including utilization diagnostics and correction ledger paths when available.
|
||||
|
||||
## Diagnostics directory behavior
|
||||
|
||||
- Each run creates (when possible) a per-run diagnostics directory.
|
||||
- Typical artifacts include transcript, normalization, chunking, invocation, effective config, LLM diagnostics, `utilization-diagnostics.json`, `correction-ledger.json`, `report.json`, and `error.log` on failure.
|
||||
- Failed runs retain diagnostics.
|
||||
- Under `auto` retention, successful runs with skipped/rejected corrections are retained; clean successful runs may be removed.
|
||||
|
||||
## Exit codes
|
||||
|
||||
- `0`: success.
|
||||
- Nonzero: failure (input/schema/config/module/LLM/runtime/output/report/diagnostics errors).
|
||||
|
||||
Treat any nonzero as a failed subprocess invocation.
|
||||
|
||||
## Timeout and cancellation
|
||||
|
||||
- Runtime operations propagate context cancellation and request timeouts through LLM/scheduler paths.
|
||||
- On cancellation or timeout, the process exits nonzero and should not hang.
|
||||
- If diagnostics were initialized before failure, failure artifacts remain available for debugging.
|
||||
|
||||
## Secret redaction expectations
|
||||
|
||||
API keys and configured secret values are redacted from:
|
||||
- reports (`--report-json` and run-dir `report.json`);
|
||||
- diagnostics artifacts (including effective config and LLM interaction artifacts);
|
||||
- surfaced adapter/runtime errors;
|
||||
- test fixtures and regression outputs.
|
||||
|
||||
Parent-process logs should still avoid printing raw environment variables.
|
||||
|
||||
## Parent-process pipe guidance
|
||||
|
||||
To avoid deadlocks in orchestrators:
|
||||
- always read both stdout and stderr concurrently when invoking as a subprocess;
|
||||
- prefer file outputs (`--output`, `--report-json`) for machine workflows;
|
||||
- treat stderr as human-readable diagnostics, not structured data;
|
||||
- parse structured results from output/report files.
|
||||
|
||||
For Go callers, prefer `exec.CommandContext` with explicit timeout/cancellation and buffered/streamed readers for both pipes.
|
||||
@@ -1,339 +0,0 @@
|
||||
# Narratio -> Scriptorium CLI Integration
|
||||
|
||||
## 1. Purpose
|
||||
|
||||
This document defines how Narratio should invoke Scriptorium through the **public CLI**.
|
||||
|
||||
This is a **subprocess integration contract**, not an internal Go API contract.
|
||||
|
||||
## 2. Assumptions
|
||||
|
||||
- `scriptorium` is installed and available on `PATH`.
|
||||
- Scriptorium is configured with `config.yml`.
|
||||
- `config.yml` provides `prompt_dir`, `profile_dir`, and `schema_dir` as needed.
|
||||
- Prompt and profile libraries are already deployed for the environment.
|
||||
- Narratio provides prepared artifact files (for example polished transcript, glossary, previous recap, campaign notes).
|
||||
- Initial integration is synchronous subprocess execution.
|
||||
- Narratio remains the orchestrator.
|
||||
|
||||
In normal operation, Narratio does not need to pass `--prompt-dir` and `--profile-dir` if they are supplied by Scriptorium config.
|
||||
|
||||
Narratio may pass `--config <PATH>` when it must use a non-default Scriptorium config file.
|
||||
|
||||
## 3. Core Commands Narratio May Call
|
||||
|
||||
Primary commands for subprocess integration:
|
||||
|
||||
- `scriptorium run`
|
||||
- `scriptorium render`
|
||||
|
||||
For production generation, use `scriptorium run`.
|
||||
|
||||
`scriptorium render` is for debugging, dry-runs, test assertions, and validating command construction without LLM execution.
|
||||
|
||||
Note: `scriptorium serve` and HTTP API exist, but they are not the initial integration path.
|
||||
|
||||
## 4. Command Selection Guidance
|
||||
|
||||
- Use `run` to generate an output artifact.
|
||||
- Use `render` to inspect the prepared prompt and effective settings without calling the LLM.
|
||||
- Use `render --format json` when Narratio/tests need structured prepare output.
|
||||
|
||||
## 5. Recommended `run` Invocation Shape
|
||||
|
||||
Production shape:
|
||||
|
||||
```bash
|
||||
scriptorium run \
|
||||
--prompt <prompt_id> \
|
||||
--input transcript=<processed-transcript-path> \
|
||||
--out <output-artifact-path>
|
||||
```
|
||||
|
||||
Common optional additions:
|
||||
|
||||
- `--config <path>`: use a specific Scriptorium config file.
|
||||
- `--profile <profile_id>`: override prompt default profile.
|
||||
- `--var name=value` (repeatable): small metadata values.
|
||||
- `--input name=path` (repeatable): additional named artifacts.
|
||||
- `--timeout <duration>`: per-run timeout override.
|
||||
- Runtime model override flags (`--llm-base-url`, `--model`, etc.) only for exceptional/operator-directed cases.
|
||||
|
||||
## 6. Recommended `render` Invocation Shape
|
||||
|
||||
Human-readable debug shape:
|
||||
|
||||
```bash
|
||||
scriptorium render \
|
||||
--prompt <prompt_id> \
|
||||
--input transcript=<processed-transcript-path> \
|
||||
--format text
|
||||
```
|
||||
|
||||
Structured debug/test shape:
|
||||
|
||||
```bash
|
||||
scriptorium render \
|
||||
--prompt <prompt_id> \
|
||||
--input transcript=<processed-transcript-path> \
|
||||
--format json \
|
||||
--out <render-debug-path>
|
||||
```
|
||||
|
||||
`render` does **not** call the LLM, does **not** validate model output, and does **not** perform repair.
|
||||
|
||||
## 7. Inputs
|
||||
|
||||
- Pass inputs as repeated `--input name=path` flags.
|
||||
- `name` must match the Prompt Definition input name.
|
||||
- Prefer absolute paths, or paths relative to a working directory controlled by Narratio.
|
||||
- Pass Audita output as the primary transcript input.
|
||||
- Additional inputs may include glossary, previous recap, campaign notes, event logs, final state maps, or other prompt-specific artifacts.
|
||||
- Scriptorium reads input files directly; Narratio does not need to inline file content for CLI use.
|
||||
|
||||
## 8. Variables
|
||||
|
||||
Use repeated `--var name=value` for small metadata values.
|
||||
|
||||
Typical examples:
|
||||
|
||||
- `session_date`
|
||||
- `session_id`
|
||||
- `campaign_name`
|
||||
- `previous_session_id`
|
||||
- `output_kind`
|
||||
|
||||
Large content belongs in input files, not `--var` values.
|
||||
|
||||
## 9. Prompt IDs and Output Artifact Types
|
||||
|
||||
Narratio should treat prompt IDs as configuration, not hardcoded business logic.
|
||||
|
||||
Narratio config may map stage/output names to prompt IDs, for example:
|
||||
|
||||
- session recap prompt
|
||||
- structured event extraction prompt
|
||||
- glossary suggestion prompt
|
||||
- player-facing summary prompt
|
||||
|
||||
Prompt IDs used by Narratio should come from the deployed Scriptorium prompt library.
|
||||
|
||||
## 10. Profiles
|
||||
|
||||
- Prompts may declare `default_profile`.
|
||||
- Narratio may omit `--profile` to use prompt default profile.
|
||||
- Narratio may pass `--profile` to force profile selection.
|
||||
- This enables environment/profile selection like `local-fast`, `local-quality`, `frontier`, `batch`, or test profiles.
|
||||
- Profile names should generally be Narratio configuration values.
|
||||
|
||||
## 11. Runtime Overrides
|
||||
|
||||
Supported runtime override flags:
|
||||
|
||||
- `--llm-base-url`
|
||||
- `--model`
|
||||
- `--api-key-env`
|
||||
- `--temperature`
|
||||
- `--max-tokens`
|
||||
- `--top-p`
|
||||
- `--timeout`
|
||||
|
||||
Guidance:
|
||||
|
||||
- Keep normal model/runtime settings in Execution Profiles.
|
||||
- Use runtime overrides only for explicit per-run exceptions, tests, or operator overrides.
|
||||
- Never pass raw API keys on the command line.
|
||||
- `--api-key-env` names an environment variable; Narratio must ensure that variable is set in subprocess environment.
|
||||
|
||||
## 12. Config Behavior
|
||||
|
||||
- Default config path: `/etc/scriptorium/config.yml`.
|
||||
- `--config <PATH>` overrides default path.
|
||||
- Missing default config is allowed by Scriptorium.
|
||||
- If `--config` is provided explicitly, the file must exist and be valid.
|
||||
- CLI flags override `config.yml`.
|
||||
- `config.yml` overrides built-in application defaults.
|
||||
|
||||
Narratio can either:
|
||||
|
||||
- rely on system default config path, or
|
||||
- carry an explicit config path and pass `--config`.
|
||||
|
||||
## 13. Environment Handling
|
||||
|
||||
Subprocess environment recommendations:
|
||||
|
||||
- Pass through required API-key environment variables referenced by `api_key_env`.
|
||||
- Do not pass raw API keys as CLI arguments.
|
||||
- Avoid logging full environment dumps.
|
||||
- Capture stdout and stderr separately.
|
||||
- Use a controlled working directory.
|
||||
- Prefer absolute artifact paths.
|
||||
|
||||
## 14. Output Handling
|
||||
|
||||
For `scriptorium run`:
|
||||
|
||||
- Use `--out` when Narratio needs durable artifact files.
|
||||
- Without `--out`, artifact content is written to stdout.
|
||||
- Preferred orchestration pattern: always use `--out`, then treat the file as stage output artifact.
|
||||
- Capture stderr for diagnostics.
|
||||
|
||||
For `scriptorium render`:
|
||||
|
||||
- Use `--out` to store render diagnostics.
|
||||
- Use `--format json` when tests need to inspect selected profile, effective runtime settings, input hashes, prompt hash, and rendered messages.
|
||||
|
||||
## 15. Exit Status and Errors
|
||||
|
||||
Current CLI behavior (verified from implementation/tests):
|
||||
|
||||
- `0`: success.
|
||||
- `1`: runtime/parse/config/load/render/generation/IO error.
|
||||
- `2`: run completed but output validation failed (`ValidationFailed`).
|
||||
|
||||
Additional details:
|
||||
|
||||
- On `run`, output artifact write happens before exit code selection. If validation fails, artifact may still be written and exit code is `2`.
|
||||
- `stderr` carries both errors and normal run summary output; non-empty stderr alone does not imply failure.
|
||||
- `render` returns `0` on success and `1` on failures.
|
||||
|
||||
Narratio should treat non-zero exit codes as failed stage execution, but may record generated artifact paths if a run exited `2` and output file exists.
|
||||
|
||||
## 16. Recommended Narratio Integration Pattern
|
||||
|
||||
1. Build CLI args from Narratio stage configuration.
|
||||
2. Use subprocess context cancellation/timeout.
|
||||
3. Pass absolute input paths.
|
||||
4. Pass `--out` to a session-scoped artifact path.
|
||||
5. Add `--var` metadata values.
|
||||
6. Optionally add `--config`.
|
||||
7. Optionally add `--profile`.
|
||||
8. Ensure required API-key env vars are present.
|
||||
9. Run subprocess synchronously.
|
||||
10. Capture stdout/stderr separately.
|
||||
11. On success, store output artifact path and invocation metadata in stage artifacts.
|
||||
12. On failure, store exit code and stderr diagnostics in stage status.
|
||||
|
||||
## 17. Suggested Narratio Configuration Shape
|
||||
|
||||
Illustrative `pipeline.yml` shape:
|
||||
|
||||
```yaml
|
||||
scriptorium:
|
||||
binary: scriptorium
|
||||
config_path: /etc/scriptorium/config.yml
|
||||
timeout: 10m
|
||||
render_debug: false
|
||||
artifacts:
|
||||
session_recap:
|
||||
enabled: true
|
||||
prompt_id: dnd.session_recap
|
||||
profile_id: local-quality # optional
|
||||
output_path: artifacts/session_recap.md
|
||||
timeout: 10m
|
||||
render_debug: false # optional artifact override
|
||||
inputs:
|
||||
transcript:
|
||||
source: trimmed_transcript
|
||||
required: true
|
||||
previous_recap:
|
||||
source: previous_session_artifact
|
||||
artifact: session_recap
|
||||
path: "" # optional
|
||||
required: false
|
||||
vars:
|
||||
session_id: true
|
||||
session_date: true
|
||||
campaign_name: true
|
||||
previous_session_id: true
|
||||
output_kind: session_recap
|
||||
```
|
||||
|
||||
The key idea: map Narratio artifact names to prompt ID, optional profile, expected inputs, vars, and output destination.
|
||||
|
||||
## 18. Testing Strategy for Narratio Integration
|
||||
|
||||
- Use `scriptorium render --format json` to verify command construction without LLM calls.
|
||||
- Use dedicated test prompt/profile libraries for integration tests.
|
||||
- Use small fixture transcripts.
|
||||
- Verify missing-input failure behavior.
|
||||
- Verify prompt `default_profile` behavior.
|
||||
- Verify explicit `--profile` override behavior.
|
||||
- Verify `--config` behavior (default and explicit).
|
||||
- Verify output file creation when `--out` is used.
|
||||
- Verify stderr capture on failures.
|
||||
- Avoid real API keys in tests.
|
||||
|
||||
## 19. Security and Privacy Notes
|
||||
|
||||
- Never pass raw API keys on command line.
|
||||
- Do not log full rendered prompts by default; transcripts may contain sensitive content.
|
||||
- Avoid logging prompt content unless explicit debug mode is enabled.
|
||||
- Treat generated artifacts as potentially sensitive.
|
||||
- Use session-scoped, access-controlled output paths.
|
||||
- `api_key_env` names should come from environment management, not embedded secrets.
|
||||
|
||||
## 20. Initial D&D Artifact Generation Examples
|
||||
|
||||
These are examples only. Use prompt IDs from the deployed prompt library.
|
||||
|
||||
Session recap:
|
||||
|
||||
```bash
|
||||
scriptorium run \
|
||||
--prompt dnd.session_recap \
|
||||
--input transcript=/work/campaign-7/session-42/transcript.polished.md \
|
||||
--input glossary=/work/campaign-7/session-42/glossary.yml \
|
||||
--out /work/campaign-7/session-42/artifacts/session_recap.md
|
||||
```
|
||||
|
||||
Structured events:
|
||||
|
||||
```bash
|
||||
scriptorium run \
|
||||
--prompt dnd.structured_events \
|
||||
--input transcript=/work/campaign-7/session-42/transcript.polished.md \
|
||||
--out /work/campaign-7/session-42/artifacts/structured_events.json
|
||||
```
|
||||
|
||||
Glossary suggestions:
|
||||
|
||||
```bash
|
||||
scriptorium run \
|
||||
--prompt dnd.glossary_suggestions \
|
||||
--input transcript=/work/campaign-7/session-42/transcript.polished.md \
|
||||
--input previous_recap=/work/campaign-7/session-41/artifacts/session_recap.md \
|
||||
--out /work/campaign-7/session-42/artifacts/glossary_suggestions.md
|
||||
```
|
||||
|
||||
Player-facing summary:
|
||||
|
||||
```bash
|
||||
scriptorium run \
|
||||
--prompt dnd.player_summary \
|
||||
--input transcript=/work/campaign-7/session-42/transcript.polished.md \
|
||||
--input structured_events=/work/campaign-7/session-42/artifacts/structured_events.json \
|
||||
--out /work/campaign-7/session-42/artifacts/player_summary.md
|
||||
```
|
||||
|
||||
## 21. Non-Goals
|
||||
|
||||
Initial Narratio integration should not:
|
||||
|
||||
- call Scriptorium internal Go packages
|
||||
- use HTTP API as the primary path
|
||||
- expect Scriptorium to read S3 refs directly
|
||||
- make Scriptorium responsible for Narratio stage state
|
||||
- make Scriptorium responsible for notification
|
||||
- require Scriptorium to understand D&D workflow semantics beyond prompt definitions
|
||||
|
||||
## 22. Future Extension Notes
|
||||
|
||||
Possible later extensions:
|
||||
|
||||
- HTTP API integration
|
||||
- S3 artifact references if Scriptorium adds S3 reader support
|
||||
- richer render diagnostics and policy controls
|
||||
- token budgeting/prompt-size checks
|
||||
- batch execution if Scriptorium later adds batch support
|
||||
@@ -1,403 +0,0 @@
|
||||
# seriatim
|
||||
|
||||
`seriatim` merges per-speaker WhisperX-style JSON transcripts into a single JSON transcript that preserves speaker identity and chronological order.
|
||||
|
||||
The current implementation supports the `merge` command. It reads one or more input JSON files, optionally maps each input file to a canonical speaker using `speakers.yml`, sorts all segments by timestamp, detects and resolves overlaps when word-level timing is available, assigns consecutive numeric `id` values, and writes a merged JSON artifact.
|
||||
|
||||
## Usage
|
||||
|
||||
Run from source:
|
||||
|
||||
```sh
|
||||
go run ./cmd/seriatim merge \
|
||||
--input-file samples/raw/2026-04-19-Eric_Rakestraw.json \
|
||||
--input-file samples/raw/2026-04-19-Mike_Brown.json \
|
||||
--output-file merged.json
|
||||
```
|
||||
|
||||
Optional report output:
|
||||
|
||||
```sh
|
||||
go run ./cmd/seriatim merge \
|
||||
--input-file eric.json \
|
||||
--input-file mike.json \
|
||||
--output-file merged.json \
|
||||
--report-file report.json
|
||||
```
|
||||
|
||||
## CLI
|
||||
|
||||
```text
|
||||
seriatim merge [flags]
|
||||
```
|
||||
|
||||
Global flags:
|
||||
|
||||
| Flag | Description |
|
||||
| --- | --- |
|
||||
| `--help` | Show command help. |
|
||||
| `--version` | Show application version. Local builds default to `dev`; release builds inject the release version. |
|
||||
|
||||
`merge` flags:
|
||||
|
||||
| Flag | Required | Default | Description |
|
||||
| --- | --- | --- | --- |
|
||||
| `--input-file` | Yes | none | Input transcript JSON file. Repeat once per speaker/input file. |
|
||||
| `--output-file` | Yes | none | Merged transcript JSON output path. |
|
||||
| `--report-file` | No | none | Optional report JSON output path. |
|
||||
| `--speakers` | No | none | Speaker map YAML file. When omitted, input file basenames are used as speaker labels. |
|
||||
| `--autocorrect` | No | none | Autocorrect rules YAML file. When omitted, the default `autocorrect` module leaves text unchanged. |
|
||||
| `--input-reader` | No | `json-files` | Input reader module. |
|
||||
| `--output-modules` | No | `json` | Comma-separated output modules. |
|
||||
| `--output-schema` | No | `seriatim-intermediate` | JSON output contract. Allowed values are `seriatim-minimal`, `seriatim-intermediate`, and `seriatim-full`. If omitted, the runtime default is used; consumers that depend on a specific shape should set this explicitly. |
|
||||
| `--preprocessing-modules` | No | `validate-raw,normalize-speakers,trim-text` | Comma-separated preprocessing modules, evaluated in order. |
|
||||
| `--postprocessing-modules` | No | `detect-overlaps,resolve-overlaps,backchannel,filler,resolve-danglers,coalesce,detect-overlaps,autocorrect,assign-ids,validate-output` | Comma-separated postprocessing modules, evaluated in order. |
|
||||
| `--coalesce-gap` | No | `3.0` | Maximum same-speaker gap in seconds for `coalesce`; also used as the `resolve-overlaps` context window. Must be a non-negative float. |
|
||||
|
||||
Environment variables:
|
||||
|
||||
| Environment Variable | Default | Description |
|
||||
| --- | --- | --- |
|
||||
| `SERIATIM_OUTPUT_SCHEMA` | `seriatim-intermediate` | Output schema used when `--output-schema` is not explicitly provided. Allowed values are `seriatim-minimal`, `seriatim-intermediate`, and `seriatim-full`. The CLI flag takes precedence. |
|
||||
| `SERIATIM_OVERLAP_WORD_RUN_GAP` | `1.0` | Maximum gap in seconds between adjacent timed words when `resolve-overlaps` builds word-run replacement segments. Must be a positive float. |
|
||||
| `SERIATIM_OVERLAP_WORD_RUN_REORDER_WINDOW` | `1.0` | Near-start window in seconds for ordering replacement word runs shortest-first. Must be a positive float. |
|
||||
| `SERIATIM_BACKCHANNEL_MAX_DURATION` | `2.0` | Maximum duration in seconds for `backchannel` classification. Must be a positive float. |
|
||||
| `SERIATIM_FILLER_MAX_DURATION` | `1.25` | Maximum duration in seconds for `filler` classification. Must be a positive float. |
|
||||
|
||||
## Input JSON Format
|
||||
|
||||
Each input file must be valid JSON with a top-level `segments` array. The current parser accepts the WhisperX segment subset needed for merging:
|
||||
|
||||
```json
|
||||
{
|
||||
"segments": [
|
||||
{
|
||||
"start": 1.25,
|
||||
"end": 3.5,
|
||||
"text": "Hello there.",
|
||||
"words": [
|
||||
{"word": "Hello", "start": 1.25, "end": 1.55, "score": 0.98},
|
||||
{"word": "there.", "start": 1.7, "end": 2.0}
|
||||
]
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
Required segment fields:
|
||||
|
||||
- `start`: number, must be `>= 0`.
|
||||
- `end`: number, must be `>= start`.
|
||||
- `text`: string.
|
||||
|
||||
Optional word fields:
|
||||
|
||||
- `words`: array of word timing objects.
|
||||
- `words[].word`: string.
|
||||
- `words[].start`: optional number, must be `>= 0` when present.
|
||||
- `words[].end`: optional number, must be `>= start` when present with `start`.
|
||||
- `words[].score`: optional number.
|
||||
- `words[].speaker`: optional raw speaker label string.
|
||||
|
||||
Word-level timing is preserved internally for overlap resolution. If a word is missing `start` or `end`, seriatim keeps the word text, emits a warning in the optional report, and does not use that word as a timing anchor. Word timing is not emitted in the final JSON artifact.
|
||||
|
||||
## Speaker Map Format
|
||||
|
||||
`speakers.yml` maps input files to canonical speaker names using ordered substring rules:
|
||||
|
||||
This file is optional. If `--speakers` is omitted, `seriatim` uses each input file basename as the segment speaker label.
|
||||
|
||||
```yaml
|
||||
match:
|
||||
- speaker: "Eric Rakestraw"
|
||||
match:
|
||||
- "Eric_Rakestraw"
|
||||
- "Eric"
|
||||
|
||||
- speaker: "Mike Brown"
|
||||
match:
|
||||
- "Mike_Brown"
|
||||
- "mb"
|
||||
```
|
||||
|
||||
For each `--input-file`, `seriatim` takes the file basename and evaluates the rules in order. The first rule with a matching substring wins, and no later rules are evaluated.
|
||||
|
||||
For example, this input:
|
||||
|
||||
```text
|
||||
samples/raw/2026-04-19-Eric_Rakestraw.json
|
||||
```
|
||||
|
||||
matches this rule because the basename contains `Eric_Rakestraw`:
|
||||
|
||||
```yaml
|
||||
- speaker: "Eric Rakestraw"
|
||||
match:
|
||||
- "Eric_Rakestraw"
|
||||
```
|
||||
|
||||
Important details:
|
||||
|
||||
- Matching is against the input file basename, not the full path.
|
||||
- Matching is case-insensitive.
|
||||
- Rules are evaluated from first to last.
|
||||
- Each rule must have a non-empty `speaker`.
|
||||
- Each rule must have at least one non-empty `match` string.
|
||||
- Duplicate speaker names are invalid.
|
||||
- Every input file must match at least one rule or the command fails.
|
||||
|
||||
Deprecated old format:
|
||||
|
||||
```yaml
|
||||
inputs:
|
||||
eric.json:
|
||||
speaker: "Eric Rakestraw"
|
||||
```
|
||||
|
||||
The old `inputs:` direct mapping format is no longer supported.
|
||||
|
||||
## Output JSON Format
|
||||
|
||||
`--output-modules json` controls the writer. `--output-schema` controls the JSON contract that writer serializes.
|
||||
|
||||
The named schemas are stable public contracts. If a consumer depends on a specific shape, it should request that schema explicitly at runtime. The runtime default selection may change in a future release.
|
||||
|
||||
The `seriatim-intermediate` schema is the current default selection when neither `--output-schema` nor `SERIATIM_OUTPUT_SCHEMA` is set. It stays close to the minimal schema, but adds optional `categories` on each segment:
|
||||
|
||||
```json
|
||||
{
|
||||
"metadata": {
|
||||
"application": "seriatim",
|
||||
"version": "dev",
|
||||
"output_schema": "seriatim-intermediate"
|
||||
},
|
||||
"segments": [
|
||||
{
|
||||
"id": 1,
|
||||
"start": 1.25,
|
||||
"end": 3.5,
|
||||
"speaker": "Eric Rakestraw",
|
||||
"text": "Hello there.",
|
||||
"categories": ["backchannel"]
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
The `seriatim-full` schema uses the full seriatim envelope:
|
||||
|
||||
```json
|
||||
{
|
||||
"metadata": {
|
||||
"application": "seriatim",
|
||||
"version": "dev",
|
||||
"input_reader": "json-files",
|
||||
"input_files": ["eric.json", "mike.json"],
|
||||
"preprocessing_modules": ["validate-raw", "normalize-speakers", "trim-text"],
|
||||
"postprocessing_modules": ["detect-overlaps", "resolve-overlaps", "backchannel", "filler", "resolve-danglers", "coalesce", "detect-overlaps", "autocorrect", "assign-ids", "validate-output"],
|
||||
"output_modules": ["json"]
|
||||
},
|
||||
"segments": [
|
||||
{
|
||||
"id": 1,
|
||||
"source": "eric.json",
|
||||
"source_segment_index": 0,
|
||||
"speaker": "Eric Rakestraw",
|
||||
"start": 1.25,
|
||||
"end": 3.5,
|
||||
"text": "Hello there.",
|
||||
"overlap_group_id": 1
|
||||
},
|
||||
{
|
||||
"id": 2,
|
||||
"source": "eric.json",
|
||||
"source_ref": "word-run:1:1:1",
|
||||
"derived_from": ["eric.json#0"],
|
||||
"speaker": "Eric Rakestraw",
|
||||
"start": 2.0,
|
||||
"end": 2.5,
|
||||
"text": "Resolved word run",
|
||||
"categories": ["backchannel"]
|
||||
}
|
||||
],
|
||||
"overlap_groups": [
|
||||
{
|
||||
"id": 1,
|
||||
"start": 1.25,
|
||||
"end": 4.0,
|
||||
"segments": ["eric.json#0", "mike.json#0"],
|
||||
"speakers": ["Eric Rakestraw", "Mike Brown"],
|
||||
"class": "unknown",
|
||||
"resolution": "unresolved"
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
The `seriatim-minimal` schema emits minimal metadata and compact ordered segments:
|
||||
|
||||
```json
|
||||
{
|
||||
"metadata": {
|
||||
"application": "seriatim",
|
||||
"version": "dev",
|
||||
"output_schema": "seriatim-minimal"
|
||||
},
|
||||
"segments": [
|
||||
{
|
||||
"id": 1,
|
||||
"start": 1.25,
|
||||
"end": 3.5,
|
||||
"speaker": "Eric Rakestraw",
|
||||
"text": "Hello there."
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
Minimal output intentionally omits categories, overlap groups, source/provenance fields, and pipeline configuration metadata.
|
||||
|
||||
Intermediate output intentionally omits overlap groups and source/provenance fields, but keeps optional `categories` and minimal metadata.
|
||||
|
||||
Segments are sorted deterministically by:
|
||||
|
||||
```text
|
||||
(start, end, source, source_segment_index/source_ref, speaker)
|
||||
```
|
||||
|
||||
Final segment IDs are assigned after sorting and start at `1`.
|
||||
|
||||
The public Go output contract is available from:
|
||||
|
||||
```go
|
||||
import "gitea.maximumdirect.net/eric/seriatim/schema"
|
||||
```
|
||||
|
||||
The same package embeds machine-readable JSON Schemas in `schema/full-output.schema.json`, `schema/intermediate-output.schema.json`, and `schema/minimal-output.schema.json`. The default `validate-output` postprocessor validates the selected output shape and verifies final segment IDs are present, sequential, and start at `1`.
|
||||
|
||||
## Overlap Detection
|
||||
|
||||
The default postprocessing pipeline detects overlapping segment groups.
|
||||
|
||||
Overlap behavior:
|
||||
|
||||
- A strict timing overlap is required: `next.start < current_group_end`.
|
||||
- Segments that only touch at a boundary are not grouped.
|
||||
- Groups require at least two distinct speakers.
|
||||
- Transitive overlaps are grouped together.
|
||||
- Segments in detected groups receive `overlap_group_id`.
|
||||
- `overlap_groups[].segments` contains stable references in `source#source_segment_index` format.
|
||||
- `class` is currently `unknown`.
|
||||
- `resolution` is `unresolved` until `resolve-overlaps` replaces the group.
|
||||
|
||||
## Overlap Resolution
|
||||
|
||||
The default postprocessing pipeline runs `detect-overlaps`, then `resolve-overlaps`, then `backchannel`, then `filler`, then `resolve-danglers`, then `coalesce`, then a second `detect-overlaps` pass.
|
||||
|
||||
For each detected overlap group, `resolve-overlaps` uses preserved WhisperX word timing to build smaller word-run replacement segments:
|
||||
|
||||
- The resolution window expands the detected overlap group by `--coalesce-gap` seconds on both sides.
|
||||
- Nearby same-speaker context segments are included when they intersect the expanded window and their start or end is within `--coalesce-gap` of the original overlap boundary.
|
||||
- Once a segment is selected for replacement, all timed words from that segment participate in word-run construction; the window controls segment selection, not per-word clipping.
|
||||
- Context segments that are part of another detected overlap group are not pulled into the current group.
|
||||
- Untimed words are included in replacement text in original word order when nearby timed words create a replacement run.
|
||||
- Untimed words do not affect replacement segment start/end times or word-run gap splitting.
|
||||
- Words for the same speaker are merged into one run when the gap between adjacent words is no greater than `SERIATIM_OVERLAP_WORD_RUN_GAP`.
|
||||
- The default word-run gap is `1.0` seconds.
|
||||
- Set `SERIATIM_OVERLAP_WORD_RUN_GAP` to a positive number of seconds to override the default.
|
||||
- Near-start replacement word runs are reordered so shorter segments come first when adjacent starts are within `SERIATIM_OVERLAP_WORD_RUN_REORDER_WINDOW`.
|
||||
- The default word-run reorder window is `1.0` seconds.
|
||||
- Set `SERIATIM_OVERLAP_WORD_RUN_REORDER_WINDOW` to a positive number of seconds to override the default.
|
||||
- Replacement segment text is built by joining word text with single spaces.
|
||||
- Replacement segments include `source_ref` and `derived_from`.
|
||||
- Replacement segments omit `source_segment_index` because they are derived from one or more original segments.
|
||||
- Resolved overlap groups are removed before the second detection pass.
|
||||
- Replacement segments are left without `overlap_group_id` until the second detection pass annotates any remaining overlap.
|
||||
- If a speaker has no usable word timing in a group, that speaker's original segment is kept.
|
||||
- If no speakers in a group have usable word timing, the original group and annotations remain unchanged.
|
||||
|
||||
## Backchannels
|
||||
|
||||
The default pipeline runs `backchannel` before `coalesce`. It tags short acknowledgement segments with:
|
||||
|
||||
```json
|
||||
"categories": ["backchannel"]
|
||||
```
|
||||
|
||||
Backchannel matching is case-insensitive, ignores punctuation for matching and word-count purposes, trims surrounding whitespace, and requires a matching acknowledgement phrase, no more than three whitespace-delimited words, and duration no greater than `SERIATIM_BACKCHANNEL_MAX_DURATION` seconds. The default maximum duration is `2.0` seconds.
|
||||
|
||||
## Fillers
|
||||
|
||||
The default pipeline runs `filler` after `backchannel` and before `coalesce`. It tags short filler utterances with:
|
||||
|
||||
```json
|
||||
"categories": ["filler"]
|
||||
```
|
||||
|
||||
Filler matching is case-insensitive, ignores punctuation for matching and word-count purposes, trims surrounding whitespace, and requires only filler tokens such as `um`, `uh`, `er`, `erm`, `ah`, `eh`, `hmm`, `mm`, or repeated combinations of those tokens. Matching segments must contain no more than three whitespace-delimited words and have duration no greater than `SERIATIM_FILLER_MAX_DURATION` seconds. The default maximum duration is `1.25` seconds.
|
||||
|
||||
## Dangler Resolution
|
||||
|
||||
The default pipeline runs `resolve-danglers` before `coalesce` and before the second overlap detection pass. It repairs short derived fragments when they share provenance with a nearby segment:
|
||||
|
||||
- Dangling-end fragments have no more than two words and end in punctuation.
|
||||
- Dangling-start fragments have no more than two words.
|
||||
- Matching uses same-speaker segments with any shared `derived_from` value.
|
||||
- Merged segments use `source_ref` values such as `resolve-danglers:1`, keep the target segment's transcript position, and union `derived_from`.
|
||||
|
||||
## Coalescing
|
||||
|
||||
The default pipeline runs `coalesce` after `resolve-danglers` and before the second overlap detection pass. It merges adjacent same-speaker segments in the transcript's current order when `next.start - current.end <= --coalesce-gap`.
|
||||
|
||||
Coalesced segments use `source_ref` values such as `coalesce:1`, include `derived_from`, and omit `source_segment_index`.
|
||||
|
||||
Different-speaker backchannel and filler segments do not block coalescing of surrounding same-speaker segments. Same-speaker backchannel and filler segments are merged normally when they are within `--coalesce-gap`. When same-speaker segments are coalesced, any `backchannel` or `filler` category from the merged inputs is dropped from the coalesced segment.
|
||||
|
||||
## Autocorrect
|
||||
|
||||
Autocorrect is included in the default postprocessing pipeline. If `--autocorrect` is omitted, the module leaves transcript text unchanged and records a skip event in the optional report.
|
||||
|
||||
Enable corrections by passing `--autocorrect`:
|
||||
|
||||
```sh
|
||||
go run ./cmd/seriatim merge \
|
||||
--input-file input.json \
|
||||
--autocorrect autocorrect.yml \
|
||||
--output-file merged.json
|
||||
```
|
||||
|
||||
`autocorrect.yml` format:
|
||||
|
||||
```yaml
|
||||
autocorrect:
|
||||
- target: "Hrank"
|
||||
match:
|
||||
- "hrank"
|
||||
- "Frank"
|
||||
|
||||
- target: "Mike Brown"
|
||||
match:
|
||||
- "Mike Pat"
|
||||
```
|
||||
|
||||
Matching behavior:
|
||||
|
||||
- Matching is case-sensitive.
|
||||
- Matches apply only to whole tokens, not substrings inside larger words.
|
||||
- Punctuation and whitespace can surround a match.
|
||||
- Multi-word and hyphenated matches are supported.
|
||||
- Duplicate match strings are invalid, including duplicates across separate rules.
|
||||
|
||||
## Current Limitations
|
||||
|
||||
- Only JSON input is supported.
|
||||
- Overlap resolution depends on WhisperX word timing; groups without usable word timing remain unresolved.
|
||||
- Alternate output formats are not implemented yet.
|
||||
|
||||
## Release Builds
|
||||
|
||||
Local builds record version metadata as `dev`. Release builds should inject the release version with `ldflags`:
|
||||
|
||||
```sh
|
||||
go build -ldflags "-X gitea.maximumdirect.net/eric/seriatim/internal/buildinfo.Version=v1.0.0" ./cmd/seriatim
|
||||
```
|
||||
24
docs/internal/README.md
Normal file
24
docs/internal/README.md
Normal file
@@ -0,0 +1,24 @@
|
||||
# Internal Documentation Index
|
||||
|
||||
## Audience
|
||||
Developers and LLM coding agents changing Narratio internals.
|
||||
|
||||
## Scope
|
||||
Implementation-accurate contracts for workspace/state, stages, and external adapter boundaries.
|
||||
|
||||
## Component Docs
|
||||
- `workspace.md`: local state model, manifests, run-local layout, promotion, and cleanup invariants.
|
||||
- `stage-prepare.md`: input materialization and provenance capture.
|
||||
- `stage-transcribe.md`: WhisperX transcript generation.
|
||||
- `stage-merge.md`: Seriatim normalization + merge.
|
||||
- `stage-polish.md`: Audita transcript polishing.
|
||||
- `stage-normalize.md`: post-polish normalization.
|
||||
- `stage-trim.md`: bounds-driven transcript trimming.
|
||||
- `stage-analyze.md`: Scriptorium session recap generation.
|
||||
- `stage-archive.md`: archive upload and current-pointer publish contract.
|
||||
- `integration-audita.md`: Audita adapter invocation/validation contract.
|
||||
- `integration-seriatim.md`: Seriatim merge/normalize/trim adapter contract.
|
||||
- `integration-scriptorium.md`: Scriptorium run/render adapter contract.
|
||||
|
||||
## Canonical Owner
|
||||
`docs/internal/` is the canonical home for implemented internals per `docs/documentation/policy.md`.
|
||||
66
docs/internal/integration-audita.md
Normal file
66
docs/internal/integration-audita.md
Normal file
@@ -0,0 +1,66 @@
|
||||
# Integration: audita
|
||||
|
||||
## Purpose
|
||||
Define Narratio's adapter contract for transcript polishing via Audita CLI subprocess execution.
|
||||
|
||||
## Inputs and Outputs
|
||||
Inputs (`audita.PolishRequest`):
|
||||
- merged transcript path
|
||||
- glossary path
|
||||
- output processed transcript path
|
||||
- optional report path (required when report enabled)
|
||||
- work dir
|
||||
- generated config path
|
||||
- stdout/stderr log paths
|
||||
- optional module/model/base URL and concurrency knobs
|
||||
|
||||
Outputs (`audita.PolishResult`):
|
||||
- processed transcript path
|
||||
- optional report path
|
||||
- generated config path
|
||||
- stdout/stderr log paths
|
||||
- exit code, duration, invoked binary
|
||||
- adapter metadata
|
||||
|
||||
## Boundaries
|
||||
Owns:
|
||||
- Deterministic CLI argument construction for `audita process`
|
||||
- Environment bridging for API credentials
|
||||
- Invocation config emission
|
||||
- Output validation for processed transcript and report
|
||||
|
||||
Does not own:
|
||||
- Upstream/downstream stage orchestration
|
||||
- Credential sourcing policy beyond required env-var presence check
|
||||
|
||||
## Config Fields Used
|
||||
Via `pipeline.audita.*` mapped in app/stage wiring:
|
||||
- `binary`, `timeout`, `llm_api_key_env`, `modules`, `base_url`, `model`
|
||||
- `transcript_description`, `config_path`, `output_schema`, `work_dir_retention`
|
||||
- `total_llm_concurrency`, `proposal_llm_concurrency`, `validation_model`, `validation_llm_concurrency`, `report`
|
||||
|
||||
## External Adapters Used
|
||||
- Shared subprocess helper (`internal/adapters/subprocess`) to run CLI and capture logs.
|
||||
|
||||
## State and Manifest Behavior
|
||||
- No direct manifest writes.
|
||||
- Stage-level metadata records adapter provenance and credential-present signal.
|
||||
- Generated invocation YAML is written when `GeneratedConfigPath` is provided.
|
||||
|
||||
## Skip and Resume Behavior
|
||||
- Adapter has no skip/resume logic. Stage/runner controls this.
|
||||
|
||||
## Failure Behavior
|
||||
- Constructor validation fails on invalid binary/timeout/schema/concurrency/URL values.
|
||||
- Run fails on missing required paths, missing required credential env var, subprocess errors, invalid processed JSON shape, or invalid report JSON.
|
||||
- Failures preserve stdout/stderr paths in returned result metadata.
|
||||
|
||||
## Tests to Inspect Before Changing
|
||||
- `internal/adapters/audita/subprocess_test.go`
|
||||
- `internal/adapters/audita/fake_test.go`
|
||||
- `internal/stage/polish_test.go`
|
||||
|
||||
## Architectural Invariants
|
||||
- Processed output must be valid JSON with top-level `segments` array.
|
||||
- When report is enabled, report output must be valid JSON.
|
||||
- If `llm_api_key_env` is configured, credential must be present in environment.
|
||||
64
docs/internal/integration-scriptorium.md
Normal file
64
docs/internal/integration-scriptorium.md
Normal file
@@ -0,0 +1,64 @@
|
||||
# Integration: scriptorium
|
||||
|
||||
## Purpose
|
||||
Define Narratio's adapter contract for Scriptorium artifact generation and render-debug subprocess invocations.
|
||||
|
||||
## Inputs and Outputs
|
||||
Inputs:
|
||||
- `RunArtifactRequest`: binary, config path, prompt/profile IDs, input map, vars map, timeout, output path, logs/config paths, optional API env and working dir
|
||||
- `RenderArtifactRequest`: same core fields for render mode
|
||||
|
||||
Outputs (`ArtifactResult`):
|
||||
- output path
|
||||
- stdout/stderr log paths
|
||||
- generated config path
|
||||
- exit code and duration
|
||||
- command mode (`run` or `render`)
|
||||
- prompt/profile provenance
|
||||
- validation failure signal
|
||||
- adapter metadata
|
||||
|
||||
## Boundaries
|
||||
Owns:
|
||||
- Deterministic CLI arg construction for `scriptorium run` and `scriptorium render`
|
||||
- Common request validation
|
||||
- Invocation config emission
|
||||
- Output existence/non-empty checks
|
||||
- Validation-failure mapping for run exit code 2
|
||||
|
||||
Does not own:
|
||||
- Artifact selection policy (`analyze` stage)
|
||||
- Bounds semantic validation (`trim` stage)
|
||||
|
||||
## Config Fields Used
|
||||
Via `pipeline.scriptorium.*` and stage-level artifact config:
|
||||
- `binary`, `config_path`, `timeout`, `render_debug`
|
||||
- artifact-level `prompt_id`, `profile_id`, `timeout`, `inputs`, `vars`, `output_path`
|
||||
|
||||
## External Adapters Used
|
||||
- Shared subprocess helper (`internal/adapters/subprocess`).
|
||||
|
||||
## State and Manifest Behavior
|
||||
- No direct manifest writes.
|
||||
- Stage metadata records adapter outputs and command mode.
|
||||
- Generated invocation YAML is written when requested.
|
||||
|
||||
## Skip and Resume Behavior
|
||||
- Adapter has no skip/resume logic. Stage/runner controls execution.
|
||||
|
||||
## Failure Behavior
|
||||
- Request validation fails for missing binary/prompt/output, invalid timeout, invalid input/var names, or missing required API env var.
|
||||
- Subprocess errors bubble with command context.
|
||||
- `run` exit code 2 is treated as `ValidationFailed=true` and surfaced as error by calling stage.
|
||||
- Successful subprocess still fails if output file is missing/empty.
|
||||
|
||||
## Tests to Inspect Before Changing
|
||||
- `internal/adapters/scriptorium/subprocess_test.go`
|
||||
- `internal/adapters/scriptorium/fake_test.go`
|
||||
- `internal/stage/analyze_test.go`
|
||||
- `internal/stage/trim_test.go`
|
||||
|
||||
## Architectural Invariants
|
||||
- Both modes require explicit timeout > 0.
|
||||
- Input/var maps are sorted into deterministic CLI argument order.
|
||||
- Run-mode validation failures are represented explicitly, not silently skipped.
|
||||
60
docs/internal/integration-seriatim.md
Normal file
60
docs/internal/integration-seriatim.md
Normal file
@@ -0,0 +1,60 @@
|
||||
# Integration: seriatim
|
||||
|
||||
## Purpose
|
||||
Define Narratio's adapter contract for merge, normalize, and trim subprocess invocations of Seriatim.
|
||||
|
||||
## Inputs and Outputs
|
||||
Inputs:
|
||||
- `MergeRequest`: raw/normalized transcript inputs, output path, optional report, speaker/autocorrect paths, logs/config
|
||||
- `NormalizeRequest`: input transcript, output path, schema, optional report, timeout/log/config
|
||||
- `TrimRequest`: input transcript, output path, keep selector, timeout/log/config
|
||||
|
||||
Outputs:
|
||||
- `MergeResult`, `NormalizeResult`, `TrimResult` with output paths, logs/config paths, exit code, duration, binary provenance, and metadata.
|
||||
|
||||
## Boundaries
|
||||
Owns:
|
||||
- Validated deterministic CLI invocation construction
|
||||
- Optional env tuning propagation for merge
|
||||
- Invocation config file emission
|
||||
- JSON output validation
|
||||
|
||||
Does not own:
|
||||
- Transcript input selection/promotion logic (stage-owned)
|
||||
- Bounds computation (scriptorium/trim-stage-owned)
|
||||
|
||||
## Config Fields Used
|
||||
Via `pipeline.seriatim.*` mapped in app/stage wiring:
|
||||
- `binary`, `timeout`, `output_schema`, `coalesce_gap`, `report`
|
||||
- `env.overlap_word_run_gap`
|
||||
- `env.overlap_word_run_reorder_window`
|
||||
- `env.backchannel_max_duration`
|
||||
- `env.filler_max_duration`
|
||||
|
||||
## External Adapters Used
|
||||
- Shared subprocess helper (`internal/adapters/subprocess`).
|
||||
|
||||
## State and Manifest Behavior
|
||||
- No direct manifest writes.
|
||||
- Stage metadata consumes adapter result fields and preserves generated config/log references.
|
||||
|
||||
## Skip and Resume Behavior
|
||||
- Adapter has no skip/resume logic. Runner controls stage execution.
|
||||
|
||||
## Failure Behavior
|
||||
- Constructor fails for invalid binary/timeout/output-schema/coalesce-gap.
|
||||
- Merge fails on missing output path/inputs/report path (if enabled), subprocess errors, invalid merged output JSON, invalid report JSON.
|
||||
- Normalize fails on missing input/output, invalid schema, subprocess errors, invalid normalized output JSON shape, invalid report JSON.
|
||||
- Trim fails on missing input/output/keep selector, subprocess errors, invalid trimmed output JSON shape.
|
||||
|
||||
## Tests to Inspect Before Changing
|
||||
- `internal/adapters/seriatim/subprocess_test.go`
|
||||
- `internal/adapters/seriatim/fake_test.go`
|
||||
- `internal/stage/merge_test.go`
|
||||
- `internal/stage/normalize_test.go`
|
||||
- `internal/stage/trim_test.go`
|
||||
|
||||
## Architectural Invariants
|
||||
- Supported output schemas are limited to `seriatim-minimal`, `seriatim-intermediate`, `seriatim-full`.
|
||||
- Normalize/trim outputs must include `segments` arrays.
|
||||
- Merge/normalize/trim all route through deterministic subprocess invocation.
|
||||
69
docs/internal/stage-analyze.md
Normal file
69
docs/internal/stage-analyze.md
Normal file
@@ -0,0 +1,69 @@
|
||||
# Stage: analyze
|
||||
|
||||
## Purpose
|
||||
Generate the session recap artifact using configured Scriptorium artifact settings.
|
||||
|
||||
## Inputs and Outputs
|
||||
Inputs:
|
||||
- transcript inputs as requested by selected artifact config (processed/normalized/trimmed/current recap, depending on `pipeline.scriptorium.artifacts.session_recap.inputs`)
|
||||
|
||||
Outputs:
|
||||
- `artifacts/session_recap.md`
|
||||
|
||||
## Boundaries
|
||||
Owns:
|
||||
- Selecting supported analyze artifact (`session_recap` only)
|
||||
- Resolving transcript/reference inputs and vars
|
||||
- Optional render-debug execution before run
|
||||
- Main Scriptorium run and output promotion
|
||||
|
||||
Does not own:
|
||||
- Transcript processing pipeline stages
|
||||
- Archive publish/pointer behavior
|
||||
|
||||
## Config Fields Used
|
||||
- `session.session_id`
|
||||
- `session.campaign`
|
||||
- `pipeline.workspace.root`
|
||||
- `pipeline.scriptorium.binary`
|
||||
- `pipeline.scriptorium.config_path`
|
||||
- `pipeline.scriptorium.timeout`
|
||||
- `pipeline.scriptorium.render_debug`
|
||||
- `pipeline.scriptorium.artifacts.session_recap.*`
|
||||
- `enabled`
|
||||
- `prompt_id`
|
||||
- `profile_id`
|
||||
- `timeout`
|
||||
- `output_path`
|
||||
- `render_debug`
|
||||
- `inputs`
|
||||
- `vars`
|
||||
|
||||
## External Adapters Used
|
||||
- Scriptorium adapter:
|
||||
- optional `RenderArtifact` (debug diagnostics)
|
||||
- `RunArtifact` (actual recap generation)
|
||||
|
||||
## State and Manifest Behavior
|
||||
- If `pipeline.scriptorium` is nil, stage returns success metadata with `skipped=true`.
|
||||
- If no enabled artifacts exist, stage returns success metadata with `skipped=true`.
|
||||
- If enabled artifacts exist but any artifact other than `session_recap` is enabled, stage fails.
|
||||
- Uses run-local output/log/config/reports paths when run layout is enabled.
|
||||
- Promotes canonical recap output and records adapter metadata.
|
||||
|
||||
## Skip and Resume Behavior
|
||||
- Runner-level skip applies when already succeeded and not forced.
|
||||
- Forced reruns can stale downstream succeeded stages.
|
||||
- Stage-local "skipped" metadata is distinct from runner-level stage status skip.
|
||||
|
||||
## Failure Behavior
|
||||
- Fails on missing required resolved inputs, invalid transcript inputs, render/run adapter failures, or validation-failed run results.
|
||||
|
||||
## Tests to Inspect Before Changing
|
||||
- `internal/stage/analyze_test.go`
|
||||
- `internal/adapters/scriptorium/subprocess_test.go`
|
||||
|
||||
## Architectural Invariants
|
||||
- Analyze implementation supports only `artifacts.session_recap` as executable artifact.
|
||||
- Optional inputs may be omitted; required inputs must resolve.
|
||||
- Successful output must exist and be non-empty before promotion.
|
||||
68
docs/internal/stage-archive.md
Normal file
68
docs/internal/stage-archive.md
Normal file
@@ -0,0 +1,68 @@
|
||||
# Stage: archive
|
||||
|
||||
## Purpose
|
||||
Publish run records and promoted session artifacts to object storage, then atomically advance the remote current pointer.
|
||||
|
||||
## Inputs and Outputs
|
||||
Inputs:
|
||||
- session manifest and prerequisite stage records
|
||||
- run root contents under `runs/{run_id}/`
|
||||
- promotion sources from session root (`archive.promote_artifacts`)
|
||||
|
||||
Outputs:
|
||||
- uploaded run files under `{session_prefix}/runs/{run_id}/...`
|
||||
- uploaded promoted artifacts under `{session_prefix}/...`
|
||||
- `{session_prefix}/current/manifest.json`
|
||||
- `{session_prefix}/current/run_id.txt` written last
|
||||
|
||||
## Boundaries
|
||||
Owns:
|
||||
- Archive enable/disable gate behavior
|
||||
- Prerequisite stage success enforcement
|
||||
- Run file collection and upload (excluding `audio/`)
|
||||
- Promotion rule resolution and upload
|
||||
- Commit pointer publish order
|
||||
|
||||
Does not own:
|
||||
- Stage execution before archive
|
||||
- Post-archive local cleanup policy execution (handled by app cleanup logic)
|
||||
|
||||
## Config Fields Used
|
||||
- `pipeline.archive.enabled`
|
||||
- `pipeline.archive.upload_run`
|
||||
- `pipeline.archive.promote_artifacts`
|
||||
- `pipeline.storage.s3.bucket`
|
||||
- `pipeline.storage.s3.root_prefix`
|
||||
- `pipeline.workspace.root`
|
||||
- `session.campaign`
|
||||
- `session.session_id`
|
||||
|
||||
## External Adapters Used
|
||||
- Object storage backend (`env.ObjectStore`) for upload/list primitives.
|
||||
|
||||
## State and Manifest Behavior
|
||||
- Requires `prepare`, `transcribe`, `merge`, `polish`, `normalize`, `trim`, and `analyze` status `succeeded`.
|
||||
- Resolves bucket/prefix from manifest identity first, then config fallback.
|
||||
- Writes metadata including:
|
||||
- upload counts/paths
|
||||
- `current_manifest_key`
|
||||
- `current_run_id_key`
|
||||
- `current_pointer_written`
|
||||
- On skipped archive path, returns metadata with `skipped=true` and pointer not written.
|
||||
|
||||
## Skip and Resume Behavior
|
||||
- Stage may self-skip (metadata skip) when archive disabled or run upload disabled.
|
||||
- Runner-level skip also applies for previously succeeded stage unless forced.
|
||||
|
||||
## Failure Behavior
|
||||
- Fails on missing prerequisite success, missing object store when required, missing run root, missing required promotion source, upload failures, or pointer write failures.
|
||||
- Pointer semantics are fail-safe: `current/run_id.txt` is not written if prior required uploads fail.
|
||||
|
||||
## Tests to Inspect Before Changing
|
||||
- `internal/stage/archive_test.go`
|
||||
- `internal/app/post_archive_cleanup_test.go`
|
||||
|
||||
## Architectural Invariants
|
||||
- Run upload excludes `audio/` subtree.
|
||||
- `current/manifest.json` uploads before `current/run_id.txt`.
|
||||
- `current/run_id.txt` is the remote publish commit marker.
|
||||
63
docs/internal/stage-merge.md
Normal file
63
docs/internal/stage-merge.md
Normal file
@@ -0,0 +1,63 @@
|
||||
# Stage: merge
|
||||
|
||||
## Purpose
|
||||
Normalize per-speaker raw transcripts and merge them into one merged transcript via Seriatim.
|
||||
|
||||
## Inputs and Outputs
|
||||
Inputs:
|
||||
- `transcripts/raw/*.json`
|
||||
- `inputs/speakers.yml`
|
||||
- `inputs/autocorrect.yml`
|
||||
|
||||
Outputs:
|
||||
- `transcripts/merged.json`
|
||||
- optional `artifacts/seriatim.report.json` (when report enabled)
|
||||
|
||||
## Boundaries
|
||||
Owns:
|
||||
- Raw transcript discovery/validation
|
||||
- Per-input normalize calls to Seriatim
|
||||
- Final merge call to Seriatim
|
||||
- Run-local log/config/report path wiring
|
||||
- Promotion of merged/report outputs to canonical paths
|
||||
|
||||
Does not own:
|
||||
- Transcript polishing or downstream artifact generation
|
||||
|
||||
## Config Fields Used
|
||||
- `session.session_id`
|
||||
- `session.campaign`
|
||||
- `pipeline.workspace.root`
|
||||
- `pipeline.seriatim.binary`
|
||||
- `pipeline.seriatim.timeout`
|
||||
- `pipeline.seriatim.output_schema`
|
||||
- `pipeline.seriatim.coalesce_gap`
|
||||
- `pipeline.seriatim.report`
|
||||
- `pipeline.seriatim.env.*`
|
||||
|
||||
## External Adapters Used
|
||||
- Seriatim adapter:
|
||||
- `Normalize` for each raw input
|
||||
- `Run` for final merge
|
||||
|
||||
## State and Manifest Behavior
|
||||
- Reads transcript inputs from transcribe stage outputs in manifest when present; falls back to canonical raw directory.
|
||||
- Writes run-local outputs/logs/config under `runs/{run_id}/merge/...` when enabled.
|
||||
- Promotes canonical merged transcript and optional report.
|
||||
- Records normalized-input provenance and adapter metadata in stage metadata.
|
||||
|
||||
## Skip and Resume Behavior
|
||||
- Runner-level skip applies when already succeeded and not forced.
|
||||
- Forced rerun of this or upstream stages can stale downstream succeeded stages via runner invalidation.
|
||||
|
||||
## Failure Behavior
|
||||
- Fails on missing/invalid raw transcripts, missing speakers/autocorrect files, normalize failure, merge failure, invalid merged output JSON, or invalid report JSON when enabled.
|
||||
|
||||
## Tests to Inspect Before Changing
|
||||
- `internal/stage/merge_test.go`
|
||||
- `internal/adapters/seriatim/subprocess_test.go`
|
||||
|
||||
## Architectural Invariants
|
||||
- Merge consumes normalized forms of each raw transcript.
|
||||
- Merged transcript must validate before promotion.
|
||||
- Report output is optional and gated by config.
|
||||
56
docs/internal/stage-normalize.md
Normal file
56
docs/internal/stage-normalize.md
Normal file
@@ -0,0 +1,56 @@
|
||||
# Stage: normalize
|
||||
|
||||
## Purpose
|
||||
Normalize the processed transcript into a deterministic intermediate schema for trim and optionally emit a normalize report.
|
||||
|
||||
## Inputs and Outputs
|
||||
Inputs:
|
||||
- `transcripts/processed.json`
|
||||
|
||||
Outputs:
|
||||
- `transcripts/normalized.json` (or configured normalize output path)
|
||||
- optional `artifacts/seriatim.normalize.report.json`
|
||||
|
||||
## Boundaries
|
||||
Owns:
|
||||
- Processed transcript discovery/validation
|
||||
- Normalize request construction and invocation
|
||||
- Optional normalize report wiring
|
||||
- Promotion of normalized transcript and optional report
|
||||
|
||||
Does not own:
|
||||
- Bounds detection or segment trimming
|
||||
|
||||
## Config Fields Used
|
||||
- `session.session_id`
|
||||
- `session.campaign`
|
||||
- `pipeline.workspace.root`
|
||||
- `pipeline.normalize.output_path`
|
||||
- `pipeline.normalize.output_schema`
|
||||
- `pipeline.normalize.report`
|
||||
- `pipeline.seriatim.binary`
|
||||
- `pipeline.seriatim.timeout`
|
||||
|
||||
## External Adapters Used
|
||||
- Seriatim adapter (`Normalize`).
|
||||
|
||||
## State and Manifest Behavior
|
||||
- Reads processed transcript from polish outputs in manifest when present; falls back to canonical path.
|
||||
- Uses run-local output/report/log/config paths when run layout is enabled.
|
||||
- Promotes canonical normalized transcript and optional normalize report.
|
||||
- Records adapter/result metadata including source path selection.
|
||||
|
||||
## Skip and Resume Behavior
|
||||
- Runner-level skip applies when already succeeded and not forced.
|
||||
- Forced reruns can stale downstream succeeded stages.
|
||||
|
||||
## Failure Behavior
|
||||
- Fails on missing/invalid processed transcript, adapter error, invalid normalized output, or invalid report output when report enabled.
|
||||
|
||||
## Tests to Inspect Before Changing
|
||||
- `internal/stage/normalize_test.go`
|
||||
- `internal/adapters/seriatim/subprocess_test.go`
|
||||
|
||||
## Architectural Invariants
|
||||
- Normalized output must validate as processed-transcript-compatible JSON (`segments` array required).
|
||||
- Default normalize config is applied when `pipeline.normalize` is unset.
|
||||
69
docs/internal/stage-polish.md
Normal file
69
docs/internal/stage-polish.md
Normal file
@@ -0,0 +1,69 @@
|
||||
# Stage: polish
|
||||
|
||||
## Purpose
|
||||
Polish merged transcript with Audita and produce a processed transcript for downstream normalization/analyze.
|
||||
|
||||
## Inputs and Outputs
|
||||
Inputs:
|
||||
- `transcripts/merged.json`
|
||||
- `inputs/glossary.yml`
|
||||
|
||||
Outputs:
|
||||
- `transcripts/processed.json`
|
||||
- optional `artifacts/audita.report.json` (when report enabled)
|
||||
|
||||
## Boundaries
|
||||
Owns:
|
||||
- Merged transcript discovery/validation
|
||||
- Audita invocation request construction
|
||||
- Run-local logs/config/work-dir/report wiring
|
||||
- Promotion of processed transcript and optional report
|
||||
|
||||
Does not own:
|
||||
- Upstream merge normalization
|
||||
- Downstream normalize/trim/analyze logic
|
||||
|
||||
## Config Fields Used
|
||||
- `session.session_id`
|
||||
- `session.campaign`
|
||||
- `pipeline.workspace.root`
|
||||
- `pipeline.audita.binary`
|
||||
- `pipeline.audita.timeout`
|
||||
- `pipeline.audita.llm_api_key_env`
|
||||
- `pipeline.audita.modules`
|
||||
- `pipeline.audita.base_url`
|
||||
- `pipeline.audita.model`
|
||||
- `pipeline.audita.transcript_description`
|
||||
- `pipeline.audita.config_path`
|
||||
- `pipeline.audita.output_schema`
|
||||
- `pipeline.audita.work_dir_retention`
|
||||
- `pipeline.audita.total_llm_concurrency`
|
||||
- `pipeline.audita.proposal_llm_concurrency`
|
||||
- `pipeline.audita.validation_model`
|
||||
- `pipeline.audita.validation_llm_concurrency`
|
||||
- `pipeline.audita.report`
|
||||
|
||||
## External Adapters Used
|
||||
- Audita adapter (`env.Audita.Run`).
|
||||
|
||||
## State and Manifest Behavior
|
||||
- Reads merged transcript from merge manifest outputs when available; falls back to canonical merged path.
|
||||
- Uses run-local output/report/log/config/scratch paths when run layout is enabled.
|
||||
- Promotes canonical `transcripts/processed.json` and optional report.
|
||||
- Records adapter invocation metadata, credential presence signal, and output provenance in stage metadata.
|
||||
|
||||
## Skip and Resume Behavior
|
||||
- Runner-level skip applies when already succeeded and not forced.
|
||||
- Forced rerun can stale downstream succeeded stages via runner invalidation.
|
||||
|
||||
## Failure Behavior
|
||||
- Fails on missing/invalid merged transcript, missing glossary, adapter error, invalid processed output shape (`segments` array required), or invalid report JSON when enabled.
|
||||
|
||||
## Tests to Inspect Before Changing
|
||||
- `internal/stage/polish_test.go`
|
||||
- `internal/adapters/audita/subprocess_test.go`
|
||||
|
||||
## Architectural Invariants
|
||||
- Processed transcript must contain a top-level `segments` array.
|
||||
- Report behavior is strictly config-gated.
|
||||
- Stage output canonicalization always ends at `transcripts/processed.json`.
|
||||
74
docs/internal/stage-prepare.md
Normal file
74
docs/internal/stage-prepare.md
Normal file
@@ -0,0 +1,74 @@
|
||||
# Stage: prepare
|
||||
|
||||
## Purpose
|
||||
Materialize all required session inputs into canonical local workspace paths and record input provenance in the session manifest.
|
||||
|
||||
## Inputs and Outputs
|
||||
Inputs:
|
||||
- `session.yml` (resolved session config)
|
||||
- `pipeline.resolved.yml` (materialized from resolved pipeline config)
|
||||
- `speakers.yml`
|
||||
- `autocorrect.yml`
|
||||
- `glossary.yml`
|
||||
- audio source:
|
||||
- local (`session.inputs.audio_dir` or `session.inputs.audio_files`), or
|
||||
- S3 (`session.inputs.audio_s3.prefix`)
|
||||
|
||||
Outputs:
|
||||
- `inputs/session.yml`
|
||||
- `inputs/pipeline.resolved.yml`
|
||||
- `inputs/speakers.yml`
|
||||
- `inputs/autocorrect.yml`
|
||||
- `inputs/glossary.yml`
|
||||
- `audio/*.flac` in session workdir
|
||||
- `manifest.Inputs` records with checksums and source metadata
|
||||
|
||||
## Boundaries
|
||||
Owns:
|
||||
- Input path resolution and validation
|
||||
- Local copy/materialization of configs and audio files
|
||||
- S3 audio download to run-scoped spool, then copy into work audio dir
|
||||
|
||||
Does not own:
|
||||
- Transcript generation/processing
|
||||
- Archive publish behavior
|
||||
|
||||
## Config Fields Used
|
||||
- `session.session_id`
|
||||
- `session.campaign`
|
||||
- `session.inputs.speakers_file`
|
||||
- `session.inputs.autocorrect_file`
|
||||
- `session.inputs.glossary_file`
|
||||
- `session.inputs.audio_dir`
|
||||
- `session.inputs.audio_files`
|
||||
- `session.inputs.audio_s3.prefix`
|
||||
- `pipeline.workspace.root`
|
||||
- `pipeline.spool.root`
|
||||
- `pipeline.storage.s3.bucket`
|
||||
- `pipeline.storage.s3.root_prefix`
|
||||
|
||||
## External Adapters Used
|
||||
- Object storage backend (`env.ObjectStore`) for S3 audio list/download when `audio_s3` is configured.
|
||||
|
||||
## State and Manifest Behavior
|
||||
- Ensures workspace layout exists.
|
||||
- Writes resolved config and input files to canonical `inputs/` paths.
|
||||
- Records all prepared inputs into `manifest.Inputs` (sorted deterministically by kind/path).
|
||||
- For S3 audio, records `S3Bucket`, `S3Key`, `S3Size`, `S3ETag`, and `SpoolPath` in each audio input record.
|
||||
|
||||
## Skip and Resume Behavior
|
||||
- Runner-level skip applies when stage already `succeeded` and `--force` is not set.
|
||||
- Stage itself is deterministic/idempotent for unchanged inputs (`copyFileIfChanged`, `writeBytesIfChanged`).
|
||||
|
||||
## Failure Behavior
|
||||
- Fails on missing required files, invalid audio source combinations, no discoverable `.flac` files, duplicate audio basenames, missing object store for S3 mode, or S3 list/download failures.
|
||||
|
||||
## Tests to Inspect Before Changing
|
||||
- `internal/stage/prepare_test.go`
|
||||
- `internal/app/session_cli_test.go`
|
||||
- `internal/config/load_validate_test.go`
|
||||
|
||||
## Architectural Invariants
|
||||
- `audio_dir`/`audio_files` and `audio_s3` are mutually exclusive.
|
||||
- Audio files must be `.flac`.
|
||||
- Canonical `inputs/*` and `audio/*` paths are the durable source for downstream stages.
|
||||
58
docs/internal/stage-transcribe.md
Normal file
58
docs/internal/stage-transcribe.md
Normal file
@@ -0,0 +1,58 @@
|
||||
# Stage: transcribe
|
||||
|
||||
## Purpose
|
||||
Generate per-speaker raw transcripts from prepared audio using WhisperX.
|
||||
|
||||
## Inputs and Outputs
|
||||
Inputs:
|
||||
- `audio/*.flac` prepared by `prepare`
|
||||
|
||||
Outputs:
|
||||
- `transcripts/raw/<speaker>.json` for each input audio file
|
||||
|
||||
## Boundaries
|
||||
Owns:
|
||||
- Discovering prepared audio inputs
|
||||
- Deriving speaker ids from audio basenames
|
||||
- Parallel WhisperX invocation with bounded concurrency
|
||||
- Validating produced JSON and promoting run-local outputs
|
||||
|
||||
Does not own:
|
||||
- Transcript merge/polish/normalize/trim/analyze
|
||||
|
||||
## Config Fields Used
|
||||
- `session.session_id`
|
||||
- `session.campaign`
|
||||
- `pipeline.workspace.root`
|
||||
- `pipeline.whisperx.transcribe_url`
|
||||
- `pipeline.whisperx.language`
|
||||
- `pipeline.whisperx.timeout`
|
||||
- `pipeline.whisperx.retries`
|
||||
- `pipeline.whisperx.retry_delay`
|
||||
- `pipeline.whisperx.concurrency`
|
||||
|
||||
## External Adapters Used
|
||||
- WhisperX adapter (`env.WhisperX.Transcribe`).
|
||||
|
||||
## State and Manifest Behavior
|
||||
- Uses run-local output paths under `runs/{run_id}/transcribe/outputs/...` when run layout is enabled.
|
||||
- Validates each generated transcript JSON before promotion.
|
||||
- Promotes canonical outputs to `transcripts/raw/*.json`.
|
||||
- Records per-file metadata (attempts/status/duration/output path) in stage metadata.
|
||||
|
||||
## Skip and Resume Behavior
|
||||
- Runner-level skip applies for previously succeeded stage unless forced.
|
||||
- On forced upstream reruns, downstream succeeded stages can be marked `stale` by runner logic.
|
||||
|
||||
## Failure Behavior
|
||||
- Fails if no prepared audio exists, duplicate speaker basenames are detected, adapter output path mismatches expected path, any output JSON is invalid, or one worker fails.
|
||||
- Cancels in-flight workers after first terminal error.
|
||||
|
||||
## Tests to Inspect Before Changing
|
||||
- `internal/stage/transcribe_test.go`
|
||||
- `internal/app/whisperx_wiring_test.go`
|
||||
|
||||
## Architectural Invariants
|
||||
- Speaker identity is derived from `.flac` basename and must be unique.
|
||||
- Every successful speaker output must be valid JSON before promotion.
|
||||
- Canonical raw transcript set is the only supported merge input surface.
|
||||
75
docs/internal/stage-trim.md
Normal file
75
docs/internal/stage-trim.md
Normal file
@@ -0,0 +1,75 @@
|
||||
# Stage: trim
|
||||
|
||||
## Purpose
|
||||
Optionally trim the normalized transcript to session bounds; always produce a durable trimmed transcript.
|
||||
|
||||
## Inputs and Outputs
|
||||
Inputs:
|
||||
- `transcripts/normalized.json`
|
||||
|
||||
Outputs:
|
||||
- `transcripts/trimmed.json` (or configured trim output path)
|
||||
- when trim enabled: `artifacts/session_bounds.json`
|
||||
|
||||
## Boundaries
|
||||
Owns:
|
||||
- Trim-enabled switch behavior
|
||||
- Bounds generation via Scriptorium artifact run
|
||||
- Bounds validation against normalized transcript
|
||||
- Keep-selector derivation and Seriatim trim invocation
|
||||
- Copy-through behavior when disabled or bounds indicate unchanged transcript
|
||||
|
||||
Does not own:
|
||||
- Upstream normalization
|
||||
- Downstream artifact analysis
|
||||
|
||||
## Config Fields Used
|
||||
- `session.session_id`
|
||||
- `session.campaign`
|
||||
- `pipeline.workspace.root`
|
||||
- `pipeline.trim.enabled`
|
||||
- `pipeline.trim.output_path`
|
||||
- `pipeline.trim.bounds.prompt_id`
|
||||
- `pipeline.trim.bounds.profile_id`
|
||||
- `pipeline.trim.bounds.timeout`
|
||||
- `pipeline.trim.bounds.output_path`
|
||||
- `pipeline.trim.bounds.transcript_input_name`
|
||||
- `pipeline.trim.bounds.render_debug`
|
||||
- `pipeline.trim.bounds.render_output_path`
|
||||
- `pipeline.seriatim.binary`
|
||||
- `pipeline.seriatim.timeout`
|
||||
- `pipeline.scriptorium.binary`
|
||||
- `pipeline.scriptorium.config_path`
|
||||
- `pipeline.scriptorium.timeout`
|
||||
|
||||
## External Adapters Used
|
||||
- Scriptorium adapter:
|
||||
- optional `RenderArtifact` for bounds debug render
|
||||
- `RunArtifact` for bounds output
|
||||
- Seriatim adapter:
|
||||
- `Trim` when bounds indicate trimming is required
|
||||
|
||||
## State and Manifest Behavior
|
||||
- Reads normalized transcript from normalize manifest outputs when available; falls back to canonical path.
|
||||
- Uses run-local outputs/logs/reports/config/scratch paths when run layout is enabled.
|
||||
- Promotes canonical trimmed transcript; promotes session bounds when trim enabled.
|
||||
- Records bounds diagnostics, trim action, keep selector, and adapter metadata.
|
||||
|
||||
## Skip and Resume Behavior
|
||||
- Runner-level skip applies when already succeeded and not forced.
|
||||
- Forced reruns can stale downstream succeeded stages.
|
||||
- When `trim.enabled=false`, stage still succeeds by copying normalized to trimmed output.
|
||||
|
||||
## Failure Behavior
|
||||
- Fails on missing/invalid normalized transcript.
|
||||
- With trim enabled, fails on missing adapters/config, bounds generation/validation errors, invalid bounds JSON, invalid range/segment ids, trim adapter failures, or invalid trimmed output.
|
||||
|
||||
## Tests to Inspect Before Changing
|
||||
- `internal/stage/trim_test.go`
|
||||
- `internal/adapters/scriptorium/subprocess_test.go`
|
||||
- `internal/adapters/seriatim/subprocess_test.go`
|
||||
|
||||
## Architectural Invariants
|
||||
- Trim never falls back to processed transcript; normalized transcript is required input.
|
||||
- `session_bounds` output exists only for enabled trim path.
|
||||
- Render-debug artifacts are diagnostics and not declared stage outputs.
|
||||
68
docs/internal/workspace.md
Normal file
68
docs/internal/workspace.md
Normal file
@@ -0,0 +1,68 @@
|
||||
# Workspace internals
|
||||
|
||||
## Purpose
|
||||
Define the local durable and run-local workspace model used by stages, manifests, resume, and archive.
|
||||
|
||||
## Inputs and Outputs
|
||||
Inputs:
|
||||
- `pipeline.workspace.root`
|
||||
- `session.campaign`
|
||||
- `session.session_id`
|
||||
- generated `run_id`
|
||||
|
||||
Outputs:
|
||||
- Session manifest at `{workspace.root}/work/{campaign}/{session_id}/manifest.json`
|
||||
- Run manifest at `{workspace.root}/work/{campaign}/{session_id}/runs/{run_id}/manifest.json`
|
||||
- Canonical durable session directories and run-local stage trees
|
||||
|
||||
## Boundaries
|
||||
Owns:
|
||||
- Session-level path layout (`inputs/`, `audio/`, `transcripts/`, `artifacts/`, `reports/`, `logs/`, `config/`, `current/`, `runs/`)
|
||||
- Run-local stage sandbox layout under `runs/{run_id}/{stage}/`
|
||||
- Session lock acquisition/release (`.lock`)
|
||||
|
||||
Does not own:
|
||||
- Stage business logic
|
||||
- Remote archive semantics (documented in `stage-archive.md`)
|
||||
- CLI argument parsing
|
||||
|
||||
## Config Fields Used
|
||||
- `pipeline.workspace.root`
|
||||
- `pipeline.workspace.cleanup_after_archive`
|
||||
- `pipeline.spool.root`
|
||||
- `pipeline.spool.delete_audio_after_archive`
|
||||
- `session.campaign`
|
||||
- `session.session_id`
|
||||
|
||||
## External Adapters Used
|
||||
None directly in this subsystem. Stages may use object storage adapters and then write local outputs into this layout.
|
||||
|
||||
## State and Manifest Behavior
|
||||
- Session state is persisted in the session manifest (`manifest.Manifest`).
|
||||
- Invocation history is persisted per run in run manifests under `runs/{run_id}/manifest.json`.
|
||||
- During each run, stage outputs are often written run-local first (`runs/{run_id}/{stage}/outputs/...`) and promoted to canonical session paths after stage success.
|
||||
- `manifest.Artifacts` entries record `ProducerRunID` for durable outputs.
|
||||
- For S3 audio sessions, `prepare` records spool/work paths and S3 provenance in `manifest.Inputs`.
|
||||
|
||||
## Skip and Resume Behavior
|
||||
- Skip/resume decisions are made in `internal/app` (`run_control.go`, `resume.go`) using stage status in the session manifest.
|
||||
- `--force` reruns selected stages and marks downstream previously-succeeded stages as `stale`.
|
||||
- Workspace layout is idempotent (`EnsureLayoutFor`) and reused across runs.
|
||||
|
||||
## Failure Behavior
|
||||
- Failures preserve manifests and run-local files for inspection.
|
||||
- Lock conflicts fail fast via `ErrLockConflict`.
|
||||
- Cleanup can fail post-archive; failure is recorded in archive stage metadata and returned by the run.
|
||||
|
||||
## Tests to Inspect Before Changing
|
||||
- `internal/artifacts/local_test.go`
|
||||
- `internal/stage/run_local_test.go`
|
||||
- `internal/app/run_control_test.go`
|
||||
- `internal/app/resume_run_stage_test.go`
|
||||
- `internal/app/post_archive_cleanup_test.go`
|
||||
|
||||
## Architectural Invariants
|
||||
- Session root is campaign-aware: `{workspace.root}/work/{campaign}/{session_id}`.
|
||||
- Run roots are always nested: `runs/{run_id}` under the session root.
|
||||
- Run-local output promotion must end in canonical session paths.
|
||||
- Cleanup only targets run-scoped directories and must never delete configured root directories.
|
||||
170
docs/operations.md
Normal file
170
docs/operations.md
Normal file
@@ -0,0 +1,170 @@
|
||||
# Operations
|
||||
|
||||
This guide describes the implemented operator lifecycle for Narratio.
|
||||
|
||||
For field-level configuration, see [docs/config.md](./config.md). For full command/flag reference, see [docs/cli.md](./cli.md).
|
||||
|
||||
## Normal workflow (S3-first path)
|
||||
|
||||
1. Upload session `.flac` files to the session audio prefix in object storage:
|
||||
- `{root_prefix}/campaigns/{campaign}/sessions/{session_id}/{audio_s3.prefix}`
|
||||
2. Run Narratio:
|
||||
|
||||
```bash
|
||||
narratio run --session-id 2026-04-04
|
||||
```
|
||||
|
||||
3. Read success output:
|
||||
- `narratio run: session <session_id>; executed=<n> skipped=<n>; manifest=<path>`
|
||||
- `manifest=<path>` is the local session manifest path to use with `status`.
|
||||
|
||||
Notes:
|
||||
|
||||
- This command relies on discoverable `pipeline.yml` and `session.yml` unless `--config` and `--session` are passed explicitly.
|
||||
- For S3 audio input, `session.inputs.audio_s3.prefix` must be configured and audio files must already exist remotely.
|
||||
|
||||
## Local filesystem layout and state artifacts
|
||||
|
||||
Session root:
|
||||
|
||||
- `{workspace.root}/work/{campaign}/{session_id}/`
|
||||
|
||||
Primary state:
|
||||
|
||||
- `manifest.json`: session-level manifest (authoritative local stage state).
|
||||
- `runs/{run_id}/manifest.json`: run-level manifest for one invocation.
|
||||
- `.lock`: session lock file while a run is active.
|
||||
|
||||
Canonical session directories:
|
||||
|
||||
- `inputs/`
|
||||
- `audio/`
|
||||
- `transcripts/`
|
||||
- `artifacts/`
|
||||
- `reports/`
|
||||
- `logs/`
|
||||
- `config/`
|
||||
- `current/`
|
||||
- `runs/`
|
||||
|
||||
Run-local stage directories:
|
||||
|
||||
- `runs/{run_id}/{stage}/`
|
||||
- Stage runtime files are written under deterministic run-local subdirectories such as:
|
||||
- `outputs/`, `logs/`, `reports/`, `config/`, `scratch/`
|
||||
|
||||
Behavior notes:
|
||||
|
||||
- Layout creation is idempotent.
|
||||
- Durable outputs are promoted to canonical session paths after stage success.
|
||||
- Run-local artifacts remain in `runs/{run_id}/...` unless configured post-archive cleanup removes that run scope.
|
||||
|
||||
## Remote archive layout and publish contract
|
||||
|
||||
When archive is enabled and run upload is enabled, archive publishes to object storage under:
|
||||
|
||||
- Session prefix: `{root_prefix}/campaigns/{campaign}/sessions/{session_id}/`
|
||||
- Run prefix: `{session_prefix}/runs/{run_id}/`
|
||||
|
||||
Archive uploads:
|
||||
|
||||
- Run record files from run root (including stage subtrees and run manifest), excluding local `audio/`.
|
||||
- Promoted artifacts from `archive.promote_artifacts` to session-level keys.
|
||||
|
||||
Publish order (commit contract):
|
||||
|
||||
1. Upload `current/manifest.json`
|
||||
2. Upload `current/run_id.txt` last
|
||||
|
||||
Meaning of `current/run_id.txt`:
|
||||
|
||||
- It is the effective remote commit marker for published session state.
|
||||
- It is written only after required run uploads and required promotions succeed.
|
||||
|
||||
## Resume, retry, and safe rerun behavior
|
||||
|
||||
Default skip behavior:
|
||||
|
||||
- `run` and `run-stage` skip stages already marked `succeeded` unless `--force` is set.
|
||||
|
||||
Resume behavior:
|
||||
|
||||
- `resume` starts at the first non-`succeeded` stage in canonical stage order.
|
||||
- If all stages are `succeeded`, `resume` prints that no stages remain.
|
||||
- `resume --force` runs full stage order rather than starting at first non-succeeded.
|
||||
|
||||
Forced rerun behavior:
|
||||
|
||||
- Successful forced rerun of an upstream stage marks downstream previously `succeeded` stages as `stale`.
|
||||
- `stale` stages are not treated as complete and are eligible to run in subsequent commands.
|
||||
|
||||
Targeted rerun with one stage:
|
||||
|
||||
```bash
|
||||
narratio run-stage --force <stage>
|
||||
```
|
||||
|
||||
Valid stage names:
|
||||
|
||||
- `prepare`, `transcribe`, `merge`, `polish`, `normalize`, `trim`, `analyze`, `archive`, `notify`
|
||||
|
||||
Safe operator pattern:
|
||||
|
||||
1. Force-rerun the stage that changed.
|
||||
2. Run `resume` to rebuild downstream stages in order.
|
||||
|
||||
## Cleanup behavior
|
||||
|
||||
Cleanup is considered only after run execution completes and only when archive stage both executed and succeeded.
|
||||
|
||||
Configured cleanup toggles:
|
||||
|
||||
- `pipeline.spool.delete_audio_after_archive=true`
|
||||
- deletes only run-scoped spool audio directory: `{spool.root}/{campaign}/{session_id}/{run_id}/audio/`
|
||||
- `pipeline.workspace.cleanup_after_archive=true`
|
||||
- deletes only run-scoped local run directory: `{workspace.root}/work/{campaign}/{session_id}/runs/{run_id}/`
|
||||
|
||||
Eligibility gates for cleanup:
|
||||
|
||||
- archive is enabled
|
||||
- archive run upload is enabled
|
||||
- archive metadata indicates run record upload happened
|
||||
- archive metadata indicates `current` pointer write completed (`current/run_id.txt` written)
|
||||
|
||||
Cleanup does not run for:
|
||||
|
||||
- failed runs
|
||||
- incomplete runs
|
||||
- unarchived runs
|
||||
- archive-skipped runs (`archive.enabled=false` or `archive.upload_run=false`)
|
||||
|
||||
## Failure and recovery playbooks
|
||||
|
||||
What remains after failure:
|
||||
|
||||
- Session manifest remains on disk.
|
||||
- Run manifest remains under `runs/{run_id}/manifest.json`.
|
||||
- Run-local stage artifacts/logs/config/reports remain under `runs/{run_id}/...`.
|
||||
- Failed/incomplete runs remain local-only.
|
||||
- Remote current pointer is not committed if archive prerequisite or pointer-write steps fail.
|
||||
|
||||
Recommended recovery flow:
|
||||
|
||||
1. Inspect current state:
|
||||
|
||||
```bash
|
||||
narratio status --manifest <manifest-path-from-run-output>
|
||||
```
|
||||
|
||||
2. Fix the root cause (config, input, credentials, adapter availability, etc.).
|
||||
3. Continue with:
|
||||
- `narratio resume --session-id <id>` for ordered continuation, or
|
||||
- `narratio run-stage --force <stage>` for targeted correction, then `resume`.
|
||||
|
||||
## Operational caveats
|
||||
|
||||
- `status` requires an explicit manifest path; there is no direct session-id lookup command.
|
||||
- S3 audio mode and local audio mode are mutually exclusive in session config.
|
||||
- Archive verifies stage prerequisites (`prepare` through `analyze`) before publishing.
|
||||
- By default, archive does not upload local `audio/` into run history.
|
||||
- Unknown CLI commands fail and print usage.
|
||||
@@ -1,119 +0,0 @@
|
||||
# Archive Storage
|
||||
|
||||
This document describes implemented archive-stage publish behavior.
|
||||
|
||||
## S3 Paths
|
||||
|
||||
Session root:
|
||||
|
||||
`{root_prefix}/campaigns/{campaign}/sessions/{session_id}/`
|
||||
|
||||
Run prefix:
|
||||
|
||||
`{root_prefix}/campaigns/{campaign}/sessions/{session_id}/runs/{run_id}/`
|
||||
|
||||
## Scope
|
||||
|
||||
Implemented:
|
||||
|
||||
- archive uploads successful run records to remote object storage through the storage backend abstraction.
|
||||
- archive uploads configured promoted outputs to session-level keys.
|
||||
- archive uploads `current/manifest.json`.
|
||||
- archive uploads `current/run_id.txt` last as the effective commit marker.
|
||||
- optional post-archive local cleanup:
|
||||
- `pipeline.spool.delete_audio_after_archive: true` removes only the run-scoped spool audio directory
|
||||
- `pipeline.workspace.cleanup_after_archive: true` removes only the run-scoped local workdir
|
||||
- tests use fake storage and do not require live S3.
|
||||
|
||||
Future work:
|
||||
|
||||
- `notify` stage behavior
|
||||
- stale detection
|
||||
- optional future source-audio upload mode
|
||||
- additional artifact generation beyond current implemented set
|
||||
|
||||
## Prerequisites
|
||||
|
||||
Archive verifies these stages succeeded before upload:
|
||||
|
||||
- `prepare`
|
||||
- `transcribe`
|
||||
- `merge`
|
||||
- `polish`
|
||||
- `normalize`
|
||||
- `trim`
|
||||
- `analyze`
|
||||
|
||||
If any prerequisite is missing or not succeeded, archive fails and does not upload.
|
||||
Failed or incomplete runs remain local only.
|
||||
|
||||
## Run Upload
|
||||
|
||||
Archive uploads existing files from the run workdir when present:
|
||||
|
||||
- `inputs/`
|
||||
- `transcripts/`
|
||||
- `artifacts/`
|
||||
- `reports/` (optional)
|
||||
- `config/`
|
||||
- `logs/`
|
||||
- `manifest.json`
|
||||
|
||||
Relative paths are preserved under `runs/{run_id}/`.
|
||||
|
||||
## Promotion Rules
|
||||
|
||||
Archive applies `archive.promote_artifacts` in config order.
|
||||
|
||||
Rule behavior:
|
||||
|
||||
- `from`: local workdir-relative source path
|
||||
- `to`: session-root-relative destination key
|
||||
- `required: true`: missing source fails archive
|
||||
- `required: false`: missing source is skipped and recorded
|
||||
|
||||
Default promoted outputs:
|
||||
|
||||
- `transcripts/trimmed.json`
|
||||
- `artifacts/session_recap.md`
|
||||
|
||||
## Current Pointers
|
||||
|
||||
Archive writes:
|
||||
|
||||
1. `current/manifest.json` (after run upload + promotions)
|
||||
2. `current/run_id.txt` last
|
||||
|
||||
`current/run_id.txt` contains exactly:
|
||||
|
||||
- `{run_id}` plus trailing newline
|
||||
|
||||
Writing `current/run_id.txt` last makes it the effective commit marker for published session state.
|
||||
|
||||
If any required run upload, promotion upload, or current-manifest upload fails, archive returns failure and does not write `current/run_id.txt`.
|
||||
Cleanup runs only after this commit-marker write has succeeded.
|
||||
|
||||
## Audio Upload Policy
|
||||
|
||||
Archive does not upload local `audio/` by default.
|
||||
Original audio is expected at the session-level audio prefix and is not duplicated under `runs/{run_id}/`.
|
||||
|
||||
## Config Controls
|
||||
|
||||
- `archive.enabled: false` skips archive cleanly.
|
||||
- `archive.upload_run: false` skips run upload cleanly.
|
||||
- both skip cases also skip post-archive local cleanup.
|
||||
|
||||
## Metadata
|
||||
|
||||
Archive stage metadata includes non-secret upload context (for example):
|
||||
|
||||
- `s3_bucket`
|
||||
- `s3_run_prefix`
|
||||
- run upload counts/paths
|
||||
- promoted upload counts/paths
|
||||
- skipped optional promotions
|
||||
- `current_manifest_key`
|
||||
- `current_run_id_key`
|
||||
- `current_pointer_written`
|
||||
- `audio_upload_skipped`
|
||||
@@ -1,84 +0,0 @@
|
||||
# S3 Audio Input
|
||||
|
||||
This document describes implemented S3 audio input behavior in `prepare`.
|
||||
|
||||
## Scope
|
||||
|
||||
Implemented:
|
||||
|
||||
- `prepare` can acquire source audio from S3 when `session.inputs.audio_s3.prefix` is configured.
|
||||
- object listing and download go through the storage backend abstraction.
|
||||
- tests use fake storage; no live S3 service is required for test runs.
|
||||
|
||||
Not implemented:
|
||||
|
||||
- uploads of failed runs
|
||||
|
||||
## Required Configuration
|
||||
|
||||
`pipeline.yml`:
|
||||
|
||||
- `storage.s3.bucket` must be set when S3 audio input is used.
|
||||
- `storage.s3.root_prefix` defaults to `dnd`.
|
||||
- `storage.s3.access_key_id_env` defaults to `OBJECT_STORAGE_KEY_ID`.
|
||||
- `storage.s3.secret_access_key_env` defaults to `OBJECT_STORAGE_KEY`.
|
||||
- `spool.root` defaults to `/var/spool/narratio`.
|
||||
|
||||
`session.yml`:
|
||||
|
||||
- configure `session.campaign` and `session.session_id`.
|
||||
- configure `session.inputs.audio_s3.prefix` for S3 audio input.
|
||||
- do not configure `inputs.audio_dir` or `inputs.audio_files` at the same time as `inputs.audio_s3`.
|
||||
|
||||
## Prefix Shape
|
||||
|
||||
Session S3 root:
|
||||
|
||||
`{root_prefix}/campaigns/{campaign}/sessions/{session_id}/`
|
||||
|
||||
Audio prefix:
|
||||
|
||||
`{session_root}/{audio_s3.prefix}`
|
||||
|
||||
Example:
|
||||
|
||||
`dnd/campaigns/forsaken/sessions/2026-04-19/audio/`
|
||||
|
||||
Audio files must already exist in S3 before running Narratio.
|
||||
|
||||
## Prepare Behavior
|
||||
|
||||
When `inputs.audio_s3.prefix` is configured, `prepare`:
|
||||
|
||||
1. lists objects under the computed S3 audio prefix
|
||||
2. filters to `.flac` objects
|
||||
3. fails when no `.flac` objects are found
|
||||
4. downloads selected objects to spool audio:
|
||||
- `{spool.root}/{campaign}/{session_id}/{run_id}/audio/`
|
||||
5. materializes audio into workdir audio:
|
||||
- `{workspace.root}/work/{campaign}/{session_id}/runs/{run_id}/audio/`
|
||||
6. records input provenance in the manifest (bucket, key, metadata, local paths, checksum)
|
||||
|
||||
Notes:
|
||||
|
||||
- `.flac` filtering is case-insensitive.
|
||||
- ETag is recorded as provider metadata only and is not treated as a checksum.
|
||||
|
||||
## Local Audio Development
|
||||
|
||||
Local audio workflows remain supported:
|
||||
|
||||
- `inputs.audio_dir`
|
||||
- `inputs.audio_files`
|
||||
|
||||
These options are mutually exclusive with `inputs.audio_s3`.
|
||||
|
||||
## Archive Boundary
|
||||
|
||||
Current archive behavior relevant to S3 audio input:
|
||||
|
||||
- successful runs are uploaded by archive under `runs/{run_id}/`
|
||||
- configured promotions are uploaded to session-level destinations
|
||||
- `current/manifest.json` and `current/run_id.txt` are published
|
||||
- local source audio is not re-uploaded by default
|
||||
- failed or incomplete runs are not uploaded
|
||||
229
docs/troubleshooting.md
Normal file
229
docs/troubleshooting.md
Normal file
@@ -0,0 +1,229 @@
|
||||
# Troubleshooting
|
||||
|
||||
## Purpose
|
||||
Canonical operator troubleshooting guide for recurring implemented Narratio failures.
|
||||
|
||||
## Config file discovery failure
|
||||
|
||||
Symptom:
|
||||
- `run`, `plan`, `resume`, or `run-stage` fails saying config/session file was not found.
|
||||
|
||||
Likely Cause:
|
||||
- `pipeline.yml` or `session.yml` is missing from default search paths.
|
||||
- Wrong working directory when relying on `./session.yml`.
|
||||
|
||||
Diagnostics:
|
||||
|
||||
```bash
|
||||
pwd
|
||||
ls -l ./session.yml
|
||||
ls -l /usr/local/etc/narratio/pipeline.yml /etc/narratio/pipeline.yml
|
||||
```
|
||||
|
||||
Safe Fix:
|
||||
- Pass explicit paths with `--config` and `--session`.
|
||||
- Or place files in documented discovery paths.
|
||||
|
||||
Links:
|
||||
- [docs/config.md](./config.md)
|
||||
- [docs/cli.md](./cli.md)
|
||||
|
||||
## Session template rendering failure
|
||||
|
||||
Symptom:
|
||||
- Load fails with unresolved template placeholder or `session_id` mismatch.
|
||||
|
||||
Likely Cause:
|
||||
- `session.yml` contains `{{session_id}}`/`{{ session_id }}` but `--session-id` was omitted.
|
||||
- Provided `--session-id` does not match rendered `session_id`.
|
||||
|
||||
Diagnostics:
|
||||
|
||||
```bash
|
||||
narratio plan --session ./session.yml --session-id 2026-04-04
|
||||
```
|
||||
|
||||
Safe Fix:
|
||||
- Always pass `--session-id` when using template placeholders.
|
||||
- Ensure rendered `session_id` equals intended run session id.
|
||||
|
||||
Links:
|
||||
- [docs/config.md](./config.md)
|
||||
|
||||
## Strict YAML decode or validation failure
|
||||
|
||||
Symptom:
|
||||
- Config load fails with unknown field, missing required field, invalid duration, or invalid cross-field constraint.
|
||||
|
||||
Likely Cause:
|
||||
- YAML key typo or stale field name.
|
||||
- Required fields missing.
|
||||
- Invalid value format (for example duration/URL/env var name).
|
||||
|
||||
Diagnostics:
|
||||
|
||||
```bash
|
||||
narratio plan --config /path/to/pipeline.yml --session /path/to/session.yml --session-id 2026-04-04
|
||||
```
|
||||
|
||||
Safe Fix:
|
||||
- Correct fields/values to match canonical reference and examples.
|
||||
- Validate against `docs/examples/` shapes.
|
||||
|
||||
Links:
|
||||
- [docs/config.md](./config.md)
|
||||
- [docs/examples/](./examples/)
|
||||
|
||||
## Manifest/status path failure
|
||||
|
||||
Symptom:
|
||||
- `status` fails because manifest path is missing, unreadable, or invalid.
|
||||
|
||||
Likely Cause:
|
||||
- Wrong manifest path.
|
||||
- Manifest removed after cleanup.
|
||||
- Trying to run `status` without `--manifest`.
|
||||
|
||||
Diagnostics:
|
||||
|
||||
```bash
|
||||
narratio status --manifest /path/to/manifest.json
|
||||
ls -l /path/to/manifest.json
|
||||
```
|
||||
|
||||
Safe Fix:
|
||||
- Use manifest path printed by `run`, `resume`, or `run-stage` output.
|
||||
- Re-run with correct session/config if inspecting a different session.
|
||||
|
||||
Links:
|
||||
- [docs/cli.md](./cli.md)
|
||||
- [docs/operations.md](./operations.md)
|
||||
|
||||
## Session lock conflict (`.lock`)
|
||||
|
||||
Symptom:
|
||||
- Run fails with lock conflict indicating session workdir is already locked.
|
||||
|
||||
Likely Cause:
|
||||
- Another Narratio process is actively running the same session.
|
||||
- Prior run exited unexpectedly and left a stale lock file.
|
||||
|
||||
Diagnostics:
|
||||
|
||||
```bash
|
||||
ls -l {workspace.root}/work/{campaign}/{session_id}/.lock
|
||||
cat {workspace.root}/work/{campaign}/{session_id}/.lock
|
||||
ps aux | grep narratio
|
||||
```
|
||||
|
||||
Safe Fix:
|
||||
- If another run is active, wait for it to finish.
|
||||
- If no process is active and lock is stale, remove only that session `.lock` file and retry.
|
||||
|
||||
Links:
|
||||
- [docs/operations.md](./operations.md)
|
||||
- [docs/internal/workspace.md](./internal/workspace.md)
|
||||
|
||||
## Secrets env-dir or credential env failure
|
||||
|
||||
Symptom:
|
||||
- Startup fails loading secrets directory, or a stage fails because required credential env var is missing.
|
||||
|
||||
Likely Cause:
|
||||
- `pipeline.secrets.env_dir` path is wrong/unreadable.
|
||||
- Credential env var referenced in config is unset or empty.
|
||||
|
||||
Diagnostics:
|
||||
|
||||
```bash
|
||||
ls -la /path/to/secrets_dir
|
||||
env | grep -E 'AUDITA|OBJECT_STORAGE|AWS|SCRIPTORIUM'
|
||||
```
|
||||
|
||||
Safe Fix:
|
||||
- Fix `pipeline.secrets.env_dir` path/permissions.
|
||||
- Ensure required env vars are set to non-empty values.
|
||||
- Keep secrets out of YAML; use env references only.
|
||||
|
||||
Links:
|
||||
- [docs/config.md](./config.md)
|
||||
- [docs/operations.md](./operations.md)
|
||||
|
||||
## S3-audio prepare failure
|
||||
|
||||
Symptom:
|
||||
- `prepare` fails in S3 mode (no audio found, list/download failure, backend missing, path conflict).
|
||||
|
||||
Likely Cause:
|
||||
- Wrong `session.inputs.audio_s3.prefix`.
|
||||
- No `.flac` files at expected prefix.
|
||||
- Missing or invalid S3 backend credentials/config.
|
||||
- Conflicting audio-source settings (`audio_s3` plus local audio fields).
|
||||
|
||||
Diagnostics:
|
||||
|
||||
```bash
|
||||
narratio run-stage --config /path/to/pipeline.yml --session /path/to/session.yml --session-id 2026-04-04 prepare
|
||||
```
|
||||
|
||||
Safe Fix:
|
||||
- Ensure `audio_s3` is the only audio source configured for that session.
|
||||
- Confirm `.flac` objects exist under the resolved session audio prefix.
|
||||
- Fix S3 storage configuration and credentials.
|
||||
|
||||
Links:
|
||||
- [docs/config.md](./config.md)
|
||||
- [docs/operations.md](./operations.md)
|
||||
- [docs/internal/stage-prepare.md](./internal/stage-prepare.md)
|
||||
|
||||
## Archive prerequisite or promotion/current-pointer failure
|
||||
|
||||
Symptom:
|
||||
- `archive` fails due to prerequisite stage status, missing required promotion source, or pointer write failure.
|
||||
|
||||
Likely Cause:
|
||||
- One or more prerequisite stages are not `succeeded`.
|
||||
- Required promoted artifact does not exist.
|
||||
- Remote upload failure before `current/run_id.txt` write.
|
||||
|
||||
Diagnostics:
|
||||
|
||||
```bash
|
||||
narratio status --manifest /path/to/manifest.json
|
||||
narratio run-stage --config /path/to/pipeline.yml --session /path/to/session.yml --session-id 2026-04-04 archive
|
||||
```
|
||||
|
||||
Safe Fix:
|
||||
- Resume or rerun failed upstream stage(s).
|
||||
- Ensure required promoted artifact paths exist locally before archive.
|
||||
- Retry archive after storage/connectivity issue is resolved.
|
||||
|
||||
Links:
|
||||
- [docs/operations.md](./operations.md)
|
||||
- [docs/config.md](./config.md)
|
||||
- [docs/internal/stage-archive.md](./internal/stage-archive.md)
|
||||
|
||||
## `run-stage` invalid stage name or invalid flags
|
||||
|
||||
Symptom:
|
||||
- `run-stage` fails with unknown stage or invalid flag/argument usage.
|
||||
|
||||
Likely Cause:
|
||||
- Stage name typo.
|
||||
- Missing positional stage argument.
|
||||
- Unsupported/incorrect flag syntax.
|
||||
|
||||
Diagnostics:
|
||||
|
||||
```bash
|
||||
narratio run-stage --config /path/to/pipeline.yml --session /path/to/session.yml --session-id 2026-04-04 normalize
|
||||
```
|
||||
|
||||
Safe Fix:
|
||||
- Use only supported stage names.
|
||||
- Provide exactly one positional stage argument.
|
||||
- Align flags to documented command reference.
|
||||
|
||||
Links:
|
||||
- [docs/cli.md](./cli.md)
|
||||
- [docs/operations.md](./operations.md)
|
||||
@@ -1,26 +0,0 @@
|
||||
workspace:
|
||||
root: ./tmp/narratio-workspace
|
||||
|
||||
whisperx:
|
||||
transcribe_url: "https://transcription.example.com/transcribe"
|
||||
|
||||
seriatim:
|
||||
binary: "seriatim"
|
||||
timeout: "10m"
|
||||
output_schema: "seriatim-intermediate"
|
||||
coalesce_gap: 3.0
|
||||
|
||||
audita:
|
||||
binary: "audita"
|
||||
timeout: "3h"
|
||||
base_url: "https://openrouter.ai/api/v1"
|
||||
model: "openrouter/google/gemma-4-31b-it"
|
||||
llm_api_key_env: "AUDITA_LLM_API_KEY"
|
||||
modules: ["glossary", "homophones", "spoken_word", "grammar"]
|
||||
output_schema: "audita-v1"
|
||||
work_dir_retention: "auto"
|
||||
total_llm_concurrency: 2
|
||||
proposal_llm_concurrency: 1
|
||||
validation_model: "openrouter/google/gemma-4-31b-it"
|
||||
validation_llm_concurrency: 1
|
||||
report: true
|
||||
@@ -1,55 +0,0 @@
|
||||
workspace:
|
||||
root: ./tmp/narratio-workspace
|
||||
cleanup_after_archive: false
|
||||
|
||||
storage:
|
||||
backend: s3
|
||||
s3:
|
||||
bucket: "my-dnd-archive"
|
||||
root_prefix: "dnd"
|
||||
region: "us-east-1"
|
||||
# Optional credential env-var names (defaulted when omitted):
|
||||
# access_key_id_env: "OBJECT_STORAGE_KEY_ID"
|
||||
# secret_access_key_env: "OBJECT_STORAGE_KEY"
|
||||
|
||||
spool:
|
||||
root: "/var/spool/narratio"
|
||||
delete_audio_after_archive: false
|
||||
|
||||
archive:
|
||||
enabled: true
|
||||
upload_run: true
|
||||
|
||||
whisperx:
|
||||
transcribe_url: "https://transcription.example.com/transcribe"
|
||||
|
||||
# Optional. When omitted entirely, Narratio defaults to seriatim binary + runtime defaults.
|
||||
seriatim: {}
|
||||
|
||||
# Optional runtime overrides. Model/provider can be owned by Audita runtime config.
|
||||
audita:
|
||||
config_path: "/usr/local/etc/audita/config.yml"
|
||||
llm_api_key_env: "AUDITA_LLM_API_KEY"
|
||||
|
||||
# Optional Scriptorium integration for analyze artifacts.
|
||||
scriptorium:
|
||||
config_path: "/usr/local/etc/scriptorium/config.yml"
|
||||
artifacts:
|
||||
session_recap:
|
||||
enabled: true
|
||||
prompt_id: "dnd.session_recap"
|
||||
output_path: "artifacts/session_recap.md"
|
||||
inputs:
|
||||
transcript:
|
||||
source: "trimmed_transcript"
|
||||
required: true
|
||||
previous_recap:
|
||||
source: "previous_session_artifact"
|
||||
artifact: "session_recap"
|
||||
required: false
|
||||
vars:
|
||||
session_id: true
|
||||
session_date: true
|
||||
campaign_name: true
|
||||
previous_session_id: true
|
||||
output_kind: "session_recap"
|
||||
@@ -1,13 +0,0 @@
|
||||
session_id: 2026-05-03
|
||||
campaign: sample-campaign
|
||||
date: 2026-05-03
|
||||
title: Sample Session
|
||||
inputs:
|
||||
audio_dir: ./audio
|
||||
# Optional S3 input alternative. Do not configure with audio_dir/audio_files.
|
||||
# Narratio prepare lists this prefix and downloads .flac files.
|
||||
# audio_s3:
|
||||
# prefix: "audio/"
|
||||
speakers_file: ./speakers.yml
|
||||
autocorrect_file: ./autocorrect.yml
|
||||
glossary_file: ./glossary.yml
|
||||
@@ -1,12 +0,0 @@
|
||||
session_id: "{{ session_id }}"
|
||||
campaign: sample-campaign
|
||||
date: ""
|
||||
title: ""
|
||||
inputs:
|
||||
audio_dir: ./audio
|
||||
# Optional S3 input alternative. Do not configure with audio_dir/audio_files.
|
||||
# audio_s3:
|
||||
# prefix: "audio/{{ session_id }}/"
|
||||
speakers_file: ./speakers.yml
|
||||
autocorrect_file: ./autocorrect.yml
|
||||
glossary_file: ./glossary.yml
|
||||
@@ -866,15 +866,59 @@ func TestValidateMissingAudioSource(t *testing.T) {
|
||||
}
|
||||
|
||||
func TestExamplesLoadAndValidate(t *testing.T) {
|
||||
pipelinePath := filepath.Join("..", "..", "examples", "pipeline.minimal.yml")
|
||||
sessionPath := filepath.Join("..", "..", "examples", "session.minimal.yml")
|
||||
|
||||
cfg, err := Load(pipelinePath, sessionPath)
|
||||
if err != nil {
|
||||
t.Fatalf("Load(examples) error = %v", err)
|
||||
examplesDir := filepath.Join("..", "..", "docs", "examples")
|
||||
tests := []struct {
|
||||
name string
|
||||
pipelineFile string
|
||||
sessionFile string
|
||||
sessionOpts SessionLoadOptions
|
||||
}{
|
||||
{
|
||||
name: "minimal pipeline with local audio session",
|
||||
pipelineFile: "pipeline.minimal.yml",
|
||||
sessionFile: "session.local-audio.yml",
|
||||
},
|
||||
{
|
||||
name: "production pipeline with s3 audio session",
|
||||
pipelineFile: "pipeline.production.yml",
|
||||
sessionFile: "session.s3-audio.yml",
|
||||
},
|
||||
{
|
||||
name: "full annotated pipeline with local audio session",
|
||||
pipelineFile: "pipeline.full.annotated.yml",
|
||||
sessionFile: "session.local-audio.yml",
|
||||
},
|
||||
{
|
||||
name: "template session renders with session_id option",
|
||||
pipelineFile: "pipeline.minimal.yml",
|
||||
sessionFile: "session.template.yml",
|
||||
sessionOpts: SessionLoadOptions{
|
||||
SessionID: "2026-05-03",
|
||||
},
|
||||
},
|
||||
}
|
||||
if err := Validate(cfg); err != nil {
|
||||
t.Fatalf("Validate(examples) error = %v", err)
|
||||
|
||||
for _, tt := range tests {
|
||||
t.Run(tt.name, func(t *testing.T) {
|
||||
pipelinePath := filepath.Join(examplesDir, tt.pipelineFile)
|
||||
sessionPath := filepath.Join(examplesDir, tt.sessionFile)
|
||||
|
||||
var (
|
||||
cfg *Config
|
||||
err error
|
||||
)
|
||||
if strings.TrimSpace(tt.sessionOpts.SessionID) == "" {
|
||||
cfg, err = Load(pipelinePath, sessionPath)
|
||||
} else {
|
||||
cfg, err = LoadWithSessionOptions(pipelinePath, sessionPath, tt.sessionOpts)
|
||||
}
|
||||
if err != nil {
|
||||
t.Fatalf("load example config error = %v", err)
|
||||
}
|
||||
if err := Validate(cfg); err != nil {
|
||||
t.Fatalf("validate example config error = %v", err)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
Reference in New Issue
Block a user