diff --git a/README.md b/README.md index 60edc37..7ecf49d 100644 --- a/README.md +++ b/README.md @@ -1,136 +1,157 @@ # narratio -`narratio` is a Go-based orchestration application for processing D&D session audio into transcripts and downstream artifacts. +`narratio` is a Go orchestration application for processing D&D session audio into transcripts and generated artifacts. -This repository currently contains a **working scaffold** with strict config loading, local workdir/manifest handling, resumable stage control, real WhisperX/Seriatim/Audita adapters, and real `prepare` + `transcribe` + `merge` + `polish` stages. - -## Expected Config Files - -`narratio` expects two YAML files: - -- `pipeline.yml`: pipeline/workspace settings (`workspace`, `storage`, `whisperx`, `seriatim`, `audita`, optional `scriptorium`, `analyzer`, `notification`) -- `session.yml`: per-session settings (`session_id`, `inputs`, optional metadata) - -WhisperX config contract in `pipeline.yml`: - -- required: `whisperx.transcribe_url` -- defaulted when omitted: `whisperx.language` (`en`), `whisperx.timeout` (`30m`), `whisperx.retries` (`3`), `whisperx.retry_delay` (`2s`), `whisperx.concurrency` (`2`) - -Seriatim config contract in `pipeline.yml`: - -- required: `seriatim.binary` (name or path; existence is checked at execution time, not config validation time) -- defaulted when omitted: `seriatim.timeout` (`10m`), `seriatim.output_schema` (`seriatim-intermediate`), `seriatim.coalesce_gap` (`3.0`), `seriatim.report` (`true`) -- allowed `seriatim.output_schema` values: `seriatim-minimal`, `seriatim-intermediate`, `seriatim-full` -- optional tuning: `seriatim.env.*` (`overlap_word_run_gap`, `overlap_word_run_reorder_window`, `backchannel_max_duration`, `filler_max_duration`) must be `> 0` when provided - -Audita config contract in `pipeline.yml`: - -- required: `audita.binary` (name or path; existence is checked at execution time, not config validation time) -- optional: `audita.llm_api_key_env` (environment variable name holding the API key secret; no automatic default) -- defaulted when omitted: `audita.timeout` (`3h`), `audita.modules` (`glossary,homophones,glossary,spoken_word,grammar,homophones,glossary`), `audita.base_url` (`https://openrouter.ai/api/v1`), `audita.model` (`openrouter/google/gemma-4-31b-it`), `audita.llm_concurrency` (`1`), `audita.validation_model` (`""`), `audita.validation_llm_concurrency` (`1`), `audita.report` (`true`) -- allowed `audita.modules` values: `glossary`, `homophones`, `spoken_word`, `grammar` (order and repeats are allowed) -- `audita.base_url` must be a valid URL when provided -- `audita.llm_concurrency` and `audita.validation_llm_concurrency` must be `> 0` - -Audita credentials note: - -- store only the environment variable **name** in config (`audita.llm_api_key_env`), never the API key value itself -- API key values must not be written to pipeline config, generated configs, logs, or manifest metadata -- if `audita.llm_api_key_env` is configured and the named env var is not set (or is empty), Narratio fails before invocation with a redacted error -- if `audita.llm_api_key_env` is omitted/empty, Narratio does not require a credential and omits `AUDITA_LLM_API_KEY` from the subprocess overrides - -Audita runtime note: - -- Narratio currently passes primary LLM concurrency via `AUDITA_LLM_CONCURRENCY` subprocess environment override, not a `--llm-concurrency` flag. - -Scriptorium config contract in `pipeline.yml` (optional until `analyze` is implemented): - -- optional section: `scriptorium` (when omitted, current pipeline behavior is unchanged) -- required when `scriptorium` is present: `scriptorium.binary` (name on PATH or full path) -- optional: `scriptorium.config_path` (if set, it must be non-empty; file existence is validated at execution time, not basic config validation time) -- defaulted when omitted inside present `scriptorium` section: `scriptorium.timeout` (`10m`) -- optional: `scriptorium.render_debug` (`false` by default through YAML zero-value behavior) -- artifact definitions live under `scriptorium.artifacts.` and support multiple configured artifacts -- enabled artifacts require `prompt_id` and `output_path` -- artifact `timeout` values must be valid Go durations when provided -- input definitions live under `scriptorium.artifacts..inputs.` and require non-empty `source` -- `vars` values currently accept booleans and strings only -- Narratio does not store secrets in Scriptorium config; keep credentials in environment management, not YAML - -Scriptorium initial artifact note: - -- this pass does not auto-create `session_recap`; configure `scriptorium.artifacts.session_recap` explicitly in `pipeline.yml` - -`speakers.yml` note: - -- use Seriatim’s documented `match:` format (not the legacy direct mapping style used by older scripts/scaffolds) -- see [`examples/speakers.yml`](examples/speakers.yml) for a concrete `match:` example. - -Prepared inputs note: - -- `prepare` copies session inputs into the session workdir (`work//inputs`), and downstream stages consume those prepared copies, not the original source files. -- If you edit `speakers.yml`, `autocorrect.yml`, `glossary.yml`, or session input paths after `prepare` has run, rerun `prepare` with `--force` before rerunning downstream stages. - -Decoding is strict (`KnownFields(true)`), so unknown YAML fields fail fast. - -Example minimal files are available under `examples/`: - -- `examples/pipeline.minimal.yml` -- `examples/session.minimal.yml` - -## Current Scaffold Status +## Current Implementation Implemented now: -- strict config load + validation -- local artifact/workdir creation and locking -- manifest create/load/save and stage status tracking -- stage framework with real `prepare`, `transcribe`, `merge`, and `polish` stages; placeholder downstream stages -- resumable run control (`run`, `resume`, `run-stage`, `plan` with run/skip decisions) -- real WhisperX HTTP adapter plus real Seriatim/Audita subprocess adapters (with fake/no-op adapters for test/scaffold usage) +- strict config loading/validation (`pipeline.yml` and `session.yml`) +- local workspace/session layout, locking, and manifest persistence +- resumable stage control (`run`, `plan`, `resume`, `run-stage`, `status`) +- real `prepare`, `transcribe`, `merge`, and `polish` stages +- real WhisperX, Seriatim, and Audita adapters +- real Scriptorium subprocess adapter +- real `analyze` stage for `session_recap` +- optional Scriptorium render diagnostics (`render_debug`) Not implemented yet: -- real analyzer integration -- real remote archive/storage backend -- real notification backend +- real `normalize` behavior +- real `archive` behavior +- real `notify` behavior +- additional analyze artifacts beyond `session_recap` +- generic DAG orchestration -## Run Tests +## Config Files + +Narratio expects two YAML files: + +- `pipeline.yml`: pipeline/workspace settings +- `session.yml`: per-session settings + +YAML decoding is strict (`KnownFields(true)`), so unknown fields fail fast. + +## Scriptorium Configuration + +`pipeline.scriptorium` is optional. When present, Narratio validates and uses it for analyze-stage artifact generation. + +Key points: + +- `scriptorium.binary` is required when section is present +- `scriptorium.config_path` is optional +- `scriptorium.timeout` defaults to `10m` when omitted +- `scriptorium.render_debug` enables render diagnostics globally +- artifacts are configured under `scriptorium.artifacts` (map shape supports multiple artifacts) +- enabled artifacts require `prompt_id` and `output_path` +- artifact `render_debug` may override global render setting +- `vars` currently support boolean and string values + +Example `session_recap` artifact definition: + +```yaml +scriptorium: + binary: "scriptorium" + config_path: "/etc/scriptorium/config.yml" + timeout: "10m" + render_debug: false + + artifacts: + session_recap: + enabled: true + prompt_id: "dnd.session_recap" + profile_id: "local-quality" # optional + output_path: "artifacts/session_recap.md" + timeout: "10m" + # render_debug: true # optional per-artifact override + + inputs: + transcript: + source: "processed_transcript" + required: true + + previous_recap: + source: "previous_session_artifact" + artifact: "session_recap" + path: "" # optional; set when available + required: false + + vars: + session_id: true + session_date: true + campaign_name: true + previous_session_id: true + output_kind: "session_recap" +``` + +Prompt IDs and profile IDs are configuration values. They are not hardcoded in analyze-stage logic. + +Do not put secrets in `pipeline.yml`. If API-key behavior is configured, use env var names only. + +## Scriptorium Runtime Behavior + +Narratio integrates with Scriptorium through the public CLI subprocess contract: + +- generation: `scriptorium run` +- diagnostics/testing: `scriptorium render --format json` when `render_debug` is enabled + +For the initial implementation, only `session_recap` generation is supported. + +Analyze-stage session recap behavior: + +- uses processed transcript input (`transcripts/processed.json`) +- optionally includes `previous_recap` when configured and resolvable +- omits optional previous recap when unavailable +- fails if required inputs are missing +- validates output file exists and is non-empty + +Expected session output paths: + +- `artifacts/session_recap.md` +- `logs/scriptorium.session_recap.stdout.log` +- `logs/scriptorium.session_recap.stderr.log` +- `config/scriptorium.session_recap.generated.yml` +- `artifacts/session_recap.render.json` when render diagnostics are enabled + +## Examples + +Starter files: + +- `examples/pipeline.minimal.yml` +- `examples/session.minimal.yml` +- `examples/speakers.yml` + +## Commands + +Run tests: ```bash go test ./... ``` -## Run Plan +Plan a run: ```bash go run ./cmd/narratio plan --config examples/pipeline.minimal.yml --session examples/session.minimal.yml ``` -## Run Pipeline (Current State) - -The current `run` command executes `prepare` + real `transcribe` + real `merge` + real `polish` + placeholder downstream stages and records progress in `manifest.json`. - -Default CLI wiring builds and uses: - -- real WhisperX HTTP adapter from `pipeline.whisperx` -- real Seriatim subprocess adapter from `pipeline.seriatim` -- real Audita subprocess adapter from `pipeline.audita` - -Subprocess runtime note: - -- subprocesses inherit the parent environment by default, then apply Narratio override values (override values win). -- non-zero subprocess errors include stdout/stderr log paths and a short redacted stderr tail when available to speed diagnosis. - -Real `polish` stage output paths: - -- `transcripts/processed.json` -- `artifacts/audita.report.json` (when `audita.report: true`) -- `artifacts/audita-work` -- `logs/audita.stdout.log` -- `logs/audita.stderr.log` -- `config/audita.generated.yml` +Run full pipeline: ```bash go run ./cmd/narratio run --config examples/pipeline.minimal.yml --session examples/session.minimal.yml ``` + +Run analyze only: + +```bash +go run ./cmd/narratio run-stage --config examples/pipeline.minimal.yml --session examples/session.minimal.yml analyze +``` + +## Roadmap + +Near-term roadmap: + +- extend analyze to additional configured artifacts +- support workflows where later artifacts consume earlier generated artifacts +- keep orchestration explicit without a generic DAG engine +- implement archive and notify backends diff --git a/architecture.md b/architecture.md index 7aeaffc..1742cdc 100644 --- a/architecture.md +++ b/architecture.md @@ -2,77 +2,37 @@ ## 1. Purpose -`narratio` is a Go orchestrator for D&D session processing. It coordinates a stage-based pipeline from recorded audio through transcript generation, transcript polishing, and downstream artifact generation, while preserving durable run state for skip/rerun/resume behavior. +`narratio` is a Go orchestrator for D&D session processing. It runs a stage-based local pipeline from audio input through transcript processing and artifact generation, with manifest-based skip/force/resume behavior. -The repository currently implements the orchestration scaffold, local state model, and stage framework, including real WhisperX/Seriatim/Audita adapters plus real `prepare`/`transcribe`/`merge`/`polish` stages. It intentionally does **not** yet implement real Scriptorium-backed artifact generation, remote archive, or notification integrations. +Narratio integrates with Scriptorium through the **public CLI** (`scriptorium run` and `scriptorium render`) via synchronous subprocess execution. -The future `analyze` stage will use `scriptorium` as a subprocess to generate analysis artifacts such as a session recap. The initial Scriptorium-backed artifact will be a **session recap** generated from: +## 2. Current Status -1. the processed/polished transcript, and -2. the previous session recap, when available. +Implemented: -The design should remain modular and composable so later artifact-generation workflows can produce intermediate artifacts, such as structured event logs or state maps, and then use those artifacts as inputs to later outputs such as session recaps, player summaries, or table-read analyses. +- strict `pipeline.yml` + `session.yml` loading with strict YAML field checking (`KnownFields(true)`) +- local workspace/session layout, lock file handling, artifact path helpers, checksums, and atomic writes +- manifest store and stage status transitions for resumable runs +- real `prepare`, `transcribe`, `merge`, and `polish` stages +- real WhisperX HTTP adapter +- real Seriatim subprocess adapter +- real Audita subprocess adapter +- real Scriptorium subprocess adapter +- real `analyze` stage for initial `session_recap` generation +- optional Scriptorium render diagnostics (`render_debug`) before production run -## 2. Design Goals +Still placeholder/future: -- Modular boundaries between orchestration, stages, adapters, storage, config, and manifest. -- Config-driven behavior with strict decoding and conservative validation. -- Durable manifest-based control for resumability and idempotent reruns. -- Boring, explicit control flow; no workflow engine/DAG abstraction for v1. -- Observable orchestration via structured logs plus per-tool log file paths. -- Safe extension points for incremental real-stage implementation. -- Artifact generation should be composable: generated artifacts should be reusable as named inputs to later artifact generation steps. -- Prompt IDs, profile IDs, input mappings, and output destinations should be configuration-driven, not hardcoded in stage logic. +- `normalize` stage behavior +- `archive` stage behavior +- `notify` stage behavior +- additional Scriptorium artifact types beyond `session_recap` +- artifact-to-artifact workflows beyond the initial single-artifact implementation +- generic stale detection based on input/config checksums -## 3. Non-Goals +## 3. Pipeline and Stage Boundaries -Current and near-term non-goals: - -- Reimplementing WhisperX, Seriatim, Audita, or Scriptorium internals. -- Calling Scriptorium internal Go packages. -- Using Scriptorium's HTTP API as the initial integration path. -- Making Scriptorium responsible for Narratio stage state, artifact storage, or notification. -- Building a generic distributed workflow engine. -- Adding a DB dependency for v1. -- Embedding D&D prompt text or prompt-specific business logic in the orchestration core. -- Implementing checksum-based stale detection; this is planned but not implemented. -- Implementing a full general artifact dependency DAG in the first Scriptorium integration pass. - -## 4. Current Implementation Status - -Implemented now: - -- CLI entrypoint and command dispatch (`run`, `plan`, `status`, `resume`, `run-stage`). -- Strict `pipeline.yml`/`session.yml` loading and validation. -- Local workdir and artifact-store abstraction with lock-file support. -- Durable local JSON manifest store with atomic writes. -- Stage interface, canonical stage ordering, and runner main loop. -- Real `prepare` stage: input resolution, materialization, and provenance. -- Real `transcribe` stage: prepared-audio discovery, WhisperX adapter execution, JSON transcript validation, and provenance metadata. -- Real `merge` stage: raw-transcript discovery, Seriatim adapter execution, merged/report JSON validation, and provenance metadata. -- Real `polish` stage: merged-transcript discovery, Audita adapter execution, processed/report JSON validation, and provenance metadata. -- Real WhisperX HTTP adapter: multipart POST, retries, timeout, and atomic output writes. -- Real Seriatim subprocess adapter: deterministic CLI/env construction and output/report JSON validation. -- Real Audita subprocess adapter: deterministic CLI/env construction, credential env handling, and output/report JSON validation. -- Placeholder downstream stages: `normalize`, `analyze`, `archive`, `notify`. -- Adapter interfaces and fake/no-op implementations for external boundaries. -- Reusable subprocess helper and generated YAML/config writing helper. -- Test coverage across config, manifest, artifacts, planning, runner control, and adapters. - -Still planned/future: - -- Real Scriptorium subprocess adapter. -- Real `analyze` stage backed by Scriptorium. -- Initial `session_recap` artifact generation. -- Optional previous-session artifact discovery for prior recap context. -- Future multi-artifact analysis workflows. -- Real remote archive/storage backend, such as S3/SFTP/etc. -- Real notification backend. -- Stale detection based on input/config checksums. - -## 5. High-Level Pipeline - -Canonical stage order is fixed in code: +Canonical stage order: 1. `prepare` 2. `transcribe` @@ -83,846 +43,136 @@ Canonical stage order is fixed in code: 7. `archive` 8. `notify` -Execution status: +Boundary rules: -- `prepare` is implemented with real local filesystem behavior. -- `transcribe` is implemented and validates raw transcript JSON outputs. -- `merge` is implemented and validates merged transcript/report JSON outputs. -- `polish` is implemented and validates processed transcript/report JSON outputs. -- `normalize` remains a placeholder. -- `analyze` remains a placeholder, but is planned to become the Scriptorium-backed artifact-generation stage. -- `archive` and `notify` remain placeholders. +- orchestration logic lives in `internal/app` +- stage business logic lives in `internal/stage` +- external-tool CLI construction lives in adapter packages +- Scriptorium CLI details stay in `internal/adapters/scriptorium` -### Future Analyze Stage Shape +## 4. Scriptorium Integration Model -The `analyze` stage should become the home for generated artifacts. The first implemented artifact should be: +Integration mode: -- `session_recap` +- public CLI subprocesses only (no Scriptorium internal Go packages, no HTTP API) +- production generation uses `scriptorium run` +- diagnostics/testing render uses `scriptorium render --format json` -Initial `session_recap` inputs: +Run invocation shape used by adapter: -- `transcript`: the processed transcript at `transcripts/processed.json` -- `previous_recap`: the previous session's recap, if available - -Future artifacts may include: - -- structured event log -- final state map -- glossary suggestions -- player-facing summary -- table-read analysis -- meta-analysis - -Future artifact workflows may be sequential or dependency-aware. For example: - -```text -processed transcript - ↓ -structured event log - ↓ -session recap - ↓ -player-facing summary +```bash +scriptorium run --prompt --input name=path --out ``` -Do not implement a general DAG engine initially. The design should, however, avoid hardcoding assumptions that would make later artifact dependencies difficult. +Optional flags passed when configured: -6. CLI Commands +- `--config ` +- `--profile ` +- repeated `--var name=value` +- repeated `--input name=path` +- `--timeout ` +- `--api-key-env ` when configured -cmd/narratio/main.go is intentionally thin and delegates to internal/app. +Render invocation shape used by adapter: -Current command behavior: +```bash +scriptorium render --prompt --input name=path --format json --out +``` + +Adapter behavior: + +- always passes `--out` +- captures stdout/stderr separately +- writes generated invocation metadata YAML (redacted, no secrets) +- treats exit code `0` as success +- treats exit code `1` as failure +- treats exit code `2` as failure with `validation_failed=true` and preserves output metadata when available +- validates successful output files exist and are non-empty +- does not treat non-empty stderr as failure by itself + +## 5. Configuration Contract + +`pipeline.scriptorium` is optional. Existing pipelines without Scriptorium continue to work. + +When `pipeline.scriptorium` is present: + +- `binary` is required and non-empty +- `config_path` is optional; when provided it must be non-empty +- `timeout` is optional; when provided it must parse as a Go duration +- default `timeout` is `10m` +- unknown YAML fields fail strict decode + +Artifacts are configured as a map under `pipeline.scriptorium.artifacts` so multiple artifacts are possible in the config shape. + +For each artifact definition: + +- `enabled: true` requires non-empty `prompt_id` +- `enabled: true` requires non-empty `output_path` +- `timeout` must parse as Go duration when present +- optional per-artifact `render_debug` may override global `scriptorium.render_debug` +- `inputs` are named and each input requires non-empty `source` +- inputs may be optional (`required: false`) +- `vars` values currently support `string` and `bool` + +Prompt IDs and profile IDs are configuration values, not hardcoded stage logic. + +## 6. Analyze Stage (Current Implementation) + +The current real analyze implementation supports only `scriptorium.artifacts.session_recap`. + +Behavior: + +- if `pipeline.scriptorium` is missing, analyze returns a skipped result with metadata +- if no Scriptorium artifacts are enabled, analyze returns a skipped result with metadata +- if enabled artifacts exist but `session_recap` is not enabled, analyze fails clearly +- processed transcript input is resolved from manifest (`polish` output kind `transcript_processed`) when available, otherwise fallback path `work//transcripts/processed.json` +- processed transcript is validated as JSON with top-level `segments` array +- configured inputs are resolved by source +- optional `previous_recap` is omitted when unavailable +- required `previous_recap` fails before invocation when unavailable +- vars are built from config + session metadata +- `render_debug` controls pre-run `scriptorium render` diagnostics +- render failure stops stage before production run +- render output is validated as JSON +- production call uses Scriptorium adapter `RunArtifact` +- successful run output must exist and be non-empty +- manifest records output refs, logs, generated config paths, and non-secret provenance metadata + +## 7. Session Recap Paths + +Current expected paths for `session_recap`: + +- artifact output: `artifacts/session_recap.md` +- run stdout log: `logs/scriptorium.session_recap.stdout.log` +- run stderr log: `logs/scriptorium.session_recap.stderr.log` +- run generated invocation/config: `config/scriptorium.session_recap.generated.yml` +- render output (when enabled): `artifacts/session_recap.render.json` +- render stdout log: `logs/scriptorium.session_recap.render.stdout.log` +- render stderr log: `logs/scriptorium.session_recap.render.stderr.log` +- render generated invocation/config: `config/scriptorium.session_recap.render.generated.yml` + +## 8. Security and Privacy + +- do not store secrets in pipeline YAML, generated invocation YAML, logs, or manifest metadata +- if API-key integration is configured, pass env var names only (never raw key values) +- avoid logging transcript content or rendered prompt content by default +- treat generated artifacts and logs as potentially sensitive session material + +## 9. Roadmap + +Planned next steps: + +- extend analyze beyond `session_recap` to additional configured artifacts +- support artifact inputs that consume prior generated artifacts +- keep this composable without adding a generic DAG engine in the near term +- implement real `archive` backend behavior +- implement real `notify` backend behavior +- add checksum-based stale detection and stale transitions + +Architectural invariants remain: -run --config --session [--force] -Loads and validates config. -Ensures workdir. -Acquires lock. -Loads or creates manifest. -Executes full stage plan. -Skips already-succeeded stages unless --force. -plan --config --session [--force] -Loads and validates config. -Ensures workdir. -Loads manifest if present. -Prints run/skip action per stage and totals. -resume --config --session [--force] -Resumes from first non-succeeded stage based on manifest. -Runs full pipeline when forced. -run-stage --config --session [--force] -Executes exactly one named stage. -Unknown stage is an error. -status --manifest -Loads manifest and prints session ID, updated timestamp, and stage statuses. - -Invalid usage returns non-zero with usage/error text. - -Future Scriptorium work should not require new top-level CLI commands. scriptorium integration should be exercised through: - -narratio run-stage --config pipeline.yml --session session.yml analyze - -and, eventually: - -narratio run --config pipeline.yml --session session.yml - -7. Configuration - -Package: internal/config - -Files: - -pipeline.yml -session.yml - -Key behavior: - -Strict YAML decoding via yaml.Decoder.KnownFields(true). -Unknown fields are rejected. -Combined resolved config type keeps source paths (PipelinePath, SessionPath) for provenance/errors. -Optional fields are defaulted during load when deterministic resolved values are useful. -WhisperX Config - -Current keys: - -pipeline.whisperx.transcribe_url required -pipeline.whisperx.language default: en -pipeline.whisperx.timeout default: 30m -pipeline.whisperx.retries default: 3 -pipeline.whisperx.retry_delay default: 2s -pipeline.whisperx.concurrency default: 2 -Seriatim Config - -Current keys: - -pipeline.seriatim.binary required -pipeline.seriatim.timeout default: 10m -pipeline.seriatim.output_schema default: seriatim-intermediate -allowed: seriatim-minimal|seriatim-intermediate|seriatim-full -pipeline.seriatim.coalesce_gap default: 3.0 -must be >= 0 -pipeline.seriatim.report default: true -optional tuning under pipeline.seriatim.env.* -must be > 0 when provided -Audita Config - -Current keys: - -pipeline.audita.binary required -pipeline.audita.timeout default: 3h -pipeline.audita.llm_api_key_env optional; no automatic default -pipeline.audita.modules -default sequence: glossary,homophones,glossary,spoken_word,grammar,homophones,glossary -pipeline.audita.base_url default: https://openrouter.ai/api/v1 -pipeline.audita.model default: openrouter/google/gemma-4-31b-it -pipeline.audita.llm_concurrency default: 1 -must be > 0 -pipeline.audita.validation_model default: empty string -pipeline.audita.validation_llm_concurrency default: 1 -must be > 0 -pipeline.audita.report default: true - -Audita secret-handling policy: - -llm_api_key_env stores only the environment variable name. -API key values are read from the process environment at runtime. -API key values are not stored in pipeline.yml, manifest metadata, generated configs, or logs. -If llm_api_key_env is configured and the named env var is missing/empty, the Audita adapter fails before invocation with a redacted error. -If llm_api_key_env is empty/omitted, the Audita adapter omits AUDITA_LLM_API_KEY from subprocess env overrides and continues. -Planned Scriptorium Config - -The future Scriptorium integration should add a top-level pipeline.scriptorium config section. - -Illustrative shape: - -scriptorium: - binary: "scriptorium" - config_path: "/etc/scriptorium/config.yml" - timeout: "10m" - render_debug: false - - artifacts: - session_recap: - enabled: true - prompt_id: "dnd.session_recap" - profile_id: "local-quality" - output_path: "artifacts/session_recap.md" - timeout: "10m" - - inputs: - transcript: - source: "processed_transcript" - required: true - - previous_recap: - source: "previous_session_artifact" - artifact: "session_recap" - required: false - - vars: - session_id: true - session_date: true - campaign_name: true - previous_session_id: true - output_kind: "session_recap" - -This shape is illustrative, not yet implemented. The important design points are: - -scriptorium.binary should accept either scriptorium on PATH or a full path. -scriptorium.config_path should be optional. -When omitted, Scriptorium may use its own default config behavior. -When provided, Narratio should pass --config . -Prompt IDs are configuration values, not hardcoded in the analyze stage. -Profile IDs are optional configuration values. -When omitted, Scriptorium prompt defaults may apply. -Artifact definitions should map: -artifact name -prompt ID -optional profile ID -named inputs -small metadata variables -output destination -timeout -optional render-debug behavior -Input names must match the Scriptorium prompt definition. -Large content should be passed as input files, not as --var values. -API keys should never be passed directly on the command line. -Runtime overrides such as --llm-base-url, --model, --api-key-env, --temperature, --max-tokens, --top-p, and --timeout should be used only when explicitly configured. -Validation Scope - -Validation currently enforces the implemented config sections listed above. - -Future Scriptorium validation should enforce: - -scriptorium.binary is required for real analyze. -configured artifact names are unique. -enabled artifacts have a non-empty prompt_id. -output paths are session-workdir-relative or otherwise safe. -input names are non-empty. -required configured inputs can be resolved before execution. -optional inputs may be omitted without failing the stage. -explicit config_path, when provided, exists and is a regular file. -timeout values parse as Go durations. -unknown YAML fields continue to fail strict decoding. - -Do not validate Scriptorium prompt library internals inside Narratio. Scriptorium owns prompt/profile/schema loading and validation. - -8. Manifest and Stage Status - -Package: internal/manifest - -Model includes: - -Manifest -StageRecord -ArtifactRecord -InputRecord -ErrorRecord -StageStatus - -Supported stage statuses: - -pending -running -succeeded -failed -skipped -stale -defined but not actively produced by runner yet -interrupted -defined for future use - -Store behavior: - -Local JSON store with create/load/save. -Save is atomic via same-directory temp file + rename. -updated_at refreshed on save. - -Helpers implemented: - -MarkStageRunning -MarkStageSucceeded -MarkStageFailed -MarkStageSkipped - -Runner currently records running/succeeded/failed transitions and preserves existing succeeded stage metadata when skipping. - -Manifest versioning note: - -Current manifest model includes optional pipeline_version, but does not include an explicit manifest schema/version field. -Before manifests become long-term compatibility-sensitive across releases/storage backends, add explicit manifest schema/versioning and migration policy. -Future Artifact Records for Scriptorium - -The Scriptorium-backed analyze stage should record each generated artifact with enough metadata to support rerun, inspection, and future artifact dependency resolution. - -For each artifact, record non-secret metadata such as: - -artifact name, such as session_recap -prompt ID -profile ID, if explicitly configured -Scriptorium config path, if provided -input artifact paths -optional/missing input decisions -output path -render-debug path, if generated -stdout/stderr log paths -exit code -duration -timeout -Scriptorium command mode: run or render -validation-failed exit status when relevant -generated artifact path even if Scriptorium exits 2 and the output exists - -Do not record rendered prompt content by default. Transcripts and prompts may contain sensitive content. - -9. Workdir and Artifact Model - -Package: internal/artifacts - -Canonical local paths are resolved by SessionPaths under: - -{workspace.root}/work/{session_id}/ - -Subpaths: - -inputs/ -audio/ -transcripts/raw/ -transcripts/normalized/ -artifacts/ -config/ -logs/ -manifest.json -.lock - -Implemented store capabilities: - -Ensure layout. -Resolve session paths. -Copy inputs. -Existence checks. -SHA-256 checksums. -Atomic file writes. -Session lock acquire/release via .lock, conflict-safe through O_EXCL. - -Ref supports local path info plus optional future remote key/checksum metadata. - -Future Scriptorium Artifact Paths - -The initial Scriptorium-backed artifact should use: - -artifacts/session_recap.md -logs/scriptorium.session_recap.stdout.log -logs/scriptorium.session_recap.stderr.log -config/scriptorium.session_recap.generated.yml - -If render debugging is enabled: - -artifacts/session_recap.render.json - -or another clearly named diagnostics path under artifacts/ or logs/. - -Future artifacts should use stable, predictable paths, for example: - -artifacts/event_log.json -artifacts/final_state_map.json -artifacts/player_summary.md -artifacts/table_read.md -artifacts/meta_analysis.md - -Names should be driven by artifact configuration rather than hardcoded in general analyzer logic. - -10. Stage Architecture - -Package: internal/stage - -Core contracts: - -Stage interface with Name, Declares, and Run(ctx, env, manifest). -IODecl for declared input/output artifact intent. -StageResult for outputs, logs, generated configs, and metadata. - -Current Declares role: - -Stage.Declares is currently contract metadata. -The runner does not yet enforce declared inputs/outputs at execution time. -Real-stage implementation work should strengthen enforcement by validating declared inputs before execution and declared outputs after execution. -Current Stage Behaviors -prepare real -Ensures layout. -Resolves session-relative input paths. -Materializes required input files into inputs/. -Writes pipeline.resolved.yml. -Resolves/copies .flac audio inputs from audio_dir or audio_files. -Computes checksums and records deterministic manifest.inputs. -Uses checksum-aware write/copy reuse for idempotency. -transcribe real -Discovers prepared .flac audio inputs from manifest.inputs with kind=audio or work/.../audio fallback. -Derives per-speaker output files at transcripts/raw/{audio_basename}.json. -Calls whisperx.Client with bounded parallelism from pipeline.whisperx.concurrency. -Validates each output file exists and is valid JSON before stage success. -Records output refs plus stage metadata. -merge real -Discovers raw transcript inputs from manifest.stages.transcribe.outputs with kind=transcript_raw or work/.../transcripts/raw fallback. -Validates input transcript files as existing regular JSON files. -Uses prepared inputs/speakers.yml and inputs/autocorrect.yml directly. -Does not translate speaker-map format. -Invokes seriatim.Runner using canonical merged/report/log/generated-config paths under the session workdir. -Validates merged transcript output JSON and report JSON before stage success. -Records merged/report output refs plus stage metadata. -polish real -Discovers merged transcript input from manifest.stages.merge.outputs with kind=transcript_merged or fallback work/.../transcripts/merged.json. -Validates merged transcript as JSON. -Requires prepared inputs/glossary.yml. -Invokes audita.Runner using canonical paths: -transcripts/processed.json -artifacts/audita.report.json -artifacts/audita-work -logs/audita.stdout.log -logs/audita.stderr.log -config/audita.generated.yml -Validates processed transcript output JSON with required top-level segments array. -Validates report JSON when enabled. -Records processed/report outputs plus non-secret provenance metadata. -normalize placeholder - -normalize remains a placeholder. Audita and Seriatim both perform some normalization-like behavior internally. A separate real normalize stage should be implemented only if Narratio develops a clear stage-specific transformation that belongs outside Seriatim and Audita. - -analyze planned - -analyze will become the Scriptorium-backed artifact generation stage. - -Initial behavior should be narrowly scoped: - -Generate a session_recap artifact. -Consume transcripts/processed.json. -Include the previous session recap as previous_recap input when configured and available. -Invoke scriptorium run. -Write output to artifacts/session_recap.md. -Capture stdout/stderr logs. -Generate a redacted invocation/config file for provenance. -Validate that the output artifact file exists and is non-empty. -Record artifact refs and non-secret provenance in manifest. - -Future behavior should generalize artifact generation: - -multiple configured artifact definitions -named inputs -optional inputs -artifact-to-artifact dependencies -render diagnostics for testing/debugging -structured artifacts such as JSON event logs -sequential artifact generation where one artifact feeds another - -Do not implement a generic DAG engine in the first pass. The first Scriptorium integration should be an intentionally small, synchronous subprocess implementation. - -archive placeholder - -Future remote artifact persistence. Storage backends may include local filesystem, S3-compatible object storage, SFTP, or similar. - -notify placeholder - -Future notification backend, such as email or chat notification, once pipeline execution completes. - -11. Application Environment and Adapters - -internal/app uses stage.Env as the shared dependency container to avoid duplicate environment definitions. - -Contained dependencies: - -Resolved config -Artifact store -Manifest store -Logger -Adapter interfaces - -Adapter boundaries currently include: - -whisperx.Client -seriatim.Runner -audita.Runner -analyzer.Runner -storage.Backend -notify.Sender - -All adapters currently have fake/no-op implementations for tests/scaffold execution. WhisperX, Seriatim, and Audita also have real implementations. - -Future Scriptorium Adapter Boundary - -The future analyzer integration should be implemented as a Scriptorium subprocess adapter. - -The existing analyzer.Runner may either: - -become a Scriptorium-backed implementation, or -be replaced/refined with a more explicit scriptorium.Runner adapter package. - -Either approach is acceptable if the boundary remains clear: - -internal/stage/analyze owns artifact-stage behavior. -internal/adapters/scriptorium or equivalent owns Scriptorium CLI construction and subprocess behavior. -internal/app wires the real adapter into stage.Env. -Tests can inject fake artifact-generation adapters. - -The Scriptorium adapter should support at least: - -RunArtifact(ctx, request) (result, error) -optional future RenderArtifact(ctx, request) (result, error) - -Request fields should include: - -binary path -optional config path -prompt ID -optional profile ID -named input paths -variable key/value pairs -output path -stdout/stderr log paths -generated config/invocation path -timeout -optional runtime overrides -API key env var name(s), if configured - -Result fields should include: - -output path -stdout/stderr log paths -generated config path -exit code -duration -command mode -prompt ID -profile ID -validation-failed indicator for exit code 2 -non-secret metadata -Subprocess Helper - -internal/adapters/subprocess provides reusable subprocess scaffolding: - -Context cancellation + optional timeout. -Explicit executable/args, working dir, env overrides. -Parent environment inheritance with override merge semantics. -Stdout/stderr log file handling. -Shared-stream guard when both logs target the same path. -Exit code and timing capture. -Actionable error wrapping, including stdout/stderr log paths and a short redacted stderr tail when available. -Atomic file/YAML writers for generated config/log scaffolding. - -The future Scriptorium adapter should reuse this helper. - -12. Scriptorium Integration Design -Integration Mode - -Initial integration is synchronous subprocess execution through the public CLI. - -Use: - -scriptorium run - -for production generation. - -Use: - -scriptorium render - -for debugging, dry-runs, and tests that validate command construction without LLM execution. - -Do not call Scriptorium internal Go packages. Do not use Scriptorium's HTTP API as the initial integration path. - -Production Invocation Shape - -The initial session recap invocation should follow this shape: - -scriptorium run \ - --prompt \ - --input transcript= \ - --input previous_recap= \ - --out - -When no previous recap is available, omit that optional input rather than passing an empty path: - -scriptorium run \ - --prompt \ - --input transcript= \ - --out - -Common optional flags: - ---config ---profile ---var name=value, repeatable ---input name=path, repeatable ---timeout -runtime model override flags only when explicitly configured -Render Invocation Shape - -For debug/test rendering: - -scriptorium render \ - --prompt \ - --input transcript= \ - --format json \ - --out - -render does not call the LLM and should not be treated as artifact generation. - -Inputs - -Inputs must be passed as repeated: - ---input name=path - -Rules: - -name must match the Scriptorium prompt definition input name. -Prefer absolute paths, or paths controlled by Narratio's session workdir. -The processed transcript is the primary transcript input. -Optional inputs may include previous recap, glossary, campaign notes, event logs, final state maps, or other generated artifacts. -Scriptorium reads files directly; Narratio should not inline large input content. -Variables - -Small metadata should be passed as repeated: - ---var name=value - -Examples: - -session_id -session_date -campaign_name -previous_session_id -output_kind - -Large content belongs in input files, not --var. - -Prompt IDs and Profiles - -Prompt IDs are configuration values. - -Narratio should not hardcode prompt IDs such as dnd.session_recap in stage logic. Prompt IDs should come from pipeline.yml artifact definitions. - -Profiles: - -May be omitted to use the prompt's default_profile. -May be provided to force a profile such as local-fast, local-quality, frontier, batch, or test. -Should be configuration values. -Runtime Overrides - -Scriptorium supports runtime override flags such as: - ---llm-base-url ---model ---api-key-env ---temperature ---max-tokens ---top-p ---timeout - -Guidance: - -Keep normal model/runtime settings in Scriptorium execution profiles. -Use runtime overrides only for explicit operator-directed exceptions or tests. -Never pass raw API keys on the command line. ---api-key-env names an environment variable; Narratio should ensure that variable is set in the subprocess environment when configured. -Output Handling - -For scriptorium run: - -Narratio should always pass --out. -Treat the output path as the generated artifact. -Capture stdout/stderr separately. -Non-empty stderr alone does not imply failure. - -For scriptorium render: - -Use --out for diagnostics. -Use --format json when tests need structured render output. -Exit Status - -Scriptorium exit status behavior: - -0: success. -1: runtime, parse, config, load, render, generation, or I/O error. -2: run completed but output validation failed. - -Narratio should treat non-zero exit codes as failed stage execution. - -However, when exit code is 2, Scriptorium may have already written an output artifact. Narratio may record that output path in failure metadata, but should not mark the artifact as successfully generated. - -13. Run Control, Locking, Logging, and Long-Running Stages -Run Control - -internal/app runner is sequential and manifest-driven: - -Loads/creates manifest. -Computes per-stage action: run vs skip. -Skip rule today: succeeded + not forced => skip. ---force reruns previously succeeded stages. -resume starts at first non-succeeded stage. - -Stale detection is intentionally TODO pending checksum-based invalidation logic. - -Locking - -run/resume/run-stage execution paths acquire a session lock via artifact store and release it with defer. - -Logging -Structured logger is initialized via internal/logging using slog. -Runner emits concise stage lifecycle logs. -Real WhisperX/Seriatim/Audita stages already use explicit output/log/config paths. -Future Scriptorium stage should use the same pattern: -logs/scriptorium..stdout.log -logs/scriptorium..stderr.log -config/scriptorium..generated.yml -Long-Running Stage Expectations - -The architecture expects long-running stages. - -WhisperX, Seriatim, and Audita already use context timeout/cancellation in their adapters. The future Scriptorium adapter should also use context timeout/cancellation through the shared subprocess helper. - -14. Previous Session Context - -The first Scriptorium-backed artifact, session_recap, may optionally consume the previous session recap. - -Narratio should support this as an optional configured input. - -Initial implementation may use a simple local path or session config reference. Future implementations may resolve previous-session artifacts from: - -current workspace -prior session workdir -manifest records -archive/storage backend -campaign/session index - -The important rule: missing optional previous recap should not fail the initial session_recap artifact generation. - -If previous recap is configured as required and cannot be resolved, analyze should fail clearly before invoking Scriptorium. - -Do not implement a full campaign history index in the initial Scriptorium pass. - -15. Testing Strategy - -Current tests verify scaffold behavior without real external services: - -Strict config load/validate behavior and error quality. -Manifest store round trips and transition semantics. -Workdir layout, atomic writes, checksums, and lock behavior. -Plan order and stage selection. -Run/skip/force/resume/run-stage control behavior. -Prepare-stage input materialization/provenance/idempotency. -Real transcribe-stage audio discovery/concurrency/failure handling/output validation/provenance. -Real merge-stage input discovery/validation, Seriatim adapter failure handling, merged/report output validation, and provenance recording. -Real polish-stage merged-input discovery/validation, Audita adapter failure handling, processed/report output validation, and provenance recording. -Adapter fake behavior and error propagation. -WhisperX HTTP adapter behavior. -Seriatim subprocess adapter behavior. -Audita subprocess adapter behavior. -Subprocess helper behavior. - -Future Scriptorium tests should avoid real LLM calls. - -Recommended Scriptorium testing strategy: - -Use fake Scriptorium adapters for stage tests. -Use helper subprocesses or scripts for adapter tests. -Use scriptorium render --format json where integration-style tests need to validate command construction without LLM execution. -Verify command construction: -run -render ---prompt -repeated --input -repeated --var ---out -optional --config -optional --profile -Verify missing required input behavior. -Verify optional previous recap behavior. -Verify output file creation. -Verify exit code 1 handling. -Verify exit code 2 handling where output may exist but validation failed. -Verify stderr capture. -Avoid real API keys in tests. -Do not log full rendered prompts by default. - -Tests intentionally avoid hardcoding arbitrary operational default values unless those defaults are part of Narratio's documented config contract. - -16. Security and Privacy - -General rules: - -Never pass raw API keys on the command line. -Do not log full environment dumps. -Do not store secret values in generated configs, manifest metadata, or logs. -Treat transcripts, prompts, and generated artifacts as potentially sensitive. -Avoid logging rendered prompts by default. -Avoid logging transcript content by default. -Use session-scoped output paths. -Preserve stdout/stderr logs, but do not assume stderr means failure for tools like Scriptorium. -Runtime api_key_env values should name environment variables, not contain secrets. - -For Scriptorium specifically: - -Pass --api-key-env only when configured. -Ensure the named API-key env var is present in the subprocess environment when required. -Do not record API-key values. -Generated invocation/config files should include env var names but never secret values. -17. Future Work / Implementation Roadmap - -Recommended implementation sequence: - -Update architecture/config contracts for planned Scriptorium integration. -Add Scriptorium configuration contract. -Implement Scriptorium subprocess adapter. -Implement initial real analyze behavior for session_recap. -Support optional previous-session recap input. -Add render-debug support for Scriptorium artifact definitions. -Extend analyze to support multiple configured artifact definitions. -Add artifact-to-artifact input support for future intermediate artifacts. -Implement real archive remote backend behavior. -Implement real notify backend. -Add checksum-based stale detection and stale status transitions. -Add selective parallelism where architecturally safe. - -Each step must preserve existing package boundaries and manifest-based control flow. - -Real Stage Implementation Checklist - -For each stage moved from placeholder to real behavior: - -Implement and/or use the appropriate adapter behavior for that stage boundary. -Generate and use config/log paths needed by the adapter/tool invocation. -Validate expected stage inputs before execution. -Write outputs atomically where practical. -Validate expected outputs after execution before marking success. -Record outputs, logs, generated configs, and provenance metadata in manifest stage records. -Preserve skip/force/resume semantics. -Add fake-adapter tests and at least one explicit failure-path test. -Scriptorium Artifact Implementation Checklist - -For each configured Scriptorium artifact: - -Resolve all required inputs. -Omit missing optional inputs. -Build deterministic scriptorium run args. -Use absolute or session-workdir-controlled paths. -Pass --out. -Capture stdout/stderr separately. -Generate redacted invocation/config metadata. -Validate output file exists and is non-empty. -Record prompt ID, profile ID, input paths, vars, output path, exit code, duration, and logs. -Treat exit code 2 as failure while preserving diagnostic metadata. -Avoid logging prompt content or transcript content. -18. Architectural Invariants - -The following invariants are mandatory for future changes: - -Stage orchestration logic lives in internal/app; stage business logic lives in internal/stage. -External tool details remain behind adapter interfaces. -Config loading is strict and validation-first. -Manifest is the source of truth for stage status and resume/skip decisions. -Stage outputs/provenance are recorded durably and deterministically. -Local workdir is the primary execution area; remote storage is an adapter concern. -Main entrypoint stays thin; cmd/narratio delegates to internal/app. -Pipeline behavior remains explicit and maintainable; no generic DAG engine introduction. -Every stage must consume declared artifacts, produce declared artifacts, validate them, record provenance, and remain safely skippable/rerunnable. -Artifact generation must remain configuration-driven; prompt IDs, profile IDs, and artifact input mappings should not be hardcoded in the orchestration core. -Scriptorium integration must use the public CLI subprocess contract until a deliberate future architecture change is made. -Secret values must never be stored in configs, generated invocation files, logs, or manifest metadata. \ No newline at end of file +- strict config decoding/validation +- manifest-driven run control +- clear stage/adapter separation +- configuration-driven prompt/profile/input/vars/output mapping +- Scriptorium integration through public CLI subprocess contract diff --git a/docs/integrations/scriptorium.md b/docs/integrations/scriptorium.md index c7d6bfb..dc922eb 100644 --- a/docs/integrations/scriptorium.md +++ b/docs/integrations/scriptorium.md @@ -217,23 +217,40 @@ Narratio should treat non-zero exit codes as failed stage execution, but may rec ## 17. Suggested Narratio Configuration Shape -Illustrative (not required schema): +Illustrative `pipeline.yml` shape: ```yaml scriptorium: + binary: scriptorium config_path: /etc/scriptorium/config.yml - stages: + timeout: 10m + render_debug: false + artifacts: session_recap: + enabled: true prompt_id: dnd.session_recap profile_id: local-quality # optional - inputs: [transcript, glossary, previous_recap] - vars: [session_id, session_date, campaign_name] - output_path_template: artifacts/{session_id}/session_recap.md - timeout: 2m - render_debug: false + output_path: artifacts/session_recap.md + timeout: 10m + render_debug: false # optional artifact override + inputs: + transcript: + source: processed_transcript + required: true + previous_recap: + source: previous_session_artifact + artifact: session_recap + path: "" # optional + required: false + vars: + session_id: true + session_date: true + campaign_name: true + previous_session_id: true + output_kind: session_recap ``` -The key idea: map Narratio stage/artifact names to prompt ID, optional profile, expected inputs, and output destination. +The key idea: map Narratio artifact names to prompt ID, optional profile, expected inputs, vars, and output destination. ## 18. Testing Strategy for Narratio Integration @@ -317,6 +334,6 @@ Possible later extensions: - HTTP API integration - S3 artifact references if Scriptorium adds S3 reader support -- storing render diagnostics alongside generated artifacts +- richer render diagnostics and policy controls - token budgeting/prompt-size checks - batch execution if Scriptorium later adds batch support diff --git a/examples/pipeline.minimal.yml b/examples/pipeline.minimal.yml index 1e34697..769cbcd 100644 --- a/examples/pipeline.minimal.yml +++ b/examples/pipeline.minimal.yml @@ -55,6 +55,8 @@ scriptorium: profile_id: "local-quality" output_path: "artifacts/session_recap.md" timeout: "10m" + # Optional per-artifact override of global scriptorium.render_debug. + # render_debug: true inputs: transcript: source: "processed_transcript" @@ -62,6 +64,7 @@ scriptorium: previous_recap: source: "previous_session_artifact" artifact: "session_recap" + # Optional: set when previous recap is available. path: "" required: false vars: