diff --git a/README.md b/README.md index b8775d4..6791611 100644 --- a/README.md +++ b/README.md @@ -1,467 +1,22 @@ # narratio -`narratio` is a Go orchestration application for processing D&D session audio into transcripts and generated artifacts. +Narratio is a Go orchestration application that turns D&D session audio into polished transcripts and generated session artifacts. -## Current Implementation - -Implemented now: - -- strict config loading/validation (`pipeline.yml` and `session.yml`) -- local workspace/session layout, locking, and manifest persistence -- resumable stage control (`run`, `plan`, `resume`, `run-stage`, `status`) -- real `prepare`, `transcribe`, `merge`, `polish`, `normalize`, `trim`, and `analyze` stages -- real WhisperX, Seriatim, and Audita adapters -- real Scriptorium subprocess adapter -- optional Scriptorium render diagnostics (`render_debug`) - -Not implemented yet: - -- `notify` stage behavior -- additional analyze artifacts beyond `session_recap` -- generic DAG orchestration - -## Config Files - -Narratio expects two YAML files: - -- `pipeline.yml`: pipeline/workspace settings -- `session.yml`: per-session settings - -Pipeline config lookup for CLI commands: - -- if `--config ` is provided, Narratio uses that path -- if `--config` is omitted, Narratio searches in this order: - - `/usr/local/etc/narratio/pipeline.yml` - - `/etc/narratio/pipeline.yml` - -Session config lookup for CLI commands: - -- if `--session ` is provided, Narratio uses that path -- if `--session` is omitted, Narratio searches in this order: - - `./session.yml` - - `/usr/local/etc/narratio/session.yml` - - `/etc/narratio/session.yml` - -Session template support: - -- Narratio renders `session.yml` templates before strict YAML decode. -- `--session-id ` provides the `session_id` template variable. -- Supported placeholder forms: - - `{{session_id}}` - - `{{ session_id }}` -- unresolved template placeholders fail with a clear error. -- strict YAML validation still runs after rendering. -- concrete `session.yml` files without templates remain fully supported. - -Optional secrets-from-files config: - -- `pipeline.secrets.env_dir` may point to a directory of secret files -- each top-level file with an env-var-style name is loaded as an environment variable: - - file name = env var name - - file contents = env var value (trailing newline/CRLF trimmed) -- process environment wins: existing env vars are not overwritten -- if configured, Narratio fails fast when `env_dir` is missing/unreadable -- relative `env_dir` values resolve from Narratio’s current working directory - -YAML decoding is strict (`KnownFields(true)`), so unknown fields fail fast. - -Maintainer note: application defaults are centralized in [`internal/config/defaults.go`](internal/config/defaults.go). - -## Storage And Archive Foundations - -Narratio now includes configuration and path-model foundations for archive support, plus implemented prepare-stage S3 audio input. - -Implemented foundations: - -- `pipeline.storage.s3` config shape (`bucket`, `root_prefix`, `region`, `endpoint`, `force_path_style`, `access_key_id_env`, `secret_access_key_env`) -- `pipeline.spool` config shape (`root`, `delete_audio_after_archive`) -- `pipeline.archive` config shape (`enabled`, `upload_run`, `promote_artifacts`) -- promotion-rule validation (`from`/`to` required, relative-only paths, traversal rejected) -- `session.campaign` requirement for campaign-aware path construction -- optional `session.inputs.audio_s3.prefix` modeling and prepare-stage S3 audio download -- run ID generation and S3/local path helper foundations -- manifest run/path identity fields - -Current defaults: - -- `pipeline.storage.s3.root_prefix`: `dnd` -- `pipeline.storage.s3.access_key_id_env`: `OBJECT_STORAGE_KEY_ID` -- `pipeline.storage.s3.secret_access_key_env`: `OBJECT_STORAGE_KEY` -- `pipeline.workspace.cleanup_after_archive`: `false` -- `pipeline.spool.root`: `/var/spool/narratio` -- `pipeline.spool.delete_audio_after_archive`: `false` -- `pipeline.archive.enabled`: `true` -- `pipeline.archive.upload_run`: `true` -- default `pipeline.archive.promote_artifacts`: - - `transcripts/trimmed.json` -> `transcripts/trimmed.json` (`required: true`) - - `artifacts/session_recap.md` -> `artifacts/session_recap.md` (`required: true`) - -Current boundaries: - -- local development audio (`audio_dir` / `audio_files`) still works -- `audio_dir`/`audio_files` and `audio_s3` are mutually exclusive -- real S3-compatible backend now exists in the storage adapter package -- storage backend tests use fake storage and do not require live S3 -- archive uploads successful run records under `runs/{run_id}/` -- archive does not upload local audio by default -- archive uploads promoted outputs to session-level keys using `archive.promote_artifacts` -- archive uploads `current/manifest.json` -- archive uploads `current/run_id.txt` last as the effective commit marker -- required missing promotions fail archive -- optional missing promotions are skipped and recorded -- cleanup remains conservative and opt-in: - - `pipeline.spool.delete_audio_after_archive: true` removes only the run-scoped spool audio directory after successful archive commit - - `pipeline.workspace.cleanup_after_archive: true` removes only the run-scoped local workdir after successful archive commit - - cleanup executes only after all selected stages for the command invocation succeed - - cleanup does not run for failed, incomplete, skipped, or unarchived runs -- local development `audio_dir`/`audio_files` source inputs are never deleted by spool cleanup -- S3 credentials are resolved from configured env-var names when both are present; if either is missing, Narratio falls back to the AWS SDK default credential chain - -S3 input details and current boundaries are documented in [docs/s3-audio-input.md](docs/s3-audio-input.md). - -## Remote Storage Backend - -Narratio includes an object-store backend layer for future prepare/archive work: - -- `List(ctx, prefix)` -- `Download(ctx, key, localPath)` -- `Upload(ctx, localPath, key, opts)` -- `Exists(ctx, key)` - -Implemented backends: - -- fake storage backend for deterministic tests -- S3-compatible backend built from `pipeline.storage.s3` - -Key invariant: - -- callers pass full bucket-relative object keys -- storage backends do not prepend `root_prefix` and do not infer session/campaign paths - -Current boundary: - -- `prepare` uses `List` + `Download` through the backend when `session.inputs.audio_s3` is configured -- `archive` uses `Upload` through the backend for successful run-record uploads -- `archive` also uses `Upload` for promotion writes and current pointers -- no failed or incomplete runs are uploaded -- local audio is not re-uploaded by default - -Archive run-upload details and boundaries are documented in [docs/archive-storage.md](docs/archive-storage.md). - -## Canonical Stage Order - -1. `prepare` -2. `transcribe` -3. `merge` -4. `polish` -5. `normalize` -6. `trim` -7. `analyze` -8. `archive` -9. `notify` - -## Transcript Tiers - -- `transcripts/merged.json`: canonical deterministic merged transcript from Seriatim merge -- `transcripts/processed.json`: full raw Audita-polished transcript output -- `transcripts/normalized.json`: Seriatim-normalized transcript from the normalize stage -- `transcripts/trimmed.json`: gameplay-only normalized polished transcript from trim stage - -## Seriatim Configuration - -`pipeline.seriatim` configures the Seriatim subprocess adapter used by `merge`, `normalize`, and `trim`. - -Minimal behavior: - -- `pipeline.seriatim` may be omitted entirely. -- when omitted, Narratio defaults to: - - `binary: seriatim` - - `timeout: 10m` - - `output_schema: seriatim-intermediate` - - `coalesce_gap: 3.0` - - `report: true` - -Optional overrides in `pipeline.seriatim` continue to work, including explicit binary paths and advanced `env` tuning values. - -## Audita Configuration - -`pipeline.audita` configures the real Audita subprocess adapter used by `polish`. - -Minimal behavior: - -- `pipeline.audita` may be omitted entirely. -- when omitted, Narratio defaults to: - - `binary: audita` - - `timeout: 3h` - - `report: true` - -Optional: - -- `llm_api_key_env` (when set, Narratio requires that env var and passes it to Audita as `AUDITA_LLM_API_KEY`) -- `modules` override list (when empty/omitted, Narratio does not pass `--modules`) -- `base_url` (when omitted, Narratio does not pass `--base-url`; Audita runtime defaults/config may apply) -- `model` (when omitted, Narratio does not pass `--model`; Audita runtime defaults/config may apply) -- `transcript_description` -- `config_path` -- `output_schema` (`bare-segments` or `audita-v1`) -- `work_dir_retention` (`always`, `auto`, or `never`) -- `total_llm_concurrency` (> 0 when provided) -- `proposal_llm_concurrency` (> 0 when provided) -- `validation_model` -- `validation_llm_concurrency` (> 0 when provided) -- `report` (defaults to `true`) - -Narratio passes only configured optional Audita flags. Omitted optional values are left to Audita runtime defaults/config. - -## Normalize Configuration - -`pipeline.normalize` is optional. When omitted, Narratio defaults to: - -- `output_path: transcripts/normalized.json` -- `output_schema: seriatim-intermediate` -- `report: true` - -Allowed `normalize.output_schema` values: - -- `seriatim-minimal` -- `seriatim-intermediate` -- `seriatim-full` - -`normalize.output_path` is treated as session-workdir-relative when not absolute. - -Normalize stage behavior summary: - -- normalize runs after `polish` and before `trim` -- normalize resolves `transcripts/processed.json` -- normalize runs Seriatim `normalize` to produce `transcripts/normalized.json` -- normalize diagnostics are written to: - - `artifacts/seriatim.normalize.report.json` (when enabled) - - `logs/seriatim.normalize.stdout.log` - - `logs/seriatim.normalize.stderr.log` - - `config/seriatim.normalize.generated.yml` - -## Trim Configuration - -`pipeline.trim` is optional. If omitted, no trim config is loaded. If `trim.enabled` is omitted, it defaults to `false`. - -When `trim.enabled: true`: - -- `trim.output_path` is required -- `trim.bounds.prompt_id` is required -- `trim.bounds.transcript_input_name` is required -- `trim.bounds.output_path` is required -- `trim.bounds.timeout` must be a valid Go duration when provided -- `trim.bounds.render_debug: true` requires `trim.bounds.render_output_path` -- `trim.bounds.profile_id` may be empty to use the prompt default profile - -Trim paths are treated as session-workdir-relative when not absolute. - -Example trim config: - -```yaml -trim: - enabled: true - output_path: "transcripts/trimmed.json" - bounds: - prompt_id: "dnd_session.bounds" - profile_id: "" - transcript_input_name: "transcript" - output_path: "artifacts/session_bounds.json" - timeout: "10m" - render_debug: false - render_output_path: "artifacts/session_bounds.render.json" - seriatim: - report: false -``` - -Trim behavior summary: - -- trim discovers and validates `transcripts/normalized.json` -- trim uses Scriptorium bounds (`dnd_session.bounds` by example config) to produce `artifacts/session_bounds.json` -- bounds IDs are validated against the same normalized transcript ID space that Seriatim trim will consume -- trim converts bounds to Seriatim keep selector (for example `10-868`) and runs Seriatim trim -- if trim is disabled, Narratio copies normalized transcript to trimmed transcript and records `trim_action=copy_disabled` - -Trim outputs and diagnostics: - -- `artifacts/session_bounds.json` -- `transcripts/trimmed.json` -- `logs/scriptorium.bounds.stdout.log` -- `logs/scriptorium.bounds.stderr.log` -- `config/scriptorium.bounds.generated.yml` -- `logs/seriatim.trim.stdout.log` -- `logs/seriatim.trim.stderr.log` -- `config/seriatim.trim.generated.yml` -- optional bounds render-debug outputs: - - `artifacts/session_bounds.render.json` - - `logs/scriptorium.bounds.render.stdout.log` - - `logs/scriptorium.bounds.render.stderr.log` - - `config/scriptorium.bounds.render.generated.yml` - -Render-debug files are diagnostics and are not treated as canonical stage output artifact refs. - -## Scriptorium Configuration - -`pipeline.scriptorium` is optional. When present, Narratio validates and uses it for analyze-stage artifact generation. - -Key points: - -- `scriptorium.binary` defaults to `scriptorium` when section is present -- `scriptorium.config_path` is optional -- `scriptorium.timeout` defaults to `10m` when omitted -- `scriptorium.render_debug` enables render diagnostics globally -- artifacts are configured under `scriptorium.artifacts` (map shape supports multiple artifacts) -- enabled artifacts require `prompt_id` and `output_path` -- artifact `render_debug` may override global render setting -- `vars` currently support boolean and string values - -Example `session_recap` artifact definition: - -```yaml -scriptorium: - binary: "scriptorium" - config_path: "/etc/scriptorium/config.yml" - timeout: "10m" - render_debug: false - - artifacts: - session_recap: - enabled: true - prompt_id: "dnd.session_recap" - profile_id: "local-quality" # optional - output_path: "artifacts/session_recap.md" - timeout: "10m" - # render_debug: true # optional per-artifact override - - inputs: - transcript: - source: "trimmed_transcript" - required: true - - previous_recap: - source: "previous_session_artifact" - artifact: "session_recap" - path: "" # optional; set when available - required: false - - vars: - session_id: true - session_date: true - campaign_name: true - previous_session_id: true - output_kind: "session_recap" -``` - -Prompt IDs and profile IDs are configuration values. They are not hardcoded in analyze-stage logic. - -Do not put secrets in `pipeline.yml`. If API-key behavior is configured, use env var names only. - -If `pipeline.secrets.env_dir` is configured, keep only references and secret files there; secret values are still not written to manifests, generated configs, or Narratio-managed logs. - -## Scriptorium Runtime Behavior - -Narratio integrates with Scriptorium through the public CLI subprocess contract: - -- generation: `scriptorium run` -- diagnostics/testing: `scriptorium render --format json` when `render_debug` is enabled - -For the initial implementation, only `session_recap` generation is supported. - -Analyze-stage session recap behavior: - -- preferred transcript artifact source IDs: - - `narratio.transcript.polished` - - `narratio.transcript.full` - - `narratio.transcript.trimmed` -- backward-compatible aliases remain supported: - - `processed_transcript` - - `normalized_transcript` - - `trimmed_transcript` -- session recap should use gameplay-only transcript input (`source: trimmed_transcript`) -- Narratio resolves transcript inputs from the artifact resolver (manifest producer outputs first, then canonical session paths) -- missing trimmed transcript fails clearly and advises running trim stage first -- `normalized_transcript` is the preferred full-transcript source for future table/meta-analysis artifacts -- `processed_transcript` remains supported for advanced/debug use cases -- optionally includes `previous_recap` when configured and resolvable -- omits optional previous recap when unavailable -- fails if required inputs are missing -- validates output file exists and is non-empty - -Expected session output paths: - -- `artifacts/session_recap.md` -- `logs/scriptorium.session_recap.stdout.log` -- `logs/scriptorium.session_recap.stderr.log` -- `config/scriptorium.session_recap.generated.yml` -- `artifacts/session_recap.render.json` when render diagnostics are enabled - -## Examples - -Starter files: - -- `examples/pipeline.minimal.yml` -- `examples/pipeline.audita-overrides.yml` -- `examples/session.minimal.yml` -- `examples/session.template.yml` -- `examples/speakers.yml` - -## Commands - -Run tests: +It coordinates transcription, merge/polish/normalize/trim processing, artifact generation, archive publishing, and resumable run state in one operator workflow. ```bash -go test ./... +narratio run --session-id 2026-04-04 ``` -Plan a run: +This command requires discoverable `pipeline.yml` and `session.yml` files (or explicit `--config` and `--session` flags). -```bash -go run ./cmd/narratio plan --session examples/session.minimal.yml -``` +## Documentation -Use `--config ` to override default pipeline lookup when needed. - -Run with a discoverable session template: - -```bash -go run ./cmd/narratio run --session-id 2026-04-04 -``` - -Run full pipeline: - -```bash -go run ./cmd/narratio run --config examples/pipeline.minimal.yml --session examples/session.minimal.yml -``` - -Run analyze only: - -```bash -go run ./cmd/narratio run-stage --config examples/pipeline.minimal.yml --session examples/session.minimal.yml analyze -``` - -Resume with a template session ID: - -```bash -go run ./cmd/narratio resume --config examples/pipeline.minimal.yml --session examples/session.template.yml --session-id 2026-04-04 -``` - -## Operational Note - -Checksum-based stale detection is not implemented yet. - -If prepared inputs or prompt/runtime config change, rerun the appropriate upstream stages before relying on downstream artifacts. - -Examples: - -- glossary/autocorrect/speaker-context changes: rerun at least `merge`, `polish`, `normalize`, `trim`, and `analyze` -- trim bounds prompt/profile/config changes: rerun at least `normalize`, `trim`, and `analyze` -- session recap prompt/profile/input-source changes: rerun `analyze` - -## Roadmap - -Near-term roadmap: - -- extend analyze to additional configured artifacts -- support workflows where later artifacts consume earlier generated artifacts -- keep orchestration explicit without a generic DAG engine -- implement archive and notify backends +- [Configuration](docs/config.md) +- [CLI Reference](docs/cli.md) +- [Operations and Recovery](docs/operations.md) +- [Troubleshooting](docs/troubleshooting.md) +- [Development Guide](docs/development.md) +- [Architecture Principles](docs/architecture.md) +- [Internal Component Contracts](docs/internal/README.md) +- [Config Examples](docs/examples/) diff --git a/architecture.md b/architecture.md deleted file mode 100644 index 92d0eb7..0000000 --- a/architecture.md +++ /dev/null @@ -1,502 +0,0 @@ -# Narratio Architecture - -## 1. Purpose - -`narratio` is a Go orchestrator for D&D session processing. It runs a stage-based local pipeline from audio input through transcript processing and artifact generation, with manifest-based skip/force/resume behavior. - -Narratio integrates with Scriptorium through the **public CLI** (`scriptorium run` and `scriptorium render`) via synchronous subprocess execution. - -## 2. Current Status - -Implemented: - -- strict `pipeline.yml` + `session.yml` loading with strict YAML field checking (`KnownFields(true)`) -- local workspace/session layout, lock file handling, artifact path helpers, checksums, and atomic writes -- manifest store and stage status transitions for resumable runs -- real `prepare`, `transcribe`, `merge`, and `polish` stages -- real WhisperX HTTP adapter -- real Seriatim subprocess adapter -- real Audita subprocess adapter -- real Scriptorium subprocess adapter -- real `normalize` stage producing `transcripts/normalized.json` -- real `trim` stage producing `transcripts/trimmed.json` -- real `analyze` stage for initial `session_recap` generation -- optional Scriptorium render diagnostics (`render_debug`) before production run -- storage/archive configuration and validation foundations for: - - `pipeline.storage.s3` - - `pipeline.spool` - - `pipeline.archive` promotion rules - - `session.inputs.audio_s3` -- run identity and path-model foundations: - - run ID generation (`YYYYMMDDTHHMMSSZ-xxxxxxxx`) - - S3 session/run/current key builders - - campaign/session/run local work/spool path helpers - - manifest run/path identity fields (`campaign`, `run_id`, local and S3 prefixes) -- remote storage backend layer: - - narrow object-store interface (`List`, `Download`, `Upload`, `Exists`) - - fake storage backend for deterministic tests (no network dependency) - - S3-compatible backend using AWS SDK v2 - - config-based object-store construction helper - -Still placeholder/future: - -- `notify` stage behavior -- additional Scriptorium artifact types beyond `session_recap` -- artifact-to-artifact workflows beyond the initial single-artifact implementation -- generic stale detection based on input/config checksums - -## 3. Pipeline and Stage Boundaries - -Canonical stage order: - -1. `prepare` -2. `transcribe` -3. `merge` -4. `polish` -5. `normalize` -6. `trim` -7. `analyze` -8. `archive` -9. `notify` - -Boundary rules: - -- orchestration logic lives in `internal/app` -- stage business logic lives in `internal/stage` -- external-tool CLI construction lives in adapter packages -- Scriptorium CLI details stay in `internal/adapters/scriptorium` - -## 4. Scriptorium Integration Model - -Integration mode: - -- public CLI subprocesses only (no Scriptorium internal Go packages, no HTTP API) -- production generation uses `scriptorium run` -- diagnostics/testing render uses `scriptorium render --format json` - -Run invocation shape used by adapter: - -```bash -scriptorium run --prompt --input name=path --out -``` - -Optional flags passed when configured: - -- `--config ` -- `--profile ` -- repeated `--var name=value` -- repeated `--input name=path` -- `--timeout ` -- `--api-key-env ` when configured - -Render invocation shape used by adapter: - -```bash -scriptorium render --prompt --input name=path --format json --out -``` - -Adapter behavior: - -- always passes `--out` -- captures stdout/stderr separately -- writes generated invocation metadata YAML (redacted, no secrets) -- treats exit code `0` as success -- treats exit code `1` as failure -- treats exit code `2` as failure with `validation_failed=true` and preserves output metadata when available -- validates successful output files exist and are non-empty -- does not treat non-empty stderr as failure by itself - -## 5. Configuration Contract - -CLI pipeline config path resolution: - -- when `--config ` is provided, that path is used -- when `--config` is omitted, Narratio searches defaults in order: - - `/usr/local/etc/narratio/pipeline.yml` - - `/etc/narratio/pipeline.yml` -- default values are centralized in `internal/config/defaults.go` - -CLI session config path resolution: - -- when `--session ` is provided, that path is used -- when `--session` is omitted, Narratio searches defaults in order: - - `./session.yml` - - `/usr/local/etc/narratio/session.yml` - - `/etc/narratio/session.yml` - -Session template rendering: - -- session templates are rendered before strict YAML decode -- `--session-id ` provides the `session_id` template variable -- supported placeholders: - - `{{session_id}}` - - `{{ session_id }}` -- unresolved placeholders fail clearly -- strict `KnownFields(true)` YAML validation still applies after rendering -- if rendered `session.session_id` conflicts with `--session-id`, load fails clearly - -Optional pipeline secrets directory: - -- `pipeline.secrets.env_dir` enables loading environment variables from local files before command execution -- file name = env var name; file contents = env var value (trailing newline/CRLF trimmed) -- only env-var-style file names are considered; other entries are ignored -- existing process environment values are preserved (not overwritten) -- if configured, unreadable/missing `env_dir` fails command execution early -- relative `env_dir` values are resolved from current working directory - -Storage and archive foundations: - -- `pipeline.storage.s3` is available for modeling S3 coordinates: - - `bucket` - - `root_prefix` (default `dnd`) - - `region` - - `endpoint` - - `force_path_style` (default `false`) - - `access_key_id_env` (default `OBJECT_STORAGE_KEY_ID`) - - `secret_access_key_env` (default `OBJECT_STORAGE_KEY`) -- `pipeline.spool.root` defaults to `/var/spool/narratio` -- `pipeline.workspace.cleanup_after_archive` defaults to `false` -- `pipeline.spool.delete_audio_after_archive` defaults to `false` -- `pipeline.archive` is optional and defaults to: - - `enabled: true` - - `upload_run: true` - - default `promote_artifacts`: - - `transcripts/trimmed.json` - - `artifacts/session_recap.md` -- archive promotion rules enforce safe relative paths: - - `from` and `to` are required - - absolute paths are rejected - - traversal segments such as `..` are rejected - -Session input foundations: - -- `session.campaign` is required -- local audio remains supported through `session.inputs.audio_dir` or `session.inputs.audio_files` -- optional S3 audio input shape is `session.inputs.audio_s3.prefix` -- `audio_dir`/`audio_files` and `audio_s3` are mutually exclusive -- when `audio_s3` is configured, `prepare` lists and downloads `.flac` objects through the object-store backend - -Cross-config validation scope: - -- `pipeline.storage.s3.bucket` is required only when an S3-dependent feature is explicitly configured (for current foundations, that includes `session.inputs.audio_s3`, and archive upload intent when using `storage.backend: s3`) -- no AWS credential values are stored in Narratio config; only env-var names are configured -- when both configured credential env vars resolve to non-empty values, the S3 backend uses them as static credentials -- when either configured credential value is missing, the S3 backend falls back to the AWS SDK default credential chain - -Remote object-store backend scope: - -- remote storage APIs are isolated to `internal/adapters/storage` -- AWS SDK types remain contained within the S3 backend implementation package -- S3 key/session path semantics remain outside the backend, with this invariant: - - callers pass full bucket-relative object keys - - backend methods do not prepend `root_prefix` or infer campaign/session/run paths -- `prepare` now uses object-store `List` and `Download` for S3 audio input -- `archive` now uses object-store `Upload` for successful run-record upload under the run prefix -- `archive` now uses object-store `Upload` for promoted outputs and current pointers - -Prepare S3 audio behavior (implemented): - -- compute session prefix as `{root_prefix}/campaigns/{campaign}/sessions/{session_id}/` -- resolve `session.inputs.audio_s3.prefix` under that session prefix -- list objects under the computed audio prefix and filter `.flac` keys -- fail clearly when no `.flac` objects are found -- download selected objects to spool audio path: - - `{spool.root}/{campaign}/{session_id}/{run_id}/audio/` -- materialize audio files into workdir audio path: - - `{workspace.root}/work/{campaign}/{session_id}/{run_id}/audio/` -- record S3 provenance in manifest input records (bucket/key/metadata/local paths/checksum) -- no AWS SDK types are used in stage code; storage implementation details stay in storage adapter packages - -Archive publishing behavior (implemented): - -- `archive` verifies prerequisite stage success before upload: - - `prepare`, `transcribe`, `merge`, `polish`, `normalize`, `trim`, `analyze` -- only successful/completed runs are uploaded -- uploaded run record destination is: - - `{root_prefix}/campaigns/{campaign}/sessions/{session_id}/runs/{run_id}/` -- uploaded existing local paths include: - - `inputs/`, `transcripts/`, `artifacts/`, optional `reports/`, `config/`, `logs/`, and `manifest.json` -- local `audio/` is intentionally excluded from upload by default -- file upload order is deterministic (sorted relative paths) -- `archive.enabled: false` and `archive.upload_run: false` skip upload cleanly -- stage metadata records non-secret upload context: - - run upload details, promoted output details, current manifest key, current pointer key - - no secrets, transcript contents, prompt contents, or environment dumps -- promotion rules: - - `from` resolves from local workdir - - `to` resolves under session-level S3 root - - missing required source fails archive - - missing optional source is skipped and recorded -- default promoted outputs: - - `transcripts/trimmed.json` - - `artifacts/session_recap.md` -- current pointers: - - `current/manifest.json` uploaded after run upload and promotions - - `current/run_id.txt` uploaded last with `{run_id}\n` - - `current/run_id.txt` is the effective commit marker -- if promotion or current-manifest upload fails, archive returns failure and does not write `current/run_id.txt` -- failed/incomplete runs remain local and are not uploaded -- post-archive local cleanup (implemented, opt-in): - - cleanup runs only after archive succeeded and wrote `current/run_id.txt` - - cleanup is executed after all selected stages in the command invocation succeed (for example, a later `notify` failure leaves local files intact) - - `pipeline.spool.delete_audio_after_archive: true` removes only `{spool.root}/{campaign}/{session_id}/{run_id}/audio/` - - `pipeline.workspace.cleanup_after_archive: true` removes only `{workspace.root}/work/{campaign}/{session_id}/{run_id}/` - - cleanup does not run when archive is skipped/disabled/fails or when run upload is disabled - - local development `audio_dir`/`audio_files` inputs are never removed by spool cleanup - -`pipeline.scriptorium` is optional. Existing pipelines without Scriptorium continue to work. - -`pipeline.trim` is optional. Existing pipelines without trim config continue to work. - -`pipeline.normalize` is optional. Existing pipelines without normalize config continue to work. - -`pipeline.audita` drives the real Audita subprocess adapter for the `polish` stage. - -Audita defaulted fields: - -- `binary` defaults to `audita` -- `timeout` defaults to `3h` -- `report` defaults to `true` - -Audita optional fields: - -- `llm_api_key_env` (enforced only when configured) -- `modules` override list (when omitted/empty, Narratio does not pass `--modules`) -- `base_url` (when omitted, Narratio does not pass `--base-url`) -- `model` (when omitted, Narratio does not pass `--model`) -- `transcript_description` -- `config_path` -- `output_schema` (`bare-segments` or `audita-v1`) -- `work_dir_retention` (`always`, `auto`, `never`) -- `total_llm_concurrency` (> 0 when provided) -- `proposal_llm_concurrency` (> 0 when provided) -- `validation_model` -- `validation_llm_concurrency` (> 0 when provided) -- `report` override - -Narratio passes only configured optional Audita flags; omitted optional values defer to Audita runtime defaults/config. - -Seriatim defaults: - -- `pipeline.seriatim` may be omitted -- `binary` defaults to `seriatim` -- `timeout` defaults to `10m` -- `output_schema` defaults to `seriatim-intermediate` -- `coalesce_gap` defaults to `3.0` -- `report` defaults to `true` - -When `pipeline.normalize` is omitted, defaults are applied: - -- `output_path: transcripts/normalized.json` -- `output_schema: seriatim-intermediate` -- `report: true` - -When `pipeline.normalize` is present: - -- `output_path` must be non-empty -- `output_schema` must be one of `seriatim-minimal`, `seriatim-intermediate`, or `seriatim-full` -- relative `output_path` values are session-workdir-relative paths -- Seriatim binary settings still come from `pipeline.seriatim` - -When `pipeline.trim` is present: - -- `enabled` is optional and defaults to `false` when omitted -- relative `output_path`, `bounds.output_path`, and `bounds.render_output_path` values are session-workdir-relative paths -- do not store secrets in trim config values - -When `pipeline.trim.enabled: true`: - -- `output_path` is required and non-empty -- `bounds.prompt_id` is required and non-empty -- `bounds.transcript_input_name` is required and non-empty -- `bounds.output_path` is required and non-empty -- `bounds.timeout` must parse as a Go duration when provided -- `bounds.render_debug: true` requires non-empty `bounds.render_output_path` -- `bounds.profile_id` may be empty to use the prompt default profile -- prompt IDs are config values, not hardcoded stage logic - -When `pipeline.scriptorium` is present: - -- `binary` defaults to `scriptorium` when omitted -- `config_path` is optional; when provided it must be non-empty -- `timeout` is optional; when provided it must parse as a Go duration -- default `timeout` is `10m` -- unknown YAML fields fail strict decode - -Artifacts are configured as a map under `pipeline.scriptorium.artifacts` so multiple artifacts are possible in the config shape. - -For each artifact definition: - -- `enabled: true` requires non-empty `prompt_id` -- `enabled: true` requires non-empty `output_path` -- `timeout` must parse as Go duration when present -- optional per-artifact `render_debug` may override global `scriptorium.render_debug` -- `inputs` are named and each input requires non-empty `source` -- inputs may be optional (`required: false`) -- `vars` values currently support `string` and `bool` - -Prompt IDs and profile IDs are configuration values, not hardcoded stage logic. - -Trim config shape: - -```yaml -trim: - enabled: true - output_path: "transcripts/trimmed.json" - bounds: - prompt_id: "dnd_session.bounds" - profile_id: "" - transcript_input_name: "transcript" - output_path: "artifacts/session_bounds.json" - timeout: "10m" - render_debug: false - render_output_path: "artifacts/session_bounds.render.json" - seriatim: - report: false -``` - -## 6. Transcript Tiers - -Narratio currently produces and uses four transcript tiers: - -- `transcripts/merged.json`: canonical deterministic merged transcript from Seriatim merge -- `transcripts/processed.json`: full raw Audita-polished transcript output (includes pre/post-game content) -- `transcripts/normalized.json`: normalized transcript generated by Seriatim normalize -- `transcripts/trimmed.json`: gameplay-only normalized polished transcript from trim stage - -Trim reads `transcripts/normalized.json`, validates bounds IDs against that same transcript ID space, and writes `transcripts/trimmed.json`. - -## 7. Normalize Stage (Current Implementation) - -Normalize stage behavior: - -- stage order position: after `polish` and before `trim` -- discovers processed transcript from manifest polish outputs (`transcript_processed`) when present, else `work//transcripts/processed.json` -- validates processed transcript JSON shape (`segments` array required) -- runs Seriatim `normalize` to produce normalized transcript -- validates normalized transcript JSON shape (`segments` array required) -- validates normalize report JSON when enabled - -Expected normalize outputs and diagnostics: - -- `transcripts/normalized.json` -- `artifacts/seriatim.normalize.report.json` (when normalize report is enabled) -- `logs/seriatim.normalize.stdout.log` -- `logs/seriatim.normalize.stderr.log` -- `config/seriatim.normalize.generated.yml` - -## 8. Trim Stage (Current Implementation) - -Trim stage behavior: - -- stage order position: after `normalize` and before `analyze` -- discovers normalized transcript from manifest normalize outputs (`transcript_normalized`) when present, else `work//transcripts/normalized.json` -- validates normalized transcript JSON shape (`segments` array required) -- when `trim.enabled: false` (or trim config omitted), deterministically copies normalized transcript to `transcripts/trimmed.json` and records `trim_action=copy_disabled` -- when `trim.enabled: true`: - - runs Scriptorium bounds prompt using configured `trim.bounds.prompt_id` - - writes bounds output to configured path (typically `artifacts/session_bounds.json`) - - parses and validates bounds output against the same normalized transcript being trimmed - - converts bounds range to Seriatim keep selector (for example `10-868`) - - runs Seriatim `trim` to produce `transcripts/trimmed.json` - - supports no-trim bounds actions (`none`/`copy`) by copying normalized transcript unchanged -- validates trimmed transcript JSON shape (`segments` array required) - -Expected trim outputs and diagnostics: - -- `artifacts/session_bounds.json` -- `transcripts/trimmed.json` -- `logs/scriptorium.bounds.stdout.log` -- `logs/scriptorium.bounds.stderr.log` -- `config/scriptorium.bounds.generated.yml` -- `logs/seriatim.trim.stdout.log` -- `logs/seriatim.trim.stderr.log` -- `config/seriatim.trim.generated.yml` -- optional bounds render-debug outputs when enabled: - - `artifacts/session_bounds.render.json` - - `logs/scriptorium.bounds.render.stdout.log` - - `logs/scriptorium.bounds.render.stderr.log` - - `config/scriptorium.bounds.render.generated.yml` - -Render-debug files are diagnostics. They are recorded in stage metadata/log/config refs and are not treated as canonical stage output artifact refs. - -## 9. Analyze Stage (Current Implementation) - -The current real analyze implementation supports only `scriptorium.artifacts.session_recap`. - -Behavior: - -- if `pipeline.scriptorium` is missing, analyze returns a skipped result with metadata -- if no Scriptorium artifacts are enabled, analyze returns a skipped result with metadata -- if enabled artifacts exist but `session_recap` is not enabled, analyze fails clearly -- available transcript input sources for configured artifacts: `processed_transcript`, `normalized_transcript`, `trimmed_transcript` -- `session_recap` should use `trimmed_transcript` input (`transcripts/trimmed.json`) for in-universe recap generation -- `trimmed_transcript` input is resolved from manifest (`trim` output kind `transcript_trimmed`) when available, otherwise fallback path `work//transcripts/trimmed.json` -- `normalized_transcript` input is resolved from manifest (`normalize` output kind `transcript_normalized`) when available, otherwise fallback path `work//transcripts/normalized.json` -- `processed_transcript` input is resolved from manifest (`polish` output kind `transcript_processed`) when available, otherwise fallback path `work//transcripts/processed.json` -- `normalized_transcript` is the preferred full-transcript source for future table/meta-analysis artifacts -- `processed_transcript` remains available for advanced/debug use cases -- transcript inputs are validated as JSON with top-level `segments` array -- configured inputs are resolved by source -- optional `previous_recap` is omitted when unavailable -- required `previous_recap` fails before invocation when unavailable -- vars are built from config + session metadata -- `render_debug` controls pre-run `scriptorium render` diagnostics -- render failure stops stage before production run -- render output is validated as JSON -- production call uses Scriptorium adapter `RunArtifact` -- successful run output must exist and be non-empty -- missing `trimmed_transcript` input for configured `trimmed_transcript` source fails clearly with guidance to run trim stage first -- manifest records output refs, logs, generated config paths, and non-secret provenance metadata - -## 10. Session Recap Paths - -Current expected paths for `session_recap`: - -- artifact output: `artifacts/session_recap.md` -- run stdout log: `logs/scriptorium.session_recap.stdout.log` -- run stderr log: `logs/scriptorium.session_recap.stderr.log` -- run generated invocation/config: `config/scriptorium.session_recap.generated.yml` -- render output (when enabled): `artifacts/session_recap.render.json` -- render stdout log: `logs/scriptorium.session_recap.render.stdout.log` -- render stderr log: `logs/scriptorium.session_recap.render.stderr.log` -- render generated invocation/config: `config/scriptorium.session_recap.render.generated.yml` - -## 11. Security and Privacy - -- do not store secrets in pipeline YAML, generated invocation YAML, logs, or manifest metadata -- if API-key integration is configured, pass env var names only (never raw key values) -- with `pipeline.secrets.env_dir`, secret file values are loaded into process env only and are not persisted in manifest metadata or generated configs -- avoid logging transcript content or rendered prompt content by default -- treat generated artifacts and logs as potentially sensitive session material - -## 12. Operational Caveat (Pre-Stale-Detection) - -Checksum-based stale detection is not implemented yet. - -If prepared inputs or prompt/runtime configuration change (for example glossary files, prompt IDs, profile IDs, or relevant pipeline settings), rerun the appropriate prior stages to refresh downstream artifacts. - -Examples: - -- glossary or autocorrect changes usually require rerunning at least `merge`, `polish`, `normalize`, `trim`, and `analyze` -- trim prompt/profile changes require rerunning at least `normalize`, `trim`, and `analyze` -- session recap prompt/profile/input-source changes require rerunning `analyze` - -## 13. Roadmap - -Planned next steps: - -- extend analyze beyond `session_recap` to additional configured artifacts -- support artifact inputs that consume prior generated artifacts -- keep this composable without adding a generic DAG engine in the near term -- implement real `archive` backend behavior -- implement real `notify` backend behavior -- add checksum-based stale detection and stale transitions - -Architectural invariants remain: - -- strict config decoding/validation -- manifest-driven run control -- clear stage/adapter separation -- configuration-driven prompt/profile/input/vars/output mapping -- Scriptorium integration through public CLI subprocess contract diff --git a/docs/architecture.md b/docs/architecture.md new file mode 100644 index 0000000..3142fa2 --- /dev/null +++ b/docs/architecture.md @@ -0,0 +1,202 @@ +# Narratio Architecture + +## Purpose + +`narratio` is a Go orchestration application for processing D&D session audio into polished transcripts and generated session artifacts. + +This document defines the development principles for the project. It is inward-facing: its audience is developers and LLM coding agents. It should guide future changes, not serve as a complete implementation reference. + +Implemented component details belong under `docs/internal/`. + +## Project Shape + +Narratio is a modular, stage-driven orchestrator. + +It coordinates specialized downstream systems rather than reimplementing their domains: + +- WhisperX handles transcription. +- Seriatim handles deterministic transcript merge/normalization/trim behavior. +- Audita handles transcript correction and polishing. +- Scriptorium handles prompt execution and generated artifacts. + +Narratio owns orchestration, configuration loading, session/run state, local and remote path modeling, manifest persistence, stage sequencing, resume behavior, and archive semantics. + +Narratio should remain explicit and comprehensible. It is not intended to become a generic workflow engine. + +## Core Principles + +### Modular and composable + +Code should be organized around clear responsibilities. Stages, adapters, config loading, manifest persistence, path construction, and storage behavior should remain separable and independently testable. + +### Hexagonal boundaries + +External systems should be isolated behind narrow adapters. Stage logic should depend on Narratio-level interfaces and data structures, not on external SDK types, subprocess argument construction, or transport-specific details. + +### Standard library preference + +Prefer the Go standard library. Add dependencies only when they provide substantial value, are necessary for an external integration, or are a widely used de facto standard. + +Accepted examples include a YAML library for configuration and the AWS SDK for S3-compatible storage. + +### Explicit orchestration + +The pipeline should remain stage-driven and explicit. New behavior should be added through clear stage, adapter, config, or manifest contracts rather than implicit side effects or generic workflow abstraction. + +## Stage Design + +Each stage should have a clear scope of responsibility. + +A stage should define: + +- its purpose; +- required input state; +- produced output state; +- config fields it consumes; +- external adapters it uses; +- manifest refs it reads or writes; +- skip, force, and resume behavior; +- failure behavior; +- tests that protect its contract. + +Stages should avoid reaching across boundaries. If shared behavior is needed, prefer a helper or service with a narrow interface over duplicating ad hoc logic between stages. + +## Transactionality and Resume + +A stage should behave transactionally. + +A stage is complete only when its outputs have been written, validated, and recorded in the manifest. If a stage fails, Narratio should preserve enough local state for inspection, recovery, and resume. + +A failed or incomplete run must not be treated as successful. Later stages should depend on manifest-recorded success, not merely on incidental files existing on disk. + +## Manifest Model + +The manifest is the durable local ledger for a run. + +It should record: + +- run identity; +- stage status; +- input and output refs; +- logs and generated config refs; +- checksums or provenance where useful; +- non-secret adapter and archive metadata. + +Resume behavior should be manifest-driven. Filesystem state may be inspected and validated, but it should not replace manifest stage state as the source of run progress. + +## Adapter Boundaries + +Adapters own external integration details. + +Expected boundaries: + +- WhisperX HTTP details stay in the WhisperX adapter. +- Seriatim CLI construction stays in the Seriatim adapter. +- Audita CLI construction stays in the Audita adapter. +- Scriptorium CLI construction stays in the Scriptorium adapter. +- Object-storage details stay behind the storage adapter interface. +- AWS SDK types stay inside the S3 storage implementation. + +Stage code should express intent in Narratio terms and call adapters through narrow contracts. + +## Configuration Philosophy + +Configuration should be strict, explicit, and operator-friendly. + +Principles: + +- YAML decoding should reject unknown fields. +- Defaults should be centralized and testable. +- Empty configured values should not silently override meaningful defaults. +- Session templating should remain narrow and deterministic. +- Template support should serve operator convenience, not become a general configuration language. + +Narratio should not become a secondary configuration system for downstream tools. Seriatim, Audita, and Scriptorium should own their runtime defaults wherever practical. Narratio should pass required stage-contract paths and explicit operator overrides. + +## Path and Storage Discipline + +Local and remote paths are part of Narratio’s application contract. + +Code should use centralized path helpers for workspace, spool, session, run, artifact, log, config, and archive paths. Stages should avoid reconstructing canonical paths through scattered string concatenation. + +Storage backends should receive explicit bucket-relative keys. Storage implementations should not infer campaign, session, run, or root-prefix semantics. + +## Archive Invariants + +Archive behavior must preserve a clear commit boundary. + +A remote run is current only after the archive stage has successfully uploaded the run record, required promoted outputs, `current/manifest.json`, and finally `current/run_id.txt`. + +`current/run_id.txt` is the final remote commit marker and must be written last. + +Failed, incomplete, skipped, or uncommitted archive attempts must not be presented as current remote state. Local cleanup is permitted only after successful archive commit and only when explicitly configured. + +## Security and Privacy + +Narratio handles private campaign material. + +Rules: + +- Do not store raw secrets in pipeline or session YAML. +- Use environment variable names or secret-file references for secret handling. +- Do not write raw secret values to manifests, logs, generated configs, or archive metadata. +- Treat transcripts, generated artifacts, prompts, reports, and logs as potentially sensitive. +- Avoid logging transcript or prompt content unless there is a deliberate diagnostic reason. + +## Diagnostics + +Diagnostics should be durable and discoverable, but distinct from canonical outputs. + +Logs, reports, generated invocation/config files, and render-debug files support debugging. Transcript tiers and configured artifacts are pipeline products. + +Manifest refs should preserve that distinction. + +## Determinism + +Where practical, Narratio should prefer deterministic behavior: + +- stable local path layout; +- stable remote key layout; +- sorted upload order; +- predictable generated config files; +- repeatable command construction; +- tests that do not depend on live external services. + +Run IDs and timestamps may be intentionally variable, but surrounding behavior should remain testable. + +## Testing Expectations + +Core behavior should be testable without live external services. + +Tests should cover: + +- config loading, defaults, and validation; +- CLI parsing and command construction; +- path helpers; +- manifest transitions; +- stage success, failure, skip, and resume behavior; +- adapter command construction; +- fake storage behavior; +- archive commit ordering; +- example config validity where practical. + +Live S3, WhisperX, LLM, or subprocess integration tests should be explicit integration tests, not required for ordinary unit test runs. + +## Documentation Expectations + +Documentation must follow `docs/documentation/policy.md`. + +Current behavior belongs in user-facing docs and `docs/internal/`. Future, planned, aspirational, experimental, or unimplemented work belongs only under `docs/roadmap/`. + +`docs/architecture.md` should remain concise and principle-focused. It should not duplicate the full config reference, CLI reference, operations guide, or internal stage documentation. + +## Non-Goals + +Narratio is not: + +- a generic DAG or workflow engine; +- a replacement configuration layer for Seriatim, Audita, or Scriptorium; +- a storage backend abstraction beyond the needs of this pipeline; +- a place to embed raw secrets; +- a place for stage logic to depend directly on AWS SDK types or downstream tool internals; +- a prompt-authoring system. diff --git a/docs/cli.md b/docs/cli.md new file mode 100644 index 0000000..c68f237 --- /dev/null +++ b/docs/cli.md @@ -0,0 +1,230 @@ +# CLI + +## Shortest Useful Command + +```bash +narratio run --session-id 2026-04-04 +``` + +This uses default config discovery for `pipeline.yml` and `session.yml`; both files must be discoverable for this command to run. + +## Command Overview + +Implemented commands: + +- `run`: execute the full stage plan and persist manifest state. +- `plan`: validate config, prepare workdir, and print run/skip decisions. +- `resume`: continue from the first non-succeeded stage in the manifest. +- `status`: read and print stage statuses from an existing manifest file. +- `run-stage`: execute exactly one selected stage. + +Unknown commands print usage (`Usage: narratio `) and exit non-zero. + +For configuration field details, see [docs/config.md](./config.md). For operational lifecycle details, see [docs/operations.md](./operations.md). + +## Complete Flag Reference + +### `run` + +- `--config `: optional explicit `pipeline.yml` path; if omitted, default locations are searched. +- `--session `: optional explicit `session.yml` path; if omitted, default locations are searched. +- `--session-id `: session template variable value for `session.yml` rendering. +- `--force`: force stage execution (prevents skip of already-succeeded stages). + +### `plan` + +- `--config ` +- `--session ` +- `--session-id ` +- `--force`: show forced run decisions instead of normal skip behavior. + +### `resume` + +- `--config ` +- `--session ` +- `--session-id ` +- `--force`: run full stage order rather than starting at first non-succeeded stage. + +### `run-stage` + +- `--config ` +- `--session ` +- `--session-id ` +- `--force` +- positional ``: required stage name. + +Valid stage names: + +- `prepare` +- `transcribe` +- `merge` +- `polish` +- `normalize` +- `trim` +- `analyze` +- `archive` +- `notify` + +### `status` + +- `--manifest `: required manifest path. + +## Command Reference + +### `run` + +Purpose: + +- Validate configuration and execute all stages in canonical order. + +Syntax: + +```bash +narratio run [--config ] [--session ] [--session-id ] [--force] +``` + +Success output: + +- `narratio run: session ; executed= skipped=; manifest=` + +Common failure cases: + +- no pipeline config found in default search paths when `--config` is omitted. +- no session config found in default search paths when `--session` is omitted. +- invalid flags or unexpected positional arguments. +- config/template/validation errors. + +### `plan` + +Purpose: + +- Validate config, load secrets (if configured), prepare workspace layout, and print per-stage run/skip decisions. + +Syntax: + +```bash +narratio plan [--config ] [--session ] [--session-id ] [--force] +``` + +Success output includes: + +- `narratio plan: workdir prepared at ` +- one line per stage (`: run|skip`) +- `totals: run= skip=` + +Common failure cases: + +- same discovery, template, and validation failures as `run`. +- secrets directory read failures when `pipeline.secrets.env_dir` is configured. + +### `resume` + +Purpose: + +- Continue execution from manifest state for the same session. + +Syntax: + +```bash +narratio resume [--config ] [--session ] [--session-id ] [--force] +``` + +Success output: + +- either `narratio resume: session has no remaining stages` +- or `narratio resume: session ; executed= skipped=; manifest=` + +Common failure cases: + +- same discovery/template/validation failures as `run`. +- manifest load errors when an existing manifest is unreadable. + +### `status` + +Purpose: + +- Inspect an existing manifest file without running stages. + +Syntax: + +```bash +narratio status --manifest +``` + +Success output includes: + +- `session_id: ` +- `updated_at: ` +- `stages:` section with `- : ` entries. + +Common failure cases: + +- missing `--manifest`. +- manifest path unreadable or invalid JSON shape. + +### `run-stage` + +Purpose: + +- Execute exactly one stage from the supported stage set. + +Syntax: + +```bash +narratio run-stage [--config ] [--session ] [--session-id ] [--force] +``` + +Success output: + +- `narratio run-stage: stage= executed= skipped= force=; manifest=` + +Common failure cases: + +- missing stage positional argument. +- unknown stage name. +- same discovery/template/validation failures as `run`. + +## Common Workflows + +Default-discovery run: + +```bash +narratio run --session-id 2026-04-04 +``` + +Explicit config/session run: + +```bash +narratio run --config /etc/narratio/pipeline.yml --session ./session.yml --session-id 2026-04-04 +``` + +Plan before run: + +```bash +narratio plan --config /etc/narratio/pipeline.yml --session ./session.yml --session-id 2026-04-04 +``` + +Resume interrupted work: + +```bash +narratio resume --config /etc/narratio/pipeline.yml --session ./session.yml --session-id 2026-04-04 +``` + +Run one stage: + +```bash +narratio run-stage --config /etc/narratio/pipeline.yml --session ./session.yml --session-id 2026-04-04 polish +``` + +## Diagnostic / Recovery Commands + +Read stage status from a manifest: + +```bash +narratio status --manifest +``` + +How to get manifest path: + +- `run`, `resume`, and `run-stage` success output includes `manifest=`. +- use that path with `status` for direct inspection. diff --git a/docs/config.md b/docs/config.md new file mode 100644 index 0000000..c54c0ce --- /dev/null +++ b/docs/config.md @@ -0,0 +1,293 @@ +# Configuration + +## 1. Overview + +Narratio loads two YAML files: + +- `pipeline.yml`: pipeline-level runtime configuration. +- `session.yml`: per-session metadata and input selection. + +These commands load and validate both files before running: + +- `narratio run` +- `narratio plan` +- `narratio resume` +- `narratio run-stage` + +Configuration behavior: + +- strict YAML decode is enabled (`KnownFields(true)`): unknown fields fail. +- session templates are rendered before session YAML decode. +- defaults are applied for many optional pipeline fields. +- validation enforces required fields, value formats, and cross-field constraints. + +## 2. Config file discovery + +Pipeline config lookup for `run`, `plan`, `resume`, and `run-stage`: + +- If `--config ` is provided, that explicit path is used. +- If `--config` is omitted, Narratio searches in order: + 1. `/usr/local/etc/narratio/pipeline.yml` + 2. `/etc/narratio/pipeline.yml` +- The first existing file wins. +- If none exist, the command fails with a searched-paths error. + +## 3. Session file discovery and templating + +Session config lookup for `run`, `plan`, `resume`, and `run-stage`: + +- If `--session ` is provided, that explicit path is used. +- If `--session` is omitted, Narratio searches in order: + 1. `./session.yml` + 2. `/usr/local/etc/narratio/session.yml` + 3. `/etc/narratio/session.yml` +- The first existing file wins. +- If none exist, the command fails and asks you to pass `--session`. + +Session templating: + +- Supported placeholders: + - `{{session_id}}` + - `{{ session_id }}` +- `--session-id ` supplies the template value. +- Unresolved placeholders fail with a template-rendering error. +- If `--session-id` is provided and rendered `session_id` differs, load fails with a mismatch error. +- Strict YAML decode still applies after template rendering. + +## 4. Minimal pipeline config + +```yaml +workspace: + root: ./tmp/narratio-workspace + +whisperx: + transcribe_url: "https://transcription.example.com/transcribe" +``` + +Why this is sufficient: + +- `workspace.root` and `whisperx.transcribe_url` are the core required pipeline fields. +- Seriatim and Audita sections may be omitted; defaults are applied. +- Archive, storage, spool, normalize, and other optional sections get defaults when omitted. + +## 5. Minimal session template + +```yaml +session_id: "{{ session_id }}" +campaign: sample-campaign +inputs: + audio_dir: ./audio + speakers_file: ./examples/speakers.yml + autocorrect_file: ./examples/autocorrect.yml + glossary_file: ./examples/glossary.yml +``` + +Usage: + +```bash +narratio run --config /path/to/pipeline.yml --session ./session.yml --session-id 2026-05-03 +``` + +## 6. Production-oriented config + +```yaml +workspace: + root: /var/lib/narratio/workspace + cleanup_after_archive: true + +storage: + backend: s3 + s3: + bucket: my-dnd-archive + root_prefix: dnd + region: us-east-1 + access_key_id_env: OBJECT_STORAGE_KEY_ID + secret_access_key_env: OBJECT_STORAGE_KEY + +spool: + root: /var/spool/narratio + delete_audio_after_archive: true + +archive: + enabled: true + upload_run: true + promote_artifacts: + - from: transcripts/trimmed.json + to: transcripts/trimmed.json + required: true + - from: artifacts/session_recap.md + to: artifacts/session_recap.md + required: true + +whisperx: + transcribe_url: "https://transcription.example.com/transcribe" +``` + +Operational notes: + +- `workspace.cleanup_after_archive` controls run-scoped workspace cleanup after successful archive commit. +- `spool.delete_audio_after_archive` controls run-scoped spool-audio cleanup after successful archive commit. +- S3 archive/session-audio workflows require `storage.s3.bucket`. + +## 7. Full pipeline reference + +Defaults listed here are effective runtime defaults after load. + +| Path | Type | Required | Default | Constraints / Notes | +| --- | --- | --- | --- | --- | +| `pipeline.workspace.root` | string | Yes | none | Must be non-empty. | +| `pipeline.workspace.cleanup_after_archive` | bool | No | `false` | Run-scoped workdir cleanup after successful archive commit. | +| `pipeline.secrets.env_dir` | string | Conditional | none | If `pipeline.secrets` is set, `env_dir` must be non-empty. | +| `pipeline.storage.backend` | string | No | empty | `s3` enables S3 archive decision path checks. | +| `pipeline.storage.bucket` | string | No | empty | Accepted by schema; compatibility field. | +| `pipeline.storage.prefix` | string | No | empty | Accepted by schema; compatibility field. | +| `pipeline.storage.s3.bucket` | string | Conditional | empty | Required when S3 session audio is used or S3 archive upload is enabled. | +| `pipeline.storage.s3.root_prefix` | string | No | `dnd` | Must be relative, non-empty, no `..`. | +| `pipeline.storage.s3.region` | string | No | empty | Optional region hint for backend wiring. | +| `pipeline.storage.s3.endpoint` | string | No | empty | If provided, must not be all-whitespace. | +| `pipeline.storage.s3.force_path_style` | bool | No | `false` | S3-compatible endpoint toggle. | +| `pipeline.storage.s3.access_key_id_env` | string | No | `OBJECT_STORAGE_KEY_ID` | Must be a valid env var name. | +| `pipeline.storage.s3.secret_access_key_env` | string | No | `OBJECT_STORAGE_KEY` | Must be a valid env var name. | +| `pipeline.spool.root` | string | No | `/var/spool/narratio` | Local spool root. | +| `pipeline.spool.delete_audio_after_archive` | bool | No | `false` | Cleanup toggle for run-scoped spool audio. | +| `pipeline.archive.enabled` | bool | No | `true` | Archive stage enablement. | +| `pipeline.archive.upload_run` | bool | No | `true` | Run-record upload toggle when archive enabled. | +| `pipeline.archive.promote_artifacts[]` | list | No | two default rules | Defaults: `transcripts/trimmed.json` and `artifacts/session_recap.md`. | +| `pipeline.archive.promote_artifacts[].from` | string | Yes (per rule) | none | Must be relative, non-empty, no `..`. | +| `pipeline.archive.promote_artifacts[].to` | string | Yes (per rule) | none | Must be relative, non-empty, no `..`. | +| `pipeline.archive.promote_artifacts[].required` | bool | No | `true` | Defaults per rule if omitted. | +| `pipeline.whisperx.transcribe_url` | string | Yes | none | Must be a valid URL. | +| `pipeline.whisperx.language` | string | No | `en` | Passed to WhisperX adapter. | +| `pipeline.whisperx.timeout` | duration string | No | `30m` | Must parse as duration. | +| `pipeline.whisperx.retries` | int | No | `3` | Must be `>= 0`. | +| `pipeline.whisperx.retry_delay` | duration string | No | `2s` | Must parse as duration. | +| `pipeline.whisperx.concurrency` | int | No | `2` | Must be `> 0`. | +| `pipeline.seriatim.binary` | string | No | `seriatim` | Must be non-empty after defaults. | +| `pipeline.seriatim.timeout` | duration string | No | `10m` | Must parse as duration. | +| `pipeline.seriatim.output_schema` | string | No | `seriatim-intermediate` | Allowed: `seriatim-minimal`, `seriatim-intermediate`, `seriatim-full`. | +| `pipeline.seriatim.coalesce_gap` | float | No | `3.0` | Must be `>= 0`. | +| `pipeline.seriatim.report` | bool | No | `true` | Enables report output in Seriatim calls. | +| `pipeline.seriatim.env.overlap_word_run_gap` | float | No | unset | If set, must be `> 0`. | +| `pipeline.seriatim.env.overlap_word_run_reorder_window` | float | No | unset | If set, must be `> 0`. | +| `pipeline.seriatim.env.backchannel_max_duration` | float | No | unset | If set, must be `> 0`. | +| `pipeline.seriatim.env.filler_max_duration` | float | No | unset | If set, must be `> 0`. | +| `pipeline.audita.binary` | string | No | `audita` | Must be non-empty after defaults. | +| `pipeline.audita.timeout` | duration string | No | `3h` | Must parse as duration. | +| `pipeline.audita.llm_api_key_env` | string | No | empty | Name of env var to forward to Audita. | +| `pipeline.audita.modules[]` | list[string] | No | empty | If set, each must be one of `glossary`, `homophones`, `spoken_word`, `grammar`. | +| `pipeline.audita.base_url` | string | No | empty | If non-empty, must be a valid URL. | +| `pipeline.audita.model` | string | No | empty | Optional model override passed to Audita. | +| `pipeline.audita.total_llm_concurrency` | int | No | unset | If set, must be `> 0`. | +| `pipeline.audita.proposal_llm_concurrency` | int | No | unset | If set, must be `> 0`. | +| `pipeline.audita.validation_model` | string | No | empty | Optional validation model override. | +| `pipeline.audita.validation_llm_concurrency` | int | No | unset | If set, must be `> 0`. | +| `pipeline.audita.transcript_description` | string | No | empty | If provided, must not be all-whitespace. | +| `pipeline.audita.config_path` | string | No | empty | If provided, must not be all-whitespace. | +| `pipeline.audita.output_schema` | string | No | empty | Allowed: empty, `bare-segments`, `audita-v1`. | +| `pipeline.audita.work_dir_retention` | string | No | empty | Allowed: empty, `always`, `auto`, `never`. | +| `pipeline.audita.report` | bool | No | `true` | Enables Audita report output. | +| `pipeline.normalize.output_path` | string | No | `transcripts/normalized.json` | Must be non-empty after defaults. | +| `pipeline.normalize.output_schema` | string | No | `seriatim-intermediate` | Allowed: `seriatim-minimal`, `seriatim-intermediate`, `seriatim-full`. | +| `pipeline.normalize.report` | bool | No | `true` | Enables normalize report output. | +| `pipeline.trim.enabled` | bool | No | `false` | When `false`, trim bounds fields are not required. | +| `pipeline.trim.output_path` | string | Conditional | none | Required when `pipeline.trim.enabled=true`. | +| `pipeline.trim.bounds.prompt_id` | string | Conditional | none | Required when `pipeline.trim.enabled=true`. | +| `pipeline.trim.bounds.profile_id` | string | No | empty | Optional profile override. | +| `pipeline.trim.bounds.transcript_input_name` | string | Conditional | none | Required when `pipeline.trim.enabled=true`. | +| `pipeline.trim.bounds.output_path` | string | Conditional | none | Required when `pipeline.trim.enabled=true`. | +| `pipeline.trim.bounds.timeout` | duration string | No | `10m` | Must parse as duration when set. | +| `pipeline.trim.bounds.render_debug` | bool | No | `false` | Enables render-debug output for bounds prompt. | +| `pipeline.trim.bounds.render_output_path` | string | Conditional | none | Required when `render_debug=true`. | +| `pipeline.trim.seriatim.report` | bool | No | `false` | Trim-stage Seriatim report toggle. | +| `pipeline.scriptorium.binary` | string | No | `scriptorium` | Required only when `pipeline.scriptorium` is configured. | +| `pipeline.scriptorium.config_path` | string | No | empty | If provided, must not be all-whitespace. | +| `pipeline.scriptorium.timeout` | duration string | No | `10m` | Must parse as duration when set. | +| `pipeline.scriptorium.render_debug` | bool | No | `false` | Global render-debug toggle for Scriptorium adapter usage. | +| `pipeline.scriptorium.artifacts` | map | No | empty | Artifact-generation map keyed by artifact name. | +| `pipeline.scriptorium.artifacts..enabled` | bool | No | `false` | If `true`, `prompt_id` and `output_path` are required. | +| `pipeline.scriptorium.artifacts..render_debug` | bool | No | unset | Per-artifact render-debug override. | +| `pipeline.scriptorium.artifacts..prompt_id` | string | Conditional | none | Required when artifact is enabled. | +| `pipeline.scriptorium.artifacts..profile_id` | string | No | empty | Optional profile override. | +| `pipeline.scriptorium.artifacts..output_path` | string | Conditional | none | Required when artifact is enabled. | +| `pipeline.scriptorium.artifacts..timeout` | duration string | No | empty | If set, must parse as duration. | +| `pipeline.scriptorium.artifacts..inputs..source` | string | Conditional | none | Required when input is present; allowed values listed below. | +| `pipeline.scriptorium.artifacts..inputs..artifact` | string | No | empty | Used by `previous_session_artifact` source. | +| `pipeline.scriptorium.artifacts..inputs..path` | string | No | empty | Optional explicit path metadata. | +| `pipeline.scriptorium.artifacts..inputs..required` | bool | No | `false` | Input requirement flag for artifact generation. | +| `pipeline.scriptorium.artifacts..vars.` | map value | No | empty | Value must be string or boolean. | +| `pipeline.analyzer.binary_path` | string | No | empty | Optional analyzer binary override. | +| `pipeline.analyzer.timeout` | duration string | No | empty | If set, must parse as duration. | +| `pipeline.analyzer.artifacts.output_dir` | string | No | empty | Optional analyzer output directory hint. | +| `pipeline.analyzer.artifacts.types[]` | list[string] | No | empty | Optional analyzer artifact type list. | +| `pipeline.notification.backend` | string | No | empty | Optional notifier backend selector. | +| `pipeline.notification.recipient` | string | No | empty | Optional notification recipient target. | +| `pipeline.notification.timeout` | duration string | No | empty | If set, must parse as duration. | + +Allowed `pipeline.scriptorium.artifacts..inputs..source` values: + +- `previous_session_artifact` +- `processed_transcript` +- `normalized_transcript` +- `trimmed_transcript` +- `narratio.transcript.merged` +- `narratio.transcript.polished` +- `narratio.transcript.full` +- `narratio.transcript.trimmed` +- `narratio.bounds.session` +- `narratio.artifact.session_recap` + +## 8. Full session reference + +| Path | Type | Required | Default | Constraints / Notes | +| --- | --- | --- | --- | --- | +| `session.session_id` | string | Yes | none | Must be non-empty after template rendering. | +| `session.campaign` | string | Yes | none | Must be non-empty. Used in local/remote path modeling. | +| `session.date` | string | No | empty | Optional session metadata for prompts/artifacts. | +| `session.title` | string | No | empty | Optional session metadata for prompts/artifacts. | +| `session.inputs.audio_dir` | string | Conditional | empty | One audio-source option. Mutually exclusive with `audio_s3`. | +| `session.inputs.audio_files[]` | list[string] | Conditional | empty | One audio-source option. At least one entry can satisfy audio-source requirement. Mutually exclusive with `audio_s3`. | +| `session.inputs.audio_s3.prefix` | string | Conditional | none | Required if `audio_s3` object is present; must be relative, non-empty, no `..`. Mutually exclusive with `audio_dir` and `audio_files`. | +| `session.inputs.speakers_file` | string | Yes | none | Must be non-empty. | +| `session.inputs.autocorrect_file` | string | Yes | none | Must be non-empty. | +| `session.inputs.glossary_file` | string | Yes | none | Must be non-empty. | + +Audio-source rule: + +- You must configure exactly one audio source mode: + - `audio_dir`, or + - `audio_files` (at least one), or + - `audio_s3.prefix` +- `audio_s3` cannot be combined with `audio_dir` or `audio_files`. + +## 9. Secrets + +Narratio supports filesystem-based secret injection via `pipeline.secrets.env_dir`. + +Behavior: + +- `env_dir` may be absolute or relative. +- Relative `env_dir` is resolved from Narratio’s current working directory. +- Each top-level file with a valid env-var filename (`[A-Za-z_][A-Za-z0-9_]*`) is loaded. +- File contents become env-var values, with trailing `\n` / `\r\n` trimmed. +- Existing process environment variables are preserved and not overwritten. +- Invalid names and directories inside `env_dir` are skipped. +- Missing/unreadable `env_dir` fails command execution. + +Guidance: + +- Store secret values in secret files or pre-set environment variables. +- Do not put secret values directly in `pipeline.yml` or `session.yml`. +- Use config fields like `llm_api_key_env` and S3 credential env names to reference secret variable names, not secret data. + +## 10. Examples + +Maintained config examples: + +- `docs/examples/pipeline.minimal.yml` +- `docs/examples/pipeline.production.yml` +- `docs/examples/pipeline.full.annotated.yml` +- `docs/examples/session.template.yml` +- `docs/examples/session.local-audio.yml` +- `docs/examples/session.s3-audio.yml` + +These examples are covered by configuration load/validate tests in `internal/config`. diff --git a/docs/development.md b/docs/development.md new file mode 100644 index 0000000..fa3986a --- /dev/null +++ b/docs/development.md @@ -0,0 +1,92 @@ +# Development Guide + +## Purpose +Canonical contributor workflow and engineering conventions for implemented Narratio behavior. + +## Repository layout + +- `cmd/narratio/`: CLI entrypoint. +- `internal/app/`: command handlers, plan/run/resume orchestration, cleanup gates, secrets loading. +- `internal/config/`: strict YAML loading, defaults, and validation. +- `internal/stage/`: stage implementations and stage registry/order. +- `internal/adapters/`: external boundary adapters (WhisperX, Seriatim, Audita, Scriptorium, storage, notify). +- `internal/manifest/`: session/run manifest types and persistence. +- `internal/artifacts/`: canonical local/remote path helpers and local artifact store. +- `docs/`: canonical documentation set. +- `docs/examples/`: maintained config examples used by tests. + +## Build and test commands + +- Run focused CLI behavior checks: + +```bash +go test ./internal/app -run TestExecute -v +``` + +- Run config example load/validate checks: + +```bash +go test ./internal/config -run TestExamplesLoadAndValidate -v +``` + +- Run full test suite: + +```bash +go test ./... +``` + +## Coding conventions + +- Keep orchestration explicit and stage-driven; do not introduce generic workflow/DAG abstractions. +- Keep external-system details inside adapter packages; stages should consume Narratio-level contracts only. +- Use centralized path helpers from `internal/artifacts` rather than ad hoc path concatenation. +- Preserve manifest-driven state transitions (`running`, `succeeded`, `failed`, `skipped`, `stale`) as the source of run progress. +- Keep user/operator docs implementation-accurate; planned work belongs only under `docs/roadmap/`. + +For design principles and invariants, see [docs/architecture.md](./architecture.md). For stage/adapter contracts, see [docs/internal/README.md](./internal/README.md). + +## Dependency policy + +- Prefer Go standard library where practical. +- Add third-party dependencies only when they provide clear value for required behavior. +- Keep dependency additions narrow to the boundary package that needs them. + +## Change playbooks + +### Add config fields + +1. Add fields to config structs in `internal/config`. +2. Set defaults in `internal/config/defaults.go` when appropriate. +3. Add validation rules in `internal/config/validate.go`. +4. Add or update load/validate tests in `internal/config/*_test.go`. +5. Update canonical config docs and examples: + - [docs/config.md](./config.md) + - relevant files under `docs/examples/` + +### Add CLI flags or commands + +1. Update command parsing and behavior in `internal/app`. +2. Add or update command tests (`TestExecute` and command-specific tests). +3. Update [docs/cli.md](./cli.md) and, if operator workflow changes, [docs/operations.md](./operations.md). + +### Add or modify stages/adapters + +1. Implement stage behavior in `internal/stage` with clear input/output boundaries. +2. Keep external transport/subprocess details in `internal/adapters`. +3. Preserve manifest and promotion semantics expected by runner and archive logic. +4. Add/update stage and adapter tests. +5. Update internal component contracts in `docs/internal/`. + +### Update examples + +1. Keep canonical examples only in `docs/examples/`. +2. Ensure examples load and validate through runtime config paths. +3. Update `internal/config/load_validate_test.go` as needed. +4. Update links in `docs/config.md` if example filenames change. + +### Update docs and roadmap + +1. Keep implemented behavior in canonical docs (`README`, `docs/*.md`, `docs/internal/`). +2. Keep planned/unimplemented behavior only in `docs/roadmap/`. +3. After completing roadmap items, remove or mark them complete in `docs/roadmap/documentation.md`. +4. Run a link/path sweep before finalizing changes. diff --git a/docs/development/workspace-implementation-plan.md b/docs/development/workspace-implementation-plan.md deleted file mode 100644 index f928c45..0000000 --- a/docs/development/workspace-implementation-plan.md +++ /dev/null @@ -1,130 +0,0 @@ -# Workspace Architecture Implementation Plan (Status) - -This document tracks the implemented workspace architecture and remaining work for v1.0. - -## Current Architecture (Implemented) - -Narratio now uses a canonical campaign-aware local layout: - -```text -{workspace.root}/work/{campaign_id}/{session_id}/ - manifest.json - current/ - manifest.json - run_id.txt - inputs/ - transcripts/ - artifacts/ - reports/ - logs/ - config/ - runs/ - {run_id}/ - manifest.json - {stage}/ - outputs/ - logs/ - reports/ - config/ - scratch/ -``` - -Core behavior: - -- Session manifest remains the skip/resume source of truth. -- Each invocation creates a run manifest at `runs/{run_id}/manifest.json`. -- Stage execution writes run-local artifacts and promotes durable outputs to canonical session paths. -- Archive uploads run records under `runs/{run_id}/`, applies promotion rules, then publishes `current/manifest.json` and `current/run_id.txt`. -- Analyze input resolution uses centralized artifact IDs with alias support. -- Forced upstream reruns mark downstream succeeded stages `stale` so later runs do not skip stale outputs. - -## Section 4 Sequence Status - -### Step 1: Campaign-aware session path model - -Status: complete. - -Implemented: - -- Campaign-aware session and run path helpers. -- Campaign-aware artifact-store layout APIs. -- Canonical session manifest pathing under `work/{campaign}/{session}`. - -### Step 2: Session manifest + run manifest scaffolding - -Status: complete. - -Implemented: - -- Invocation-scoped run manifest type and store methods. -- Runner creates/saves run manifests per invocation. -- Session manifest remains authoritative for idempotent stage skipping. - -### Step 3: Run-local stage execution + promotion - -Status: complete. - -Implemented: - -- Run-local stage directory layout under `runs/{run_id}/{stage}`. -- Shared helpers for run-local output mapping and promotion to canonical durable paths. -- Producer run provenance recorded on durable artifact outputs. - -### Step 4: Archive alignment - -Status: complete. - -Implemented: - -- Canonical run-root/session-root resolution. -- Deterministic run-file collection and promotion source resolution. -- Current-pointer publication ordering retained (`current/manifest.json` then `current/run_id.txt`). - -### Step 5: Artifact registry/resolver (analyze first consumer) - -Status: complete. - -Implemented: - -- Central artifact resolver with canonical IDs: - - `narratio.transcript.merged` - - `narratio.transcript.polished` - - `narratio.transcript.full` - - `narratio.transcript.trimmed` - - `narratio.bounds.session` - - `narratio.artifact.session_recap` -- Backward-compatible aliases: - - `processed_transcript` - - `normalized_transcript` - - `trimmed_transcript` -- Analyze stage switched to resolver-based source resolution. - -### Step 6: Minimal downstream invalidation for forced reruns - -Status: complete. - -Implemented: - -- Deterministic downstream invalidation based on canonical stage order. -- On forced successful rerun of stage `X`, downstream succeeded stages are marked `stale`. -- Resume and non-forced runs naturally re-execute stale stages. - -### Step 7: Legacy layout migration strategy - -Status: intentionally skipped. - -Decision: - -- Automatic migration and legacy fallback compatibility are intentionally not implemented. -- The codebase targets canonical-only local layout behavior. -- Legacy local workspace state, if present, should be recreated or migrated manually outside Narratio. - -## Remaining Work (v1.0) - -No required workspace/run-history migration steps remain from Section 4. - -Possible future enhancements (non-blocking): - -- Full checksum/input-graph stale detection. -- Optional retention-policy expansion for run-history cleanup. -- Broader artifact-resolver adoption across additional stage consumers. diff --git a/docs/development/workspace.md b/docs/development/workspace.md deleted file mode 100644 index 287c838..0000000 --- a/docs/development/workspace.md +++ /dev/null @@ -1,744 +0,0 @@ -# Narratio Workspace, Run History, and Artifact Resolution Architecture - -## 1. Purpose - -This document defines the intended v1.0 architecture for Narratio's local workspace layout, run history model, durable session outputs, manifest responsibilities, and artifact resolution contract. - -Narratio is an idempotent session orchestrator. The command: - -```bash -narratio run --session-id 2026-05-07 -``` - -means "bring the identified session to its desired completed state." It does **not** mean "always create an entirely new independent output tree and ignore prior session state." - -This distinction drives the architecture: - -* A **session** is the durable domain object and idempotency boundary. -* A **run** is an execution attempt that may update the session's durable state. -* Durable outputs live at the session level. -* Run-specific outputs, logs, generated configs, scratch files, and diagnostics live under `runs/{run_id}/`. -* Successful stage outputs are promoted from run-local locations into canonical session-level locations. -* The session manifest records current durable state. -* Run manifests record execution history and debugging/provenance details. - -This model intentionally mirrors the S3 archive model: session-level current artifacts are distinct from run-record history. - -## 2. Core Concepts - -### 2.1 Session - -A session is the stable unit of work identified by `campaign_id` and `session_id`. - -Examples: - -```text -campaign_id = dilfs -session_id = 2026-05-07 -``` - -The session directory represents the current durable local state for that session. Re-running Narratio for the same session should consult this state, skip already-completed stages by default, and produce no changes unless work is incomplete, stale, forced, or explicitly selected. - -### 2.2 Run - -A run is a particular execution attempt identified by a generated `run_id`, for example: - -```text -20260517T174748Z-abcd1234 -``` - -A run may execute all stages or only a sparse subset of stages. Sparse runs are expected and desirable when the user invokes `--force`, `run-stage`, or a stage-limited command. - -Run directories are provenance/debug records. They should reflect what actually happened during that invocation, not a synthetic complete pipeline layout. - -### 2.3 Durable Output - -A durable output is a canonical session-level artifact intended for later stages, user consumption, archive promotion, or future idempotency decisions. - -Examples: - -```text -transcripts/merged.json -transcripts/processed.json -transcripts/normalized.json -transcripts/trimmed.json -artifacts/session_recap.md -``` - -Durable outputs live directly under the session directory, not under a particular run directory. - -### 2.4 Run-Local Output - -A run-local output is the file initially produced by a stage during a specific run. After validation, durable outputs are promoted from run-local paths to session-level canonical paths. - -Run-local outputs, logs, generated configs, reports, and scratch files should remain under: - -```text -runs/{run_id}/{stage}/... -``` - -## 3. Local Workspace Layout - -The canonical local workspace layout is: - -```text -{workspace.root}/work/{campaign_id}/{session_id}/ - manifest.json - current/ - manifest.json - run_id.txt - inputs/ - transcripts/ - artifacts/ - reports/ - logs/ - config/ - runs/ - {run_id}/ - manifest.json - prepare/ - transcribe/ - merge/ - polish/ - normalize/ - trim/ - analyze/ - archive/ - notify/ -``` - -Not every directory must exist at all times. Directories should be created idempotently when needed. - -### 3.1 Session Root - -The session root is: - -```text -{workspace.root}/work/{campaign_id}/{session_id}/ -``` - -The session root is the stable local home for the session. It is the default base for resolving canonical artifact paths. - -The only files that should live directly in the session root are core session-state files, primarily: - -```text -manifest.json -``` - -Lock files may also be session-root scoped if the implementation uses file locks there, but transient locks should not be treated as durable artifacts. - -### 3.2 Session-Level Canonical Directories - -The following directories contain current durable session state: - -```text -inputs/ -transcripts/ -artifacts/ -reports/ -logs/ -config/ -current/ -``` - -Recommended meanings: - -| Directory | Purpose | -| -------------- | ----------------------------------------------------------------------------- | -| `inputs/` | Materialized or copied input files used by the current durable session state. | -| `transcripts/` | Canonical transcript tiers. | -| `artifacts/` | User-facing and machine-readable generated artifacts. | -| `reports/` | Canonical stage reports worth preserving at the session level. | -| `logs/` | Optional session-level logs or promoted/latest logs. | -| `config/` | Optional session-level generated config snapshots or promoted/latest configs. | -| `current/` | Current published session pointers, mirroring the archive backend. | - -Canonical durable outputs should use stable paths under these directories. - -### 3.3 Run History Directory - -Run history lives under: - -```text -{workspace.root}/work/{campaign_id}/{session_id}/runs/{run_id}/ -``` - -Each run directory records what happened during that invocation. A run may contain all stage directories or only a sparse subset. - -Example full run: - -```text -runs/20260517T174748Z-abcd1234/ - manifest.json - prepare/ - transcribe/ - merge/ - polish/ - normalize/ - trim/ - analyze/ - archive/ - notify/ -``` - -Example sparse forced analyze run: - -```text -runs/20260518T030000Z-efgh5678/ - manifest.json - analyze/ -``` - -Example sparse polish-through-analyze rerun: - -```text -runs/20260518T041500Z-a1b2c3d4/ - manifest.json - polish/ - normalize/ - trim/ - analyze/ -``` - -Run directories should not create stage folders for stages that were not selected, executed, skipped, or otherwise considered during that run unless there is a clear diagnostic reason to do so. - -### 3.4 Stage Run-Local Directories - -Each stage receives a run-local directory: - -```text -runs/{run_id}/{stage}/ -``` - -Within that stage directory, the stage may use subdirectories such as: - -```text -outputs/ -logs/ -reports/ -config/ -scratch/ -``` - -For example: - -```text -runs/{run_id}/polish/ - outputs/transcripts/processed.json - reports/audita.polish.report.json - logs/stdout.log - logs/stderr.log - config/audita.polish.generated.yml - scratch/ -``` - -The exact internal layout of a stage directory may vary by stage, but it should be deterministic, documented, and generated through centralized path helpers rather than ad hoc path joins. - -## 4. Promotion Model - -Narratio uses stage-level promotion with immediate promotion after successful validation. - -The stage lifecycle is: - -1. Resolve required inputs from the current session state and/or run-local context. -2. Create the run-local stage directory. -3. Execute the stage, writing outputs under `runs/{run_id}/{stage}/...`. -4. Validate run-local outputs. -5. Promote durable outputs into session-level canonical paths. -6. Update the session manifest. -7. Update the run manifest. - -Promotion means an atomic or effectively atomic copy/rename from a run-local path to a session-level canonical path. - -Example: - -```text -runs/{run_id}/polish/outputs/transcripts/processed.json -``` - -is promoted to: - -```text -transcripts/processed.json -``` - -Promotion should be safe and deterministic: - -* Validate before promotion. -* Write promoted files atomically where possible. -* Never leave partially written durable outputs. -* Record the producing `run_id` in the session manifest. -* Preserve run-local files for debugging unless retention policy deletes them. - -## 5. Promotion Policy: Option A - -Narratio uses immediate stage-level promotion. - -If a selected stage succeeds, its durable outputs are promoted immediately, even if a later selected stage fails. - -Example: - -```bash -narratio run --session-id 2026-05-07 --force --stages polish,normalize,trim,analyze -``` - -If `polish` succeeds and `normalize` fails: - -* `transcripts/processed.json` may be updated from the new run. -* `normalize`, `trim`, and `analyze` should not be marked succeeded for the new input state. -* Downstream outputs may now be stale relative to the newly promoted polished transcript. - -This policy is simpler, transparent, and consistent with stage-level resumability. It does require explicit stale/invalidation handling. - -## 6. Stale and Invalidation Semantics - -Full checksum-based stale detection may be implemented later. Before that exists, Narratio should still use a simple deterministic invalidation rule for forced or explicit upstream reruns. - -When a stage is successfully re-executed and promoted, downstream stages should be marked stale unless they are also re-executed successfully in the same command invocation. - -Example stage order: - -```text -prepare -> transcribe -> merge -> polish -> normalize -> trim -> analyze -> archive -> notify -``` - -If `polish` is forced and promoted, then the following downstream stages should be invalidated unless rerun successfully: - -```text -normalize -trim -analyze -archive -notify -``` - -A stale stage is not equivalent to a failed stage. It means its current durable outputs may no longer correspond to current upstream inputs or configuration. - -Minimum manifest state model: - -```text -pending -running -succeeded -failed -skipped -stale -``` - -If adding a new `stale` state is too invasive for v1.0, the implementation should at least record stale metadata or clear downstream success markers in a way that prevents accidental idempotent skips based on obsolete outputs. - -## 7. Manifest Responsibilities - -Narratio should distinguish between session manifests and run manifests. - -The same underlying Go types may be reused where practical, but the concepts should remain separate. - -### 7.1 Session Manifest - -Path: - -```text -{workspace.root}/work/{campaign_id}/{session_id}/manifest.json -``` - -The session manifest answers: - -```text -What is the current durable state of this session? -``` - -It should record: - -* campaign ID -* session ID -* current or latest run ID -* current stage states -* canonical durable output refs -* artifact IDs and paths -* producing run ID for each current stage output -* relevant input/config checksums when available -* stale/invalidated stage information -* archive/current publication metadata - -A session's durable state may be a composite of multiple runs. - -For example: - -```text -transcripts/merged.json produced by run A -transcripts/processed.json produced by run B -transcripts/normalized.json produced by run B -transcripts/trimmed.json produced by run B -artifacts/session_recap.md produced by run C -``` - -This is valid and expected. - -### 7.2 Run Manifest - -Path: - -```text -{workspace.root}/work/{campaign_id}/{session_id}/runs/{run_id}/manifest.json -``` - -The run manifest answers: - -```text -What happened during this specific execution attempt? -``` - -It should record: - -* run ID -* campaign ID -* session ID -* command mode and selected stages -* force flags or stage selection flags -* stages considered during this run -* stages executed during this run -* stages skipped during this run and reasons -* run-local output paths -* promoted output paths -* logs -* reports -* generated configs -* timings -* errors -* non-secret subprocess invocation metadata - -Run manifests are primarily for debugging, auditability, and archive history. - -## 8. Idempotency and Resume Behavior - -The idempotency boundary is the session, not the run. - -By default: - -```bash -narratio run --session-id 2026-05-07 -``` - -should consult the session manifest and skip stages that are already succeeded and not stale. - -If all stages are already complete, the command should execute zero stages and report that the session is already complete. - -Forced execution creates a new run record but updates session-level durable state only for stages that actually succeed and promote outputs. - -Examples: - -```bash -narratio run --session-id 2026-05-07 --force -``` - -Creates a new run and attempts to re-execute the selected/default stage set. - -```bash -narratio run-stage --session-id 2026-05-07 analyze --force -``` - -Creates a sparse run that executes only `analyze`, then promotes updated analysis artifacts if successful. - -```bash -narratio resume --session-id 2026-05-07 -``` - -Uses the session manifest to determine what remains incomplete or stale. Resume does not need to resume the same `run_id` unless the implementation explicitly supports resuming an interrupted active run. - -## 9. Artifact Resolution Contract - -Narratio should provide a first-class artifact registry and resolver. - -The resolver maps symbolic artifact source names to canonical session-level paths and manifest output kinds. - -Stages and adapters should not hardcode path fragments when resolving cross-stage inputs. They should ask the artifact resolver for the current durable artifact by ID. - -### 9.1 Canonical Artifact IDs - -Preferred artifact IDs should be namespaced: - -```text -narratio.transcript.merged -narratio.transcript.polished -narratio.transcript.full -narratio.transcript.trimmed -narratio.bounds.session -narratio.artifact.session_recap -``` - -Recommended initial registry: - -| Artifact ID | Canonical Path | Producer Stage | Output Kind | Meaning | -| --------------------------------- | ------------------------------- | -------------- | ------------------------ | ------------------------------------- | -| `narratio.transcript.merged` | `transcripts/merged.json` | `merge` | `transcript_merged` | Deterministic Seriatim merge. | -| `narratio.transcript.polished` | `transcripts/processed.json` | `polish` | `transcript_processed` | Full Audita-polished transcript. | -| `narratio.transcript.full` | `transcripts/normalized.json` | `normalize` | `transcript_normalized` | Preferred full normalized transcript. | -| `narratio.transcript.trimmed` | `transcripts/trimmed.json` | `trim` | `transcript_trimmed` | Gameplay-only transcript. | -| `narratio.bounds.session` | `artifacts/session_bounds.json` | `trim` | `session_bounds` | Trim bounds selected for the session. | -| `narratio.artifact.session_recap` | `artifacts/session_recap.md` | `analyze` | `artifact_session_recap` | Generated session recap. | - -### 9.2 Backward-Compatible Aliases - -Existing source names should remain supported: - -| Legacy Source | Preferred Artifact ID | -| ----------------------- | ------------------------------ | -| `processed_transcript` | `narratio.transcript.polished` | -| `normalized_transcript` | `narratio.transcript.full` | -| `trimmed_transcript` | `narratio.transcript.trimmed` | - -These aliases may be supported silently for v1.0. Documentation should prefer namespaced IDs. - -### 9.3 Resolver Behavior - -Artifact resolution should follow this order: - -1. Normalize aliases to canonical artifact IDs. -2. Look for a current output reference in the session manifest. -3. Fall back to the canonical session-level path. -4. If the artifact is required, fail clearly if missing. -5. If the artifact is optional and missing, omit it from the downstream invocation. -6. Validate the artifact using the expected content validator. -7. Return a resolved artifact record containing ID, path, producer stage, output kind, and provenance. - -Example conceptual result: - -```json -{ - "id": "narratio.transcript.trimmed", - "path": "/var/lib/narratio/work/dilfs/2026-05-07/transcripts/trimmed.json", - "producer_stage": "trim", - "producer_run_id": "20260517T174748Z-abcd1234", - "output_kind": "transcript_trimmed", - "content_type": "application/json" -} -``` - -### 9.4 Artifact Validation - -Transcript artifacts must be valid JSON with a top-level `segments` array. - -Markdown/text artifacts must exist and be non-empty when required. - -Bounds artifacts must match the expected bounds schema and refer to segment IDs in the same transcript ID space used by the trim stage. - -Validation should happen before a resolved artifact is passed to another stage or external subprocess. - -## 10. Analyze Stage Implications - -The analyze stage should consume artifacts through the artifact resolver. - -Preferred Scriptorium config shape: - -```yaml -scriptorium: - artifacts: - session_recap: - enabled: true - prompt_id: "dnd.session_recap" - output_path: "artifacts/session_recap.md" - inputs: - transcript: - source: "narratio.transcript.trimmed" - required: true -``` - -Additional artifacts can choose different transcript tiers: - -```yaml -scriptorium: - artifacts: - table_summary: - enabled: true - prompt_id: "dnd.table_summary" - output_path: "artifacts/table_summary.md" - inputs: - transcript: - source: "narratio.transcript.full" - required: true -``` - -For v1.0, Narratio does not need a generic DAG engine. It may execute configured analyze artifacts in deterministic order and allow later artifacts to consume earlier artifacts only when that relationship is explicit and unambiguous. - -Rules: - -* Artifact inputs resolve from current session-level durable state. -* Outputs are first written run-locally. -* Successful analyze outputs are promoted to session-level `artifacts/` paths. -* Manifest output refs record the producing run ID. -* Optional inputs are omitted when unavailable. -* Required missing inputs fail before invoking Scriptorium. - -## 11. Archive Alignment - -Local workspace semantics should mirror archive semantics. - -Local session-level durable paths: - -```text -work/{campaign}/{session}/transcripts/trimmed.json -work/{campaign}/{session}/artifacts/session_recap.md -work/{campaign}/{session}/current/manifest.json -work/{campaign}/{session}/current/run_id.txt -work/{campaign}/{session}/runs/{run_id}/... -``` - -should map naturally to remote archive paths: - -```text -{root_prefix}/campaigns/{campaign}/sessions/{session}/transcripts/trimmed.json -{root_prefix}/campaigns/{campaign}/sessions/{session}/artifacts/session_recap.md -{root_prefix}/campaigns/{campaign}/sessions/{session}/current/manifest.json -{root_prefix}/campaigns/{campaign}/sessions/{session}/current/run_id.txt -{root_prefix}/campaigns/{campaign}/sessions/{session}/runs/{run_id}/... -``` - -The archive stage should publish run records and promoted current artifacts consistently with the local model. - -`current/run_id.txt` remains the effective commit marker for the archived current session state. - -## 12. Path Helper Requirements - -All code should use centralized path helpers for workspace paths. - -Stage code should not manually assemble durable cross-stage paths using raw string joins except through the path model. - -Recommended helper surface: - -```text -SessionRoot(campaignID, sessionID) -SessionManifestPath(campaignID, sessionID) -SessionCurrentDir(campaignID, sessionID) -SessionTranscriptsDir(campaignID, sessionID) -SessionArtifactsDir(campaignID, sessionID) -SessionReportsDir(campaignID, sessionID) -SessionLogsDir(campaignID, sessionID) -SessionConfigDir(campaignID, sessionID) -RunsDir(campaignID, sessionID) -RunRoot(campaignID, sessionID, runID) -RunManifestPath(campaignID, sessionID, runID) -RunStageDir(campaignID, sessionID, runID, stage) -RunStageOutputsDir(campaignID, sessionID, runID, stage) -RunStageLogsDir(campaignID, sessionID, runID, stage) -RunStageReportsDir(campaignID, sessionID, runID, stage) -RunStageConfigDir(campaignID, sessionID, runID, stage) -CanonicalArtifactPath(campaignID, sessionID, artifactID) -``` - -Path helpers should enforce safe relative paths for configured output paths: - -* reject absolute paths unless explicitly allowed for a particular config field -* reject `..` traversal -* normalize separators -* preserve deterministic output paths - -## 13. Directory Creation Policy - -Directory creation should be centralized and idempotent. - -Recommended policy: - -* `prepare` ensures the baseline session directory structure exists. -* Every stage also calls shared layout helpers to ensure its required run-local directories exist before writing. -* `run-stage` should not depend on a prior `prepare` invocation merely to create folders. -* Missing directories should be created with appropriate permissions. -* Directory creation should not imply stage success. - -This provides consistent layout while keeping direct stage execution robust. - -## 14. Cleanup and Retention - -Cleanup must preserve the distinction between durable session state and run history. - -Workspace cleanup after successful archive may remove selected local directories only according to explicit configuration. - -Potential retention policies: - -```text -keep_all_runs -keep_failed_runs -keep_last_n_runs -delete_run_after_success -``` - -For v1.0, conservative retention is preferred: - -* Do not delete durable session-level outputs unless explicitly requested. -* Do not delete failed run directories by default. -* If cleanup is enabled, remove only documented run-scoped or spool-scoped paths. -* Local development audio inputs must never be deleted by workspace cleanup. - -## 15. Canonical-Only Layout Policy - -Narratio now supports only the canonical campaign-aware layout: - -```text -{workspace.root}/work/{campaign_id}/{session_id}/manifest.json -{workspace.root}/work/{campaign_id}/{session_id}/runs/{run_id}/... -``` - -Legacy session-only layout compatibility is intentionally not implemented. - -If legacy workspace data exists, operators should recreate or manually migrate that data outside Narratio before running v1.0 commands. - -## 16. Documentation Updates Required - -The following documentation should be updated to reflect this architecture: - -* `README.md` -* `docs/architecture.md` -* a dedicated workspace/run-history document, such as this file -* S3/archive documentation -* analyze/artifact configuration documentation -* example pipeline files - -Documentation should consistently use the following terms: - -| Term | Meaning | -| ---------------- | ---------------------------------------------------------------------- | -| Session | Durable domain object and idempotency boundary. | -| Run | Execution attempt that may update session state. | -| Durable output | Canonical current session-level output. | -| Run-local output | Output produced inside a specific run directory before promotion. | -| Promotion | Validated copy/rename from run-local output to durable session output. | -| Session manifest | Current durable state of the session. | -| Run manifest | Execution record for a particular run. | -| Artifact ID | Symbolic source name resolved by the artifact registry. | - -## 17. Architectural Invariants - -The following invariants should hold after implementation: - -1. `session_id` remains the idempotency boundary for normal operator commands. -2. `run_id` identifies an execution attempt, not the primary durable workspace. -3. Session-level canonical artifacts are the default inputs for downstream stages. -4. Run-local outputs are promoted only after validation. -5. A session's current durable state may be composed of outputs from multiple runs. -6. Sparse run directories are valid and expected. -7. The session manifest records current stage/artifact state and producer run IDs. -8. The run manifest records what happened during one invocation. -9. Artifact consumers resolve symbolic artifact IDs through a registry/resolver. -10. Local workspace semantics mirror S3 archive semantics. -11. Directory creation is centralized and idempotent. -12. Stage code uses path helpers rather than ad hoc path construction. -13. Forced upstream reruns invalidate downstream stage success unless downstream stages are rerun successfully. -14. Cleanup never removes durable session outputs or local development inputs unless explicitly configured to do so. - -## 18. Implementation Guidance - -A practical implementation sequence is: - -1. Add this architecture document. -2. Add or revise path model helpers for session roots, run roots, stage directories, and canonical artifact paths. -3. Introduce session manifest versus run manifest concepts. -4. Route stage outputs through run-local directories. -5. Add promotion helpers with validation and atomic writes. -6. Update existing stages to promote durable outputs to session-level canonical paths. -7. Add artifact registry and resolver. -8. Update analyze to use artifact IDs and aliases. -9. Add simple downstream stale invalidation for forced upstream reruns. -10. Align archive/local path behavior and documentation. -11. Update examples and README. -12. Add tests for idempotency, sparse forced runs, promotion, manifest provenance, and artifact resolution. - -This sequence intentionally avoids introducing a generic DAG engine. The v1.0 goal is a clear, deterministic, stage-oriented orchestrator with stable session-level outputs and inspectable run history. diff --git a/docs/documentation/policy.md b/docs/documentation/policy.md new file mode 100644 index 0000000..f835c00 --- /dev/null +++ b/docs/documentation/policy.md @@ -0,0 +1,346 @@ +# Go Project Documentation Policy + +## Purpose + +Project documentation must help four audiences: + +1. users who need to run the application; +2. administrators/operators who need to configure and operate it; +3. developers who need to understand and change it safely; +4. LLM coding agents that need clear scope, boundaries, and invariants. + +Docs should be accurate, concise, task-oriented, and organized by audience. Prefer links to canonical docs over repetition. + +## Core Rules + +### 1. Keep docs concise + +Each document should cover a defined scope and only the essentials for that scope. + +Avoid: +- long background explanations; +- repeated reference material; +- implementation detail in user-facing docs; +- aspirational language outside roadmap docs; +- verbose examples where one minimal example is clearer. + +### 2. Document only implemented behavior outside roadmap files + +Unimplemented, planned, aspirational, experimental, or future work may be described only under: + +- `docs/roadmap/` + +No other documentation file, including `README.md`, should describe code, features, modules, stages, commands, config fields, or behaviors that do not currently exist. + +If a feature is partial, non-roadmap docs may describe only the implemented portion and its current boundary. + +### 3. Use canonical homes + +Each type of information should have one canonical location. + +Canonical homes: + +- project purpose and quickstart: `README.md` +- development principles: `docs/architecture.md` +- configuration reference: `docs/config.md` +- CLI reference: `docs/cli.md` +- operations and recovery: `docs/operations.md` +- troubleshooting: `docs/troubleshooting.md` +- implemented internals: `docs/internal/` +- future work: `docs/roadmap/` +- contributor workflow: `docs/development.md` +- copyable examples: `examples/` + +Other files should summarize briefly and link to the canonical source. + +### 4. Keep examples real + +Examples should be valid, maintained, and free of secrets. + +Where practical: +- example configs should load successfully; +- example commands should match real CLI syntax; +- important examples should be covered by tests. + +## Documentation Profiles + +All projects require: + +- `README.md` +- `docs/architecture.md` + +Additional docs depend on the project. + +### Small library + +Recommended: +- `docs/development.md`, if contributor conventions are non-obvious + +### Simple CLI + +Required: +- `docs/cli.md` + +Recommended: +- `docs/development.md` + +### Config-driven CLI + +Required: +- `docs/cli.md` +- `docs/config.md` + +Recommended: +- `examples/` +- `docs/development.md` + +### Stateful or operator-facing application + +Required: +- `docs/cli.md`, if CLI-based +- `docs/config.md`, if config-driven +- `docs/operations.md` + +Recommended: +- `docs/troubleshooting.md` +- `examples/` +- `docs/development.md` + +### Modular, staged, service-oriented, or orchestration application + +Required: +- `docs/cli.md`, if CLI-based +- `docs/config.md`, if config-driven +- `docs/operations.md` +- `docs/internal/` +- `docs/development.md` + +Recommended: +- `docs/troubleshooting.md` +- validated examples under `examples/` + +## Required Documents + +### README.md + +**Audience:** users, administrators, operators + +The README is the outward-facing project orientation page. + +It should include, in order: + +1. concise description; +2. elevator pitch; +3. shortest useful command or usage example; +4. links to targeted docs. + +The README should be short. It is not a manual. + +The “shortest useful command” means the simplest command that performs the project’s core use case. (It does not mean `app --help`.) + +### docs/architecture.md + +**Audience:** developers, LLM coding agents + +`docs/architecture.md` is required for every project. + +It is an inward-facing development policy document. It should describe how the project is intended to be built and changed. + +It should include: + +- project shape; +- core design principles; +- package and boundary philosophy; +- state/persistence philosophy, if applicable; +- external integration philosophy, if applicable; +- error-handling and logging principles; +- testing expectations; +- documentation expectations; +- architectural invariants; +- explicit non-goals, if useful. + +For small projects, this file may be brief. It may simply state that the project is intentionally narrow, monolithic, and dependency-light. + +### docs/config.md + +**Audience:** administrators, operators, advanced users + +Required for applications with configuration files. + +It should include, in order: + +1. config file locations and discovery precedence; +2. minimal working config; +3. production-oriented config; +4. full configuration reference; +5. secrets handling, if applicable; +6. links to maintained examples. + +The full configuration reference should be canonical. + +### docs/cli.md + +**Audience:** users, administrators, operators + +Required for CLI applications. + +It should include, in order: + +1. shortest useful command; +2. command overview; +3. complete flag reference; +4. common workflows; +5. diagnostic or recovery commands, if applicable. + +Explain when commands are useful, not just their syntax. + +### docs/operations.md + +**Audience:** administrators, operators + +Required for applications that maintain state, support resume behavior, run multiple stages, write durable artifacts, use remote storage, or require recovery procedures. + +It should cover: + +- normal workflow; +- filesystem layout; +- remote storage layout, if applicable; +- logs and manifests; +- resume/retry behavior; +- cleanup behavior; +- archive/backup behavior; +- safe recovery procedures; +- operational caveats. + +### docs/troubleshooting.md + +**Audience:** administrators, operators + +Recommended once recurring failure modes exist. + +Each entry should include: + +- symptom; +- likely cause; +- diagnostic command or inspection step; +- safe fix; +- relevant links. + +### docs/development.md + +**Audience:** developers, LLM coding agents + +Required for projects maintained by humans and LLM coding agents. + +It should include: + +- repository layout; +- build/test commands; +- coding conventions; +- dependency policy; +- how to add config fields; +- how to add CLI flags; +- how to add stages/modules/adapters, if applicable; +- how to update examples; +- documentation update expectations. + +### docs/internal/ + +**Audience:** developers, LLM coding agents + +Required for modular, staged, service-oriented, or orchestration projects. + +This directory describes implemented internal components. It is not the roadmap. + +Use one file per major component where useful. + +Each component doc should include: + +1. purpose; +2. inputs and outputs; +3. boundaries; +4. config fields used; +5. external adapters used; +6. state or manifest behavior, if applicable; +7. skip/resume behavior, if applicable; +8. failure behavior; +9. tests to inspect before changing; +10. architectural invariants. + +### docs/roadmap/ + +**Audience:** maintainers, developers, LLM coding agents + +This is the only place for planned, future, aspirational, experimental, or unimplemented work. + +Roadmap docs should clearly distinguish: + +- proposed work; +- accepted plans; +- deferred ideas; +- rejected ideas; +- implementation prompts or task breakdowns, if useful. + +Roadmap docs should not be confused with current behavior. + +## Examples Directory + +Projects with non-trivial configuration or workflows should include `examples/`. + +Useful examples include: + +- minimal working config; +- production-oriented config; +- full annotated config; +- local development config; +- remote/object-storage config; +- minimal session/input file. + +Examples should be valid, maintained, tested when practical, and linked from relevant docs. + +## Security and Privacy + +Docs and examples must not include: + +- real API keys; +- tokens; +- passwords; +- private keys; +- private environment dumps; +- sensitive user data; +- raw private transcripts; +- private infrastructure details unless intentionally public. + +Document secret-handling mechanisms, not actual secret values. + +## Maintenance Rules + +When docs change, verify the affected behavior. + +Where practical: + +- load example config files in tests; +- test CLI examples or command parser behavior; +- validate documented flags against real flags; +- remove stale references; +- update links after renames; +- keep roadmap content out of non-roadmap docs. + +If documentation and code disagree, fix the documentation and/or open a roadmap item; do not leave aspirational behavior in current-behavior docs. + +Documentation is complete only when it matches the current code. + +## Documentation Change Checklist + +Before merging documentation changes, verify: + +- README is concise and orientation-focused. +- `docs/architecture.md` describes development principles. +- Future work appears only under `docs/roadmap/`. +- User-facing docs avoid unnecessary internals. +- Developer-facing docs preserve boundaries and invariants. +- Config examples match the schema. +- CLI examples match real commands and flags. +- Defaults appear in the canonical config reference. +- No secrets or private data are included. +- Links are accurate. diff --git a/docs/examples/pipeline.full.annotated.yml b/docs/examples/pipeline.full.annotated.yml new file mode 100644 index 0000000..4dd7b79 --- /dev/null +++ b/docs/examples/pipeline.full.annotated.yml @@ -0,0 +1,155 @@ +# Full annotated pipeline example for implemented Narratio config fields. +# Values are safe placeholders and must be adapted per environment. + +workspace: + # Required: local workspace root. + root: ./tmp/narratio-workspace + # Optional: remove run-scoped workdir after successful archive commit. + cleanup_after_archive: false + +# Optional: local secret file loader (directory of ENV_VAR_NAME files). +# secrets: +# env_dir: ./secrets + +storage: + # Optional storage backend selector; use "s3" for archive + S3 audio workflows. + backend: s3 + # Legacy fields retained in schema for compatibility. + bucket: "" + prefix: "" + s3: + # Required when using S3 audio or S3 archive uploads. + bucket: my-dnd-archive + # Optional; defaults to "dnd". + root_prefix: dnd + # Optional region/endpoint settings. + region: us-east-1 + endpoint: "" + force_path_style: false + # Optional; defaults shown explicitly. + access_key_id_env: OBJECT_STORAGE_KEY_ID + secret_access_key_env: OBJECT_STORAGE_KEY + +spool: + # Optional; defaults to /var/spool/narratio. + root: /var/spool/narratio + # Optional cleanup of run-scoped spool audio after successful archive commit. + delete_audio_after_archive: false + +archive: + # Optional booleans; defaults are true. + enabled: true + upload_run: true + # Optional promotions; defaults shown explicitly. + promote_artifacts: + - from: transcripts/trimmed.json + to: transcripts/trimmed.json + required: true + - from: artifacts/session_recap.md + to: artifacts/session_recap.md + required: true + +whisperx: + # Required. + transcribe_url: "https://transcription.example.com/transcribe" + # Optional overrides; defaults shown explicitly. + language: en + timeout: 30m + retries: 3 + retry_delay: 2s + concurrency: 2 + +seriatim: + # Optional overrides; defaults shown explicitly. + binary: seriatim + timeout: 10m + output_schema: seriatim-intermediate + coalesce_gap: 3.0 + report: true + env: + # Optional advanced tuning; set only when needed. + overlap_word_run_gap: 1.0 + overlap_word_run_reorder_window: 1.0 + backchannel_max_duration: 2.0 + filler_max_duration: 1.25 + +audita: + # Optional overrides; defaults shown explicitly where applicable. + binary: audita + timeout: 3h + llm_api_key_env: AUDITA_LLM_API_KEY + modules: [glossary, homophones, spoken_word, grammar] + base_url: "" + model: "" + total_llm_concurrency: 2 + proposal_llm_concurrency: 1 + validation_model: "" + validation_llm_concurrency: 1 + transcript_description: "" + config_path: /usr/local/etc/audita/config.yml + output_schema: audita-v1 + work_dir_retention: auto + report: true + +normalize: + # Optional; defaults shown explicitly. + output_path: transcripts/normalized.json + output_schema: seriatim-intermediate + report: true + +trim: + # Keep disabled unless bounds prompt integration is configured. + enabled: false + output_path: transcripts/trimmed.json + bounds: + prompt_id: dnd.session_bounds + profile_id: local-fast + transcript_input_name: transcript + output_path: reports/session_bounds.json + timeout: 10m + render_debug: false + render_output_path: reports/session_bounds.render.json + seriatim: + report: false + +scriptorium: + binary: scriptorium + config_path: /usr/local/etc/scriptorium/config.yml + timeout: 10m + render_debug: false + artifacts: + session_recap: + enabled: true + prompt_id: dnd.session_recap + profile_id: local-fast + output_path: artifacts/session_recap.md + timeout: 10m + inputs: + transcript: + source: narratio.transcript.trimmed + required: true + previous_recap: + source: previous_session_artifact + artifact: session_recap + path: "" + required: false + vars: + session_id: true + session_date: true + campaign_name: true + previous_session_id: true + output_kind: session_recap + +analyzer: + # Optional adapter settings. + binary_path: "" + timeout: 2m + artifacts: + output_dir: "" + types: [] + +notification: + # Optional notification settings. + backend: "" + recipient: "" + timeout: 30s diff --git a/docs/examples/pipeline.minimal.yml b/docs/examples/pipeline.minimal.yml new file mode 100644 index 0000000..05c3823 --- /dev/null +++ b/docs/examples/pipeline.minimal.yml @@ -0,0 +1,5 @@ +workspace: + root: ./tmp/narratio-workspace + +whisperx: + transcribe_url: "https://transcription.example.com/transcribe" diff --git a/docs/examples/pipeline.production.yml b/docs/examples/pipeline.production.yml new file mode 100644 index 0000000..be693e6 --- /dev/null +++ b/docs/examples/pipeline.production.yml @@ -0,0 +1,95 @@ +workspace: + root: /var/lib/narratio/workspace + cleanup_after_archive: true + +storage: + backend: s3 + s3: + bucket: my-dnd-archive + root_prefix: dnd + region: us-east-1 + access_key_id_env: OBJECT_STORAGE_KEY_ID + secret_access_key_env: OBJECT_STORAGE_KEY + +spool: + root: /var/spool/narratio + delete_audio_after_archive: true + +archive: + enabled: true + upload_run: true + promote_artifacts: + - from: transcripts/trimmed.json + to: transcripts/trimmed.json + required: true + - from: artifacts/session_recap.md + to: artifacts/session_recap.md + required: true + +whisperx: + transcribe_url: "https://transcription.example.com/transcribe" + language: en + timeout: 45m + retries: 3 + retry_delay: 3s + concurrency: 2 + +seriatim: + binary: seriatim + timeout: 10m + output_schema: seriatim-intermediate + coalesce_gap: 3.0 + report: true + +audita: + binary: audita + timeout: 3h + llm_api_key_env: AUDITA_LLM_API_KEY + modules: [glossary, homophones, spoken_word, grammar] + output_schema: audita-v1 + work_dir_retention: auto + total_llm_concurrency: 2 + proposal_llm_concurrency: 1 + validation_llm_concurrency: 1 + report: true + +normalize: + output_path: transcripts/normalized.json + output_schema: seriatim-intermediate + report: true + +trim: + enabled: false + +scriptorium: + binary: scriptorium + config_path: /usr/local/etc/scriptorium/config.yml + timeout: 10m + render_debug: false + artifacts: + session_recap: + enabled: true + prompt_id: dnd.session_recap + profile_id: local-fast + output_path: artifacts/session_recap.md + timeout: 10m + inputs: + transcript: + source: narratio.transcript.trimmed + required: true + previous_recap: + source: previous_session_artifact + artifact: session_recap + required: false + vars: + session_id: true + session_date: true + campaign_name: true + previous_session_id: true + output_kind: session_recap + +analyzer: + timeout: 2m + +notification: + timeout: 30s diff --git a/docs/examples/session.local-audio.yml b/docs/examples/session.local-audio.yml new file mode 100644 index 0000000..56d6c2a --- /dev/null +++ b/docs/examples/session.local-audio.yml @@ -0,0 +1,9 @@ +session_id: 2026-05-03 +campaign: sample-campaign +date: 2026-05-03 +title: Sample Session +inputs: + audio_dir: ./audio + speakers_file: ./examples/speakers.yml + autocorrect_file: ./examples/autocorrect.yml + glossary_file: ./examples/glossary.yml diff --git a/docs/examples/session.s3-audio.yml b/docs/examples/session.s3-audio.yml new file mode 100644 index 0000000..88286e1 --- /dev/null +++ b/docs/examples/session.s3-audio.yml @@ -0,0 +1,10 @@ +session_id: 2026-05-03 +campaign: sample-campaign +date: 2026-05-03 +title: Sample Session +inputs: + audio_s3: + prefix: audio/ + speakers_file: ./examples/speakers.yml + autocorrect_file: ./examples/autocorrect.yml + glossary_file: ./examples/glossary.yml diff --git a/docs/examples/session.template.yml b/docs/examples/session.template.yml new file mode 100644 index 0000000..cd19f81 --- /dev/null +++ b/docs/examples/session.template.yml @@ -0,0 +1,7 @@ +session_id: "{{ session_id }}" +campaign: sample-campaign +inputs: + audio_dir: ./audio + speakers_file: ./examples/speakers.yml + autocorrect_file: ./examples/autocorrect.yml + glossary_file: ./examples/glossary.yml diff --git a/docs/integrations/audita.md b/docs/integrations/audita.md deleted file mode 100644 index d94aeef..0000000 --- a/docs/integrations/audita.md +++ /dev/null @@ -1,96 +0,0 @@ -# Audita Subprocess Operations - -This document describes how parent processes should invoke `audita process` safely in production orchestration. - -## Recommended command form - -Use explicit file outputs for orchestrated runs: - -```sh -audita process \ - --transcript-description "Brief context that may help resolve ambiguous terms." \ - --glossary \ - --output \ - --report-json -``` - -Additional flags that may be situationally appropriate: -- `--config ` to select an explicit versioned config file. -- `--output-schema ` to select transcript output shape. -- `--work-dir ` to control diagnostics location. -- `--work-dir-retention ` to control retained run directories. -- `--total-llm-concurrency`, `--proposal-llm-concurrency`, and `--validation-llm-concurrency` when orchestration needs to set explicit LLM throughput controls. -- `--modules ...` only when intentionally overriding the default sequence. - -For config-driven orchestration, validate config files in CI/preflight: - -```sh -audita config validate --config -``` - -## Stdout behavior - -- With `--output`: stdout is expected to be empty on success. -- Without `--output`: stdout contains transcript JSON only on success. -- Report JSON is never written to stdout. - -## Stderr behavior - -- Success path should be quiet or minimal human-readable logs. -- Failure path writes concise human-readable errors. -- When a diagnostics run directory exists, failure stderr includes its path. -- Prompt/response diagnostic payloads are not streamed to stderr. - -## Output file behavior - -- `--output` writes transcript JSON in the selected output schema to the provided path. -- Output write failures return nonzero and surface actionable errors. -- The command does not silently ignore output write errors. - -## Report JSON behavior - -- `--report-json` writes a machine-readable process report to the requested path. -- Run-directory `report.json` is written independently under diagnostics. -- Best-effort failure reports are emitted when possible without masking the primary failure. -- Report write failures return nonzero with clear stderr messaging. -- Report diagnostics metadata references run-directory artifacts including utilization diagnostics and correction ledger paths when available. - -## Diagnostics directory behavior - -- Each run creates (when possible) a per-run diagnostics directory. -- Typical artifacts include transcript, normalization, chunking, invocation, effective config, LLM diagnostics, `utilization-diagnostics.json`, `correction-ledger.json`, `report.json`, and `error.log` on failure. -- Failed runs retain diagnostics. -- Under `auto` retention, successful runs with skipped/rejected corrections are retained; clean successful runs may be removed. - -## Exit codes - -- `0`: success. -- Nonzero: failure (input/schema/config/module/LLM/runtime/output/report/diagnostics errors). - -Treat any nonzero as a failed subprocess invocation. - -## Timeout and cancellation - -- Runtime operations propagate context cancellation and request timeouts through LLM/scheduler paths. -- On cancellation or timeout, the process exits nonzero and should not hang. -- If diagnostics were initialized before failure, failure artifacts remain available for debugging. - -## Secret redaction expectations - -API keys and configured secret values are redacted from: -- reports (`--report-json` and run-dir `report.json`); -- diagnostics artifacts (including effective config and LLM interaction artifacts); -- surfaced adapter/runtime errors; -- test fixtures and regression outputs. - -Parent-process logs should still avoid printing raw environment variables. - -## Parent-process pipe guidance - -To avoid deadlocks in orchestrators: -- always read both stdout and stderr concurrently when invoking as a subprocess; -- prefer file outputs (`--output`, `--report-json`) for machine workflows; -- treat stderr as human-readable diagnostics, not structured data; -- parse structured results from output/report files. - -For Go callers, prefer `exec.CommandContext` with explicit timeout/cancellation and buffered/streamed readers for both pipes. diff --git a/docs/integrations/scriptorium.md b/docs/integrations/scriptorium.md deleted file mode 100644 index da3628a..0000000 --- a/docs/integrations/scriptorium.md +++ /dev/null @@ -1,339 +0,0 @@ -# Narratio -> Scriptorium CLI Integration - -## 1. Purpose - -This document defines how Narratio should invoke Scriptorium through the **public CLI**. - -This is a **subprocess integration contract**, not an internal Go API contract. - -## 2. Assumptions - -- `scriptorium` is installed and available on `PATH`. -- Scriptorium is configured with `config.yml`. -- `config.yml` provides `prompt_dir`, `profile_dir`, and `schema_dir` as needed. -- Prompt and profile libraries are already deployed for the environment. -- Narratio provides prepared artifact files (for example polished transcript, glossary, previous recap, campaign notes). -- Initial integration is synchronous subprocess execution. -- Narratio remains the orchestrator. - -In normal operation, Narratio does not need to pass `--prompt-dir` and `--profile-dir` if they are supplied by Scriptorium config. - -Narratio may pass `--config ` when it must use a non-default Scriptorium config file. - -## 3. Core Commands Narratio May Call - -Primary commands for subprocess integration: - -- `scriptorium run` -- `scriptorium render` - -For production generation, use `scriptorium run`. - -`scriptorium render` is for debugging, dry-runs, test assertions, and validating command construction without LLM execution. - -Note: `scriptorium serve` and HTTP API exist, but they are not the initial integration path. - -## 4. Command Selection Guidance - -- Use `run` to generate an output artifact. -- Use `render` to inspect the prepared prompt and effective settings without calling the LLM. -- Use `render --format json` when Narratio/tests need structured prepare output. - -## 5. Recommended `run` Invocation Shape - -Production shape: - -```bash -scriptorium run \ - --prompt \ - --input transcript= \ - --out -``` - -Common optional additions: - -- `--config `: use a specific Scriptorium config file. -- `--profile `: override prompt default profile. -- `--var name=value` (repeatable): small metadata values. -- `--input name=path` (repeatable): additional named artifacts. -- `--timeout `: per-run timeout override. -- Runtime model override flags (`--llm-base-url`, `--model`, etc.) only for exceptional/operator-directed cases. - -## 6. Recommended `render` Invocation Shape - -Human-readable debug shape: - -```bash -scriptorium render \ - --prompt \ - --input transcript= \ - --format text -``` - -Structured debug/test shape: - -```bash -scriptorium render \ - --prompt \ - --input transcript= \ - --format json \ - --out -``` - -`render` does **not** call the LLM, does **not** validate model output, and does **not** perform repair. - -## 7. Inputs - -- Pass inputs as repeated `--input name=path` flags. -- `name` must match the Prompt Definition input name. -- Prefer absolute paths, or paths relative to a working directory controlled by Narratio. -- Pass Audita output as the primary transcript input. -- Additional inputs may include glossary, previous recap, campaign notes, event logs, final state maps, or other prompt-specific artifacts. -- Scriptorium reads input files directly; Narratio does not need to inline file content for CLI use. - -## 8. Variables - -Use repeated `--var name=value` for small metadata values. - -Typical examples: - -- `session_date` -- `session_id` -- `campaign_name` -- `previous_session_id` -- `output_kind` - -Large content belongs in input files, not `--var` values. - -## 9. Prompt IDs and Output Artifact Types - -Narratio should treat prompt IDs as configuration, not hardcoded business logic. - -Narratio config may map stage/output names to prompt IDs, for example: - -- session recap prompt -- structured event extraction prompt -- glossary suggestion prompt -- player-facing summary prompt - -Prompt IDs used by Narratio should come from the deployed Scriptorium prompt library. - -## 10. Profiles - -- Prompts may declare `default_profile`. -- Narratio may omit `--profile` to use prompt default profile. -- Narratio may pass `--profile` to force profile selection. -- This enables environment/profile selection like `local-fast`, `local-quality`, `frontier`, `batch`, or test profiles. -- Profile names should generally be Narratio configuration values. - -## 11. Runtime Overrides - -Supported runtime override flags: - -- `--llm-base-url` -- `--model` -- `--api-key-env` -- `--temperature` -- `--max-tokens` -- `--top-p` -- `--timeout` - -Guidance: - -- Keep normal model/runtime settings in Execution Profiles. -- Use runtime overrides only for explicit per-run exceptions, tests, or operator overrides. -- Never pass raw API keys on the command line. -- `--api-key-env` names an environment variable; Narratio must ensure that variable is set in subprocess environment. - -## 12. Config Behavior - -- Default config path: `/etc/scriptorium/config.yml`. -- `--config ` overrides default path. -- Missing default config is allowed by Scriptorium. -- If `--config` is provided explicitly, the file must exist and be valid. -- CLI flags override `config.yml`. -- `config.yml` overrides built-in application defaults. - -Narratio can either: - -- rely on system default config path, or -- carry an explicit config path and pass `--config`. - -## 13. Environment Handling - -Subprocess environment recommendations: - -- Pass through required API-key environment variables referenced by `api_key_env`. -- Do not pass raw API keys as CLI arguments. -- Avoid logging full environment dumps. -- Capture stdout and stderr separately. -- Use a controlled working directory. -- Prefer absolute artifact paths. - -## 14. Output Handling - -For `scriptorium run`: - -- Use `--out` when Narratio needs durable artifact files. -- Without `--out`, artifact content is written to stdout. -- Preferred orchestration pattern: always use `--out`, then treat the file as stage output artifact. -- Capture stderr for diagnostics. - -For `scriptorium render`: - -- Use `--out` to store render diagnostics. -- Use `--format json` when tests need to inspect selected profile, effective runtime settings, input hashes, prompt hash, and rendered messages. - -## 15. Exit Status and Errors - -Current CLI behavior (verified from implementation/tests): - -- `0`: success. -- `1`: runtime/parse/config/load/render/generation/IO error. -- `2`: run completed but output validation failed (`ValidationFailed`). - -Additional details: - -- On `run`, output artifact write happens before exit code selection. If validation fails, artifact may still be written and exit code is `2`. -- `stderr` carries both errors and normal run summary output; non-empty stderr alone does not imply failure. -- `render` returns `0` on success and `1` on failures. - -Narratio should treat non-zero exit codes as failed stage execution, but may record generated artifact paths if a run exited `2` and output file exists. - -## 16. Recommended Narratio Integration Pattern - -1. Build CLI args from Narratio stage configuration. -2. Use subprocess context cancellation/timeout. -3. Pass absolute input paths. -4. Pass `--out` to a session-scoped artifact path. -5. Add `--var` metadata values. -6. Optionally add `--config`. -7. Optionally add `--profile`. -8. Ensure required API-key env vars are present. -9. Run subprocess synchronously. -10. Capture stdout/stderr separately. -11. On success, store output artifact path and invocation metadata in stage artifacts. -12. On failure, store exit code and stderr diagnostics in stage status. - -## 17. Suggested Narratio Configuration Shape - -Illustrative `pipeline.yml` shape: - -```yaml -scriptorium: - binary: scriptorium - config_path: /etc/scriptorium/config.yml - timeout: 10m - render_debug: false - artifacts: - session_recap: - enabled: true - prompt_id: dnd.session_recap - profile_id: local-quality # optional - output_path: artifacts/session_recap.md - timeout: 10m - render_debug: false # optional artifact override - inputs: - transcript: - source: trimmed_transcript - required: true - previous_recap: - source: previous_session_artifact - artifact: session_recap - path: "" # optional - required: false - vars: - session_id: true - session_date: true - campaign_name: true - previous_session_id: true - output_kind: session_recap -``` - -The key idea: map Narratio artifact names to prompt ID, optional profile, expected inputs, vars, and output destination. - -## 18. Testing Strategy for Narratio Integration - -- Use `scriptorium render --format json` to verify command construction without LLM calls. -- Use dedicated test prompt/profile libraries for integration tests. -- Use small fixture transcripts. -- Verify missing-input failure behavior. -- Verify prompt `default_profile` behavior. -- Verify explicit `--profile` override behavior. -- Verify `--config` behavior (default and explicit). -- Verify output file creation when `--out` is used. -- Verify stderr capture on failures. -- Avoid real API keys in tests. - -## 19. Security and Privacy Notes - -- Never pass raw API keys on command line. -- Do not log full rendered prompts by default; transcripts may contain sensitive content. -- Avoid logging prompt content unless explicit debug mode is enabled. -- Treat generated artifacts as potentially sensitive. -- Use session-scoped, access-controlled output paths. -- `api_key_env` names should come from environment management, not embedded secrets. - -## 20. Initial D&D Artifact Generation Examples - -These are examples only. Use prompt IDs from the deployed prompt library. - -Session recap: - -```bash -scriptorium run \ - --prompt dnd.session_recap \ - --input transcript=/work/campaign-7/session-42/transcript.polished.md \ - --input glossary=/work/campaign-7/session-42/glossary.yml \ - --out /work/campaign-7/session-42/artifacts/session_recap.md -``` - -Structured events: - -```bash -scriptorium run \ - --prompt dnd.structured_events \ - --input transcript=/work/campaign-7/session-42/transcript.polished.md \ - --out /work/campaign-7/session-42/artifacts/structured_events.json -``` - -Glossary suggestions: - -```bash -scriptorium run \ - --prompt dnd.glossary_suggestions \ - --input transcript=/work/campaign-7/session-42/transcript.polished.md \ - --input previous_recap=/work/campaign-7/session-41/artifacts/session_recap.md \ - --out /work/campaign-7/session-42/artifacts/glossary_suggestions.md -``` - -Player-facing summary: - -```bash -scriptorium run \ - --prompt dnd.player_summary \ - --input transcript=/work/campaign-7/session-42/transcript.polished.md \ - --input structured_events=/work/campaign-7/session-42/artifacts/structured_events.json \ - --out /work/campaign-7/session-42/artifacts/player_summary.md -``` - -## 21. Non-Goals - -Initial Narratio integration should not: - -- call Scriptorium internal Go packages -- use HTTP API as the primary path -- expect Scriptorium to read S3 refs directly -- make Scriptorium responsible for Narratio stage state -- make Scriptorium responsible for notification -- require Scriptorium to understand D&D workflow semantics beyond prompt definitions - -## 22. Future Extension Notes - -Possible later extensions: - -- HTTP API integration -- S3 artifact references if Scriptorium adds S3 reader support -- richer render diagnostics and policy controls -- token budgeting/prompt-size checks -- batch execution if Scriptorium later adds batch support diff --git a/docs/integrations/seriatim.md b/docs/integrations/seriatim.md deleted file mode 100644 index acb3bbc..0000000 --- a/docs/integrations/seriatim.md +++ /dev/null @@ -1,403 +0,0 @@ -# seriatim - -`seriatim` merges per-speaker WhisperX-style JSON transcripts into a single JSON transcript that preserves speaker identity and chronological order. - -The current implementation supports the `merge` command. It reads one or more input JSON files, optionally maps each input file to a canonical speaker using `speakers.yml`, sorts all segments by timestamp, detects and resolves overlaps when word-level timing is available, assigns consecutive numeric `id` values, and writes a merged JSON artifact. - -## Usage - -Run from source: - -```sh -go run ./cmd/seriatim merge \ - --input-file samples/raw/2026-04-19-Eric_Rakestraw.json \ - --input-file samples/raw/2026-04-19-Mike_Brown.json \ - --output-file merged.json -``` - -Optional report output: - -```sh -go run ./cmd/seriatim merge \ - --input-file eric.json \ - --input-file mike.json \ - --output-file merged.json \ - --report-file report.json -``` - -## CLI - -```text -seriatim merge [flags] -``` - -Global flags: - -| Flag | Description | -| --- | --- | -| `--help` | Show command help. | -| `--version` | Show application version. Local builds default to `dev`; release builds inject the release version. | - -`merge` flags: - -| Flag | Required | Default | Description | -| --- | --- | --- | --- | -| `--input-file` | Yes | none | Input transcript JSON file. Repeat once per speaker/input file. | -| `--output-file` | Yes | none | Merged transcript JSON output path. | -| `--report-file` | No | none | Optional report JSON output path. | -| `--speakers` | No | none | Speaker map YAML file. When omitted, input file basenames are used as speaker labels. | -| `--autocorrect` | No | none | Autocorrect rules YAML file. When omitted, the default `autocorrect` module leaves text unchanged. | -| `--input-reader` | No | `json-files` | Input reader module. | -| `--output-modules` | No | `json` | Comma-separated output modules. | -| `--output-schema` | No | `seriatim-intermediate` | JSON output contract. Allowed values are `seriatim-minimal`, `seriatim-intermediate`, and `seriatim-full`. If omitted, the runtime default is used; consumers that depend on a specific shape should set this explicitly. | -| `--preprocessing-modules` | No | `validate-raw,normalize-speakers,trim-text` | Comma-separated preprocessing modules, evaluated in order. | -| `--postprocessing-modules` | No | `detect-overlaps,resolve-overlaps,backchannel,filler,resolve-danglers,coalesce,detect-overlaps,autocorrect,assign-ids,validate-output` | Comma-separated postprocessing modules, evaluated in order. | -| `--coalesce-gap` | No | `3.0` | Maximum same-speaker gap in seconds for `coalesce`; also used as the `resolve-overlaps` context window. Must be a non-negative float. | - -Environment variables: - -| Environment Variable | Default | Description | -| --- | --- | --- | -| `SERIATIM_OUTPUT_SCHEMA` | `seriatim-intermediate` | Output schema used when `--output-schema` is not explicitly provided. Allowed values are `seriatim-minimal`, `seriatim-intermediate`, and `seriatim-full`. The CLI flag takes precedence. | -| `SERIATIM_OVERLAP_WORD_RUN_GAP` | `1.0` | Maximum gap in seconds between adjacent timed words when `resolve-overlaps` builds word-run replacement segments. Must be a positive float. | -| `SERIATIM_OVERLAP_WORD_RUN_REORDER_WINDOW` | `1.0` | Near-start window in seconds for ordering replacement word runs shortest-first. Must be a positive float. | -| `SERIATIM_BACKCHANNEL_MAX_DURATION` | `2.0` | Maximum duration in seconds for `backchannel` classification. Must be a positive float. | -| `SERIATIM_FILLER_MAX_DURATION` | `1.25` | Maximum duration in seconds for `filler` classification. Must be a positive float. | - -## Input JSON Format - -Each input file must be valid JSON with a top-level `segments` array. The current parser accepts the WhisperX segment subset needed for merging: - -```json -{ - "segments": [ - { - "start": 1.25, - "end": 3.5, - "text": "Hello there.", - "words": [ - {"word": "Hello", "start": 1.25, "end": 1.55, "score": 0.98}, - {"word": "there.", "start": 1.7, "end": 2.0} - ] - } - ] -} -``` - -Required segment fields: - -- `start`: number, must be `>= 0`. -- `end`: number, must be `>= start`. -- `text`: string. - -Optional word fields: - -- `words`: array of word timing objects. -- `words[].word`: string. -- `words[].start`: optional number, must be `>= 0` when present. -- `words[].end`: optional number, must be `>= start` when present with `start`. -- `words[].score`: optional number. -- `words[].speaker`: optional raw speaker label string. - -Word-level timing is preserved internally for overlap resolution. If a word is missing `start` or `end`, seriatim keeps the word text, emits a warning in the optional report, and does not use that word as a timing anchor. Word timing is not emitted in the final JSON artifact. - -## Speaker Map Format - -`speakers.yml` maps input files to canonical speaker names using ordered substring rules: - -This file is optional. If `--speakers` is omitted, `seriatim` uses each input file basename as the segment speaker label. - -```yaml -match: - - speaker: "Eric Rakestraw" - match: - - "Eric_Rakestraw" - - "Eric" - - - speaker: "Mike Brown" - match: - - "Mike_Brown" - - "mb" -``` - -For each `--input-file`, `seriatim` takes the file basename and evaluates the rules in order. The first rule with a matching substring wins, and no later rules are evaluated. - -For example, this input: - -```text -samples/raw/2026-04-19-Eric_Rakestraw.json -``` - -matches this rule because the basename contains `Eric_Rakestraw`: - -```yaml -- speaker: "Eric Rakestraw" - match: - - "Eric_Rakestraw" -``` - -Important details: - -- Matching is against the input file basename, not the full path. -- Matching is case-insensitive. -- Rules are evaluated from first to last. -- Each rule must have a non-empty `speaker`. -- Each rule must have at least one non-empty `match` string. -- Duplicate speaker names are invalid. -- Every input file must match at least one rule or the command fails. - -Deprecated old format: - -```yaml -inputs: - eric.json: - speaker: "Eric Rakestraw" -``` - -The old `inputs:` direct mapping format is no longer supported. - -## Output JSON Format - -`--output-modules json` controls the writer. `--output-schema` controls the JSON contract that writer serializes. - -The named schemas are stable public contracts. If a consumer depends on a specific shape, it should request that schema explicitly at runtime. The runtime default selection may change in a future release. - -The `seriatim-intermediate` schema is the current default selection when neither `--output-schema` nor `SERIATIM_OUTPUT_SCHEMA` is set. It stays close to the minimal schema, but adds optional `categories` on each segment: - -```json -{ - "metadata": { - "application": "seriatim", - "version": "dev", - "output_schema": "seriatim-intermediate" - }, - "segments": [ - { - "id": 1, - "start": 1.25, - "end": 3.5, - "speaker": "Eric Rakestraw", - "text": "Hello there.", - "categories": ["backchannel"] - } - ] -} -``` - -The `seriatim-full` schema uses the full seriatim envelope: - -```json -{ - "metadata": { - "application": "seriatim", - "version": "dev", - "input_reader": "json-files", - "input_files": ["eric.json", "mike.json"], - "preprocessing_modules": ["validate-raw", "normalize-speakers", "trim-text"], - "postprocessing_modules": ["detect-overlaps", "resolve-overlaps", "backchannel", "filler", "resolve-danglers", "coalesce", "detect-overlaps", "autocorrect", "assign-ids", "validate-output"], - "output_modules": ["json"] - }, - "segments": [ - { - "id": 1, - "source": "eric.json", - "source_segment_index": 0, - "speaker": "Eric Rakestraw", - "start": 1.25, - "end": 3.5, - "text": "Hello there.", - "overlap_group_id": 1 - }, - { - "id": 2, - "source": "eric.json", - "source_ref": "word-run:1:1:1", - "derived_from": ["eric.json#0"], - "speaker": "Eric Rakestraw", - "start": 2.0, - "end": 2.5, - "text": "Resolved word run", - "categories": ["backchannel"] - } - ], - "overlap_groups": [ - { - "id": 1, - "start": 1.25, - "end": 4.0, - "segments": ["eric.json#0", "mike.json#0"], - "speakers": ["Eric Rakestraw", "Mike Brown"], - "class": "unknown", - "resolution": "unresolved" - } - ] -} -``` - -The `seriatim-minimal` schema emits minimal metadata and compact ordered segments: - -```json -{ - "metadata": { - "application": "seriatim", - "version": "dev", - "output_schema": "seriatim-minimal" - }, - "segments": [ - { - "id": 1, - "start": 1.25, - "end": 3.5, - "speaker": "Eric Rakestraw", - "text": "Hello there." - } - ] -} -``` - -Minimal output intentionally omits categories, overlap groups, source/provenance fields, and pipeline configuration metadata. - -Intermediate output intentionally omits overlap groups and source/provenance fields, but keeps optional `categories` and minimal metadata. - -Segments are sorted deterministically by: - -```text -(start, end, source, source_segment_index/source_ref, speaker) -``` - -Final segment IDs are assigned after sorting and start at `1`. - -The public Go output contract is available from: - -```go -import "gitea.maximumdirect.net/eric/seriatim/schema" -``` - -The same package embeds machine-readable JSON Schemas in `schema/full-output.schema.json`, `schema/intermediate-output.schema.json`, and `schema/minimal-output.schema.json`. The default `validate-output` postprocessor validates the selected output shape and verifies final segment IDs are present, sequential, and start at `1`. - -## Overlap Detection - -The default postprocessing pipeline detects overlapping segment groups. - -Overlap behavior: - -- A strict timing overlap is required: `next.start < current_group_end`. -- Segments that only touch at a boundary are not grouped. -- Groups require at least two distinct speakers. -- Transitive overlaps are grouped together. -- Segments in detected groups receive `overlap_group_id`. -- `overlap_groups[].segments` contains stable references in `source#source_segment_index` format. -- `class` is currently `unknown`. -- `resolution` is `unresolved` until `resolve-overlaps` replaces the group. - -## Overlap Resolution - -The default postprocessing pipeline runs `detect-overlaps`, then `resolve-overlaps`, then `backchannel`, then `filler`, then `resolve-danglers`, then `coalesce`, then a second `detect-overlaps` pass. - -For each detected overlap group, `resolve-overlaps` uses preserved WhisperX word timing to build smaller word-run replacement segments: - -- The resolution window expands the detected overlap group by `--coalesce-gap` seconds on both sides. -- Nearby same-speaker context segments are included when they intersect the expanded window and their start or end is within `--coalesce-gap` of the original overlap boundary. -- Once a segment is selected for replacement, all timed words from that segment participate in word-run construction; the window controls segment selection, not per-word clipping. -- Context segments that are part of another detected overlap group are not pulled into the current group. -- Untimed words are included in replacement text in original word order when nearby timed words create a replacement run. -- Untimed words do not affect replacement segment start/end times or word-run gap splitting. -- Words for the same speaker are merged into one run when the gap between adjacent words is no greater than `SERIATIM_OVERLAP_WORD_RUN_GAP`. -- The default word-run gap is `1.0` seconds. -- Set `SERIATIM_OVERLAP_WORD_RUN_GAP` to a positive number of seconds to override the default. -- Near-start replacement word runs are reordered so shorter segments come first when adjacent starts are within `SERIATIM_OVERLAP_WORD_RUN_REORDER_WINDOW`. -- The default word-run reorder window is `1.0` seconds. -- Set `SERIATIM_OVERLAP_WORD_RUN_REORDER_WINDOW` to a positive number of seconds to override the default. -- Replacement segment text is built by joining word text with single spaces. -- Replacement segments include `source_ref` and `derived_from`. -- Replacement segments omit `source_segment_index` because they are derived from one or more original segments. -- Resolved overlap groups are removed before the second detection pass. -- Replacement segments are left without `overlap_group_id` until the second detection pass annotates any remaining overlap. -- If a speaker has no usable word timing in a group, that speaker's original segment is kept. -- If no speakers in a group have usable word timing, the original group and annotations remain unchanged. - -## Backchannels - -The default pipeline runs `backchannel` before `coalesce`. It tags short acknowledgement segments with: - -```json -"categories": ["backchannel"] -``` - -Backchannel matching is case-insensitive, ignores punctuation for matching and word-count purposes, trims surrounding whitespace, and requires a matching acknowledgement phrase, no more than three whitespace-delimited words, and duration no greater than `SERIATIM_BACKCHANNEL_MAX_DURATION` seconds. The default maximum duration is `2.0` seconds. - -## Fillers - -The default pipeline runs `filler` after `backchannel` and before `coalesce`. It tags short filler utterances with: - -```json -"categories": ["filler"] -``` - -Filler matching is case-insensitive, ignores punctuation for matching and word-count purposes, trims surrounding whitespace, and requires only filler tokens such as `um`, `uh`, `er`, `erm`, `ah`, `eh`, `hmm`, `mm`, or repeated combinations of those tokens. Matching segments must contain no more than three whitespace-delimited words and have duration no greater than `SERIATIM_FILLER_MAX_DURATION` seconds. The default maximum duration is `1.25` seconds. - -## Dangler Resolution - -The default pipeline runs `resolve-danglers` before `coalesce` and before the second overlap detection pass. It repairs short derived fragments when they share provenance with a nearby segment: - -- Dangling-end fragments have no more than two words and end in punctuation. -- Dangling-start fragments have no more than two words. -- Matching uses same-speaker segments with any shared `derived_from` value. -- Merged segments use `source_ref` values such as `resolve-danglers:1`, keep the target segment's transcript position, and union `derived_from`. - -## Coalescing - -The default pipeline runs `coalesce` after `resolve-danglers` and before the second overlap detection pass. It merges adjacent same-speaker segments in the transcript's current order when `next.start - current.end <= --coalesce-gap`. - -Coalesced segments use `source_ref` values such as `coalesce:1`, include `derived_from`, and omit `source_segment_index`. - -Different-speaker backchannel and filler segments do not block coalescing of surrounding same-speaker segments. Same-speaker backchannel and filler segments are merged normally when they are within `--coalesce-gap`. When same-speaker segments are coalesced, any `backchannel` or `filler` category from the merged inputs is dropped from the coalesced segment. - -## Autocorrect - -Autocorrect is included in the default postprocessing pipeline. If `--autocorrect` is omitted, the module leaves transcript text unchanged and records a skip event in the optional report. - -Enable corrections by passing `--autocorrect`: - -```sh -go run ./cmd/seriatim merge \ - --input-file input.json \ - --autocorrect autocorrect.yml \ - --output-file merged.json -``` - -`autocorrect.yml` format: - -```yaml -autocorrect: - - target: "Hrank" - match: - - "hrank" - - "Frank" - - - target: "Mike Brown" - match: - - "Mike Pat" -``` - -Matching behavior: - -- Matching is case-sensitive. -- Matches apply only to whole tokens, not substrings inside larger words. -- Punctuation and whitespace can surround a match. -- Multi-word and hyphenated matches are supported. -- Duplicate match strings are invalid, including duplicates across separate rules. - -## Current Limitations - -- Only JSON input is supported. -- Overlap resolution depends on WhisperX word timing; groups without usable word timing remain unresolved. -- Alternate output formats are not implemented yet. - -## Release Builds - -Local builds record version metadata as `dev`. Release builds should inject the release version with `ldflags`: - -```sh -go build -ldflags "-X gitea.maximumdirect.net/eric/seriatim/internal/buildinfo.Version=v1.0.0" ./cmd/seriatim -``` diff --git a/docs/internal/README.md b/docs/internal/README.md new file mode 100644 index 0000000..fa0fa99 --- /dev/null +++ b/docs/internal/README.md @@ -0,0 +1,24 @@ +# Internal Documentation Index + +## Audience +Developers and LLM coding agents changing Narratio internals. + +## Scope +Implementation-accurate contracts for workspace/state, stages, and external adapter boundaries. + +## Component Docs +- `workspace.md`: local state model, manifests, run-local layout, promotion, and cleanup invariants. +- `stage-prepare.md`: input materialization and provenance capture. +- `stage-transcribe.md`: WhisperX transcript generation. +- `stage-merge.md`: Seriatim normalization + merge. +- `stage-polish.md`: Audita transcript polishing. +- `stage-normalize.md`: post-polish normalization. +- `stage-trim.md`: bounds-driven transcript trimming. +- `stage-analyze.md`: Scriptorium session recap generation. +- `stage-archive.md`: archive upload and current-pointer publish contract. +- `integration-audita.md`: Audita adapter invocation/validation contract. +- `integration-seriatim.md`: Seriatim merge/normalize/trim adapter contract. +- `integration-scriptorium.md`: Scriptorium run/render adapter contract. + +## Canonical Owner +`docs/internal/` is the canonical home for implemented internals per `docs/documentation/policy.md`. diff --git a/docs/internal/integration-audita.md b/docs/internal/integration-audita.md new file mode 100644 index 0000000..849dc9f --- /dev/null +++ b/docs/internal/integration-audita.md @@ -0,0 +1,66 @@ +# Integration: audita + +## Purpose +Define Narratio's adapter contract for transcript polishing via Audita CLI subprocess execution. + +## Inputs and Outputs +Inputs (`audita.PolishRequest`): +- merged transcript path +- glossary path +- output processed transcript path +- optional report path (required when report enabled) +- work dir +- generated config path +- stdout/stderr log paths +- optional module/model/base URL and concurrency knobs + +Outputs (`audita.PolishResult`): +- processed transcript path +- optional report path +- generated config path +- stdout/stderr log paths +- exit code, duration, invoked binary +- adapter metadata + +## Boundaries +Owns: +- Deterministic CLI argument construction for `audita process` +- Environment bridging for API credentials +- Invocation config emission +- Output validation for processed transcript and report + +Does not own: +- Upstream/downstream stage orchestration +- Credential sourcing policy beyond required env-var presence check + +## Config Fields Used +Via `pipeline.audita.*` mapped in app/stage wiring: +- `binary`, `timeout`, `llm_api_key_env`, `modules`, `base_url`, `model` +- `transcript_description`, `config_path`, `output_schema`, `work_dir_retention` +- `total_llm_concurrency`, `proposal_llm_concurrency`, `validation_model`, `validation_llm_concurrency`, `report` + +## External Adapters Used +- Shared subprocess helper (`internal/adapters/subprocess`) to run CLI and capture logs. + +## State and Manifest Behavior +- No direct manifest writes. +- Stage-level metadata records adapter provenance and credential-present signal. +- Generated invocation YAML is written when `GeneratedConfigPath` is provided. + +## Skip and Resume Behavior +- Adapter has no skip/resume logic. Stage/runner controls this. + +## Failure Behavior +- Constructor validation fails on invalid binary/timeout/schema/concurrency/URL values. +- Run fails on missing required paths, missing required credential env var, subprocess errors, invalid processed JSON shape, or invalid report JSON. +- Failures preserve stdout/stderr paths in returned result metadata. + +## Tests to Inspect Before Changing +- `internal/adapters/audita/subprocess_test.go` +- `internal/adapters/audita/fake_test.go` +- `internal/stage/polish_test.go` + +## Architectural Invariants +- Processed output must be valid JSON with top-level `segments` array. +- When report is enabled, report output must be valid JSON. +- If `llm_api_key_env` is configured, credential must be present in environment. diff --git a/docs/internal/integration-scriptorium.md b/docs/internal/integration-scriptorium.md new file mode 100644 index 0000000..76b16d6 --- /dev/null +++ b/docs/internal/integration-scriptorium.md @@ -0,0 +1,64 @@ +# Integration: scriptorium + +## Purpose +Define Narratio's adapter contract for Scriptorium artifact generation and render-debug subprocess invocations. + +## Inputs and Outputs +Inputs: +- `RunArtifactRequest`: binary, config path, prompt/profile IDs, input map, vars map, timeout, output path, logs/config paths, optional API env and working dir +- `RenderArtifactRequest`: same core fields for render mode + +Outputs (`ArtifactResult`): +- output path +- stdout/stderr log paths +- generated config path +- exit code and duration +- command mode (`run` or `render`) +- prompt/profile provenance +- validation failure signal +- adapter metadata + +## Boundaries +Owns: +- Deterministic CLI arg construction for `scriptorium run` and `scriptorium render` +- Common request validation +- Invocation config emission +- Output existence/non-empty checks +- Validation-failure mapping for run exit code 2 + +Does not own: +- Artifact selection policy (`analyze` stage) +- Bounds semantic validation (`trim` stage) + +## Config Fields Used +Via `pipeline.scriptorium.*` and stage-level artifact config: +- `binary`, `config_path`, `timeout`, `render_debug` +- artifact-level `prompt_id`, `profile_id`, `timeout`, `inputs`, `vars`, `output_path` + +## External Adapters Used +- Shared subprocess helper (`internal/adapters/subprocess`). + +## State and Manifest Behavior +- No direct manifest writes. +- Stage metadata records adapter outputs and command mode. +- Generated invocation YAML is written when requested. + +## Skip and Resume Behavior +- Adapter has no skip/resume logic. Stage/runner controls execution. + +## Failure Behavior +- Request validation fails for missing binary/prompt/output, invalid timeout, invalid input/var names, or missing required API env var. +- Subprocess errors bubble with command context. +- `run` exit code 2 is treated as `ValidationFailed=true` and surfaced as error by calling stage. +- Successful subprocess still fails if output file is missing/empty. + +## Tests to Inspect Before Changing +- `internal/adapters/scriptorium/subprocess_test.go` +- `internal/adapters/scriptorium/fake_test.go` +- `internal/stage/analyze_test.go` +- `internal/stage/trim_test.go` + +## Architectural Invariants +- Both modes require explicit timeout > 0. +- Input/var maps are sorted into deterministic CLI argument order. +- Run-mode validation failures are represented explicitly, not silently skipped. diff --git a/docs/internal/integration-seriatim.md b/docs/internal/integration-seriatim.md new file mode 100644 index 0000000..250ea1e --- /dev/null +++ b/docs/internal/integration-seriatim.md @@ -0,0 +1,60 @@ +# Integration: seriatim + +## Purpose +Define Narratio's adapter contract for merge, normalize, and trim subprocess invocations of Seriatim. + +## Inputs and Outputs +Inputs: +- `MergeRequest`: raw/normalized transcript inputs, output path, optional report, speaker/autocorrect paths, logs/config +- `NormalizeRequest`: input transcript, output path, schema, optional report, timeout/log/config +- `TrimRequest`: input transcript, output path, keep selector, timeout/log/config + +Outputs: +- `MergeResult`, `NormalizeResult`, `TrimResult` with output paths, logs/config paths, exit code, duration, binary provenance, and metadata. + +## Boundaries +Owns: +- Validated deterministic CLI invocation construction +- Optional env tuning propagation for merge +- Invocation config file emission +- JSON output validation + +Does not own: +- Transcript input selection/promotion logic (stage-owned) +- Bounds computation (scriptorium/trim-stage-owned) + +## Config Fields Used +Via `pipeline.seriatim.*` mapped in app/stage wiring: +- `binary`, `timeout`, `output_schema`, `coalesce_gap`, `report` +- `env.overlap_word_run_gap` +- `env.overlap_word_run_reorder_window` +- `env.backchannel_max_duration` +- `env.filler_max_duration` + +## External Adapters Used +- Shared subprocess helper (`internal/adapters/subprocess`). + +## State and Manifest Behavior +- No direct manifest writes. +- Stage metadata consumes adapter result fields and preserves generated config/log references. + +## Skip and Resume Behavior +- Adapter has no skip/resume logic. Runner controls stage execution. + +## Failure Behavior +- Constructor fails for invalid binary/timeout/output-schema/coalesce-gap. +- Merge fails on missing output path/inputs/report path (if enabled), subprocess errors, invalid merged output JSON, invalid report JSON. +- Normalize fails on missing input/output, invalid schema, subprocess errors, invalid normalized output JSON shape, invalid report JSON. +- Trim fails on missing input/output/keep selector, subprocess errors, invalid trimmed output JSON shape. + +## Tests to Inspect Before Changing +- `internal/adapters/seriatim/subprocess_test.go` +- `internal/adapters/seriatim/fake_test.go` +- `internal/stage/merge_test.go` +- `internal/stage/normalize_test.go` +- `internal/stage/trim_test.go` + +## Architectural Invariants +- Supported output schemas are limited to `seriatim-minimal`, `seriatim-intermediate`, `seriatim-full`. +- Normalize/trim outputs must include `segments` arrays. +- Merge/normalize/trim all route through deterministic subprocess invocation. diff --git a/docs/internal/stage-analyze.md b/docs/internal/stage-analyze.md new file mode 100644 index 0000000..bbfa73f --- /dev/null +++ b/docs/internal/stage-analyze.md @@ -0,0 +1,69 @@ +# Stage: analyze + +## Purpose +Generate the session recap artifact using configured Scriptorium artifact settings. + +## Inputs and Outputs +Inputs: +- transcript inputs as requested by selected artifact config (processed/normalized/trimmed/current recap, depending on `pipeline.scriptorium.artifacts.session_recap.inputs`) + +Outputs: +- `artifacts/session_recap.md` + +## Boundaries +Owns: +- Selecting supported analyze artifact (`session_recap` only) +- Resolving transcript/reference inputs and vars +- Optional render-debug execution before run +- Main Scriptorium run and output promotion + +Does not own: +- Transcript processing pipeline stages +- Archive publish/pointer behavior + +## Config Fields Used +- `session.session_id` +- `session.campaign` +- `pipeline.workspace.root` +- `pipeline.scriptorium.binary` +- `pipeline.scriptorium.config_path` +- `pipeline.scriptorium.timeout` +- `pipeline.scriptorium.render_debug` +- `pipeline.scriptorium.artifacts.session_recap.*` + - `enabled` + - `prompt_id` + - `profile_id` + - `timeout` + - `output_path` + - `render_debug` + - `inputs` + - `vars` + +## External Adapters Used +- Scriptorium adapter: + - optional `RenderArtifact` (debug diagnostics) + - `RunArtifact` (actual recap generation) + +## State and Manifest Behavior +- If `pipeline.scriptorium` is nil, stage returns success metadata with `skipped=true`. +- If no enabled artifacts exist, stage returns success metadata with `skipped=true`. +- If enabled artifacts exist but any artifact other than `session_recap` is enabled, stage fails. +- Uses run-local output/log/config/reports paths when run layout is enabled. +- Promotes canonical recap output and records adapter metadata. + +## Skip and Resume Behavior +- Runner-level skip applies when already succeeded and not forced. +- Forced reruns can stale downstream succeeded stages. +- Stage-local "skipped" metadata is distinct from runner-level stage status skip. + +## Failure Behavior +- Fails on missing required resolved inputs, invalid transcript inputs, render/run adapter failures, or validation-failed run results. + +## Tests to Inspect Before Changing +- `internal/stage/analyze_test.go` +- `internal/adapters/scriptorium/subprocess_test.go` + +## Architectural Invariants +- Analyze implementation supports only `artifacts.session_recap` as executable artifact. +- Optional inputs may be omitted; required inputs must resolve. +- Successful output must exist and be non-empty before promotion. diff --git a/docs/internal/stage-archive.md b/docs/internal/stage-archive.md new file mode 100644 index 0000000..e46853d --- /dev/null +++ b/docs/internal/stage-archive.md @@ -0,0 +1,68 @@ +# Stage: archive + +## Purpose +Publish run records and promoted session artifacts to object storage, then atomically advance the remote current pointer. + +## Inputs and Outputs +Inputs: +- session manifest and prerequisite stage records +- run root contents under `runs/{run_id}/` +- promotion sources from session root (`archive.promote_artifacts`) + +Outputs: +- uploaded run files under `{session_prefix}/runs/{run_id}/...` +- uploaded promoted artifacts under `{session_prefix}/...` +- `{session_prefix}/current/manifest.json` +- `{session_prefix}/current/run_id.txt` written last + +## Boundaries +Owns: +- Archive enable/disable gate behavior +- Prerequisite stage success enforcement +- Run file collection and upload (excluding `audio/`) +- Promotion rule resolution and upload +- Commit pointer publish order + +Does not own: +- Stage execution before archive +- Post-archive local cleanup policy execution (handled by app cleanup logic) + +## Config Fields Used +- `pipeline.archive.enabled` +- `pipeline.archive.upload_run` +- `pipeline.archive.promote_artifacts` +- `pipeline.storage.s3.bucket` +- `pipeline.storage.s3.root_prefix` +- `pipeline.workspace.root` +- `session.campaign` +- `session.session_id` + +## External Adapters Used +- Object storage backend (`env.ObjectStore`) for upload/list primitives. + +## State and Manifest Behavior +- Requires `prepare`, `transcribe`, `merge`, `polish`, `normalize`, `trim`, and `analyze` status `succeeded`. +- Resolves bucket/prefix from manifest identity first, then config fallback. +- Writes metadata including: + - upload counts/paths + - `current_manifest_key` + - `current_run_id_key` + - `current_pointer_written` +- On skipped archive path, returns metadata with `skipped=true` and pointer not written. + +## Skip and Resume Behavior +- Stage may self-skip (metadata skip) when archive disabled or run upload disabled. +- Runner-level skip also applies for previously succeeded stage unless forced. + +## Failure Behavior +- Fails on missing prerequisite success, missing object store when required, missing run root, missing required promotion source, upload failures, or pointer write failures. +- Pointer semantics are fail-safe: `current/run_id.txt` is not written if prior required uploads fail. + +## Tests to Inspect Before Changing +- `internal/stage/archive_test.go` +- `internal/app/post_archive_cleanup_test.go` + +## Architectural Invariants +- Run upload excludes `audio/` subtree. +- `current/manifest.json` uploads before `current/run_id.txt`. +- `current/run_id.txt` is the remote publish commit marker. diff --git a/docs/internal/stage-merge.md b/docs/internal/stage-merge.md new file mode 100644 index 0000000..4839a81 --- /dev/null +++ b/docs/internal/stage-merge.md @@ -0,0 +1,63 @@ +# Stage: merge + +## Purpose +Normalize per-speaker raw transcripts and merge them into one merged transcript via Seriatim. + +## Inputs and Outputs +Inputs: +- `transcripts/raw/*.json` +- `inputs/speakers.yml` +- `inputs/autocorrect.yml` + +Outputs: +- `transcripts/merged.json` +- optional `artifacts/seriatim.report.json` (when report enabled) + +## Boundaries +Owns: +- Raw transcript discovery/validation +- Per-input normalize calls to Seriatim +- Final merge call to Seriatim +- Run-local log/config/report path wiring +- Promotion of merged/report outputs to canonical paths + +Does not own: +- Transcript polishing or downstream artifact generation + +## Config Fields Used +- `session.session_id` +- `session.campaign` +- `pipeline.workspace.root` +- `pipeline.seriatim.binary` +- `pipeline.seriatim.timeout` +- `pipeline.seriatim.output_schema` +- `pipeline.seriatim.coalesce_gap` +- `pipeline.seriatim.report` +- `pipeline.seriatim.env.*` + +## External Adapters Used +- Seriatim adapter: + - `Normalize` for each raw input + - `Run` for final merge + +## State and Manifest Behavior +- Reads transcript inputs from transcribe stage outputs in manifest when present; falls back to canonical raw directory. +- Writes run-local outputs/logs/config under `runs/{run_id}/merge/...` when enabled. +- Promotes canonical merged transcript and optional report. +- Records normalized-input provenance and adapter metadata in stage metadata. + +## Skip and Resume Behavior +- Runner-level skip applies when already succeeded and not forced. +- Forced rerun of this or upstream stages can stale downstream succeeded stages via runner invalidation. + +## Failure Behavior +- Fails on missing/invalid raw transcripts, missing speakers/autocorrect files, normalize failure, merge failure, invalid merged output JSON, or invalid report JSON when enabled. + +## Tests to Inspect Before Changing +- `internal/stage/merge_test.go` +- `internal/adapters/seriatim/subprocess_test.go` + +## Architectural Invariants +- Merge consumes normalized forms of each raw transcript. +- Merged transcript must validate before promotion. +- Report output is optional and gated by config. diff --git a/docs/internal/stage-normalize.md b/docs/internal/stage-normalize.md new file mode 100644 index 0000000..8b4e08d --- /dev/null +++ b/docs/internal/stage-normalize.md @@ -0,0 +1,56 @@ +# Stage: normalize + +## Purpose +Normalize the processed transcript into a deterministic intermediate schema for trim and optionally emit a normalize report. + +## Inputs and Outputs +Inputs: +- `transcripts/processed.json` + +Outputs: +- `transcripts/normalized.json` (or configured normalize output path) +- optional `artifacts/seriatim.normalize.report.json` + +## Boundaries +Owns: +- Processed transcript discovery/validation +- Normalize request construction and invocation +- Optional normalize report wiring +- Promotion of normalized transcript and optional report + +Does not own: +- Bounds detection or segment trimming + +## Config Fields Used +- `session.session_id` +- `session.campaign` +- `pipeline.workspace.root` +- `pipeline.normalize.output_path` +- `pipeline.normalize.output_schema` +- `pipeline.normalize.report` +- `pipeline.seriatim.binary` +- `pipeline.seriatim.timeout` + +## External Adapters Used +- Seriatim adapter (`Normalize`). + +## State and Manifest Behavior +- Reads processed transcript from polish outputs in manifest when present; falls back to canonical path. +- Uses run-local output/report/log/config paths when run layout is enabled. +- Promotes canonical normalized transcript and optional normalize report. +- Records adapter/result metadata including source path selection. + +## Skip and Resume Behavior +- Runner-level skip applies when already succeeded and not forced. +- Forced reruns can stale downstream succeeded stages. + +## Failure Behavior +- Fails on missing/invalid processed transcript, adapter error, invalid normalized output, or invalid report output when report enabled. + +## Tests to Inspect Before Changing +- `internal/stage/normalize_test.go` +- `internal/adapters/seriatim/subprocess_test.go` + +## Architectural Invariants +- Normalized output must validate as processed-transcript-compatible JSON (`segments` array required). +- Default normalize config is applied when `pipeline.normalize` is unset. diff --git a/docs/internal/stage-polish.md b/docs/internal/stage-polish.md new file mode 100644 index 0000000..dff7b44 --- /dev/null +++ b/docs/internal/stage-polish.md @@ -0,0 +1,69 @@ +# Stage: polish + +## Purpose +Polish merged transcript with Audita and produce a processed transcript for downstream normalization/analyze. + +## Inputs and Outputs +Inputs: +- `transcripts/merged.json` +- `inputs/glossary.yml` + +Outputs: +- `transcripts/processed.json` +- optional `artifacts/audita.report.json` (when report enabled) + +## Boundaries +Owns: +- Merged transcript discovery/validation +- Audita invocation request construction +- Run-local logs/config/work-dir/report wiring +- Promotion of processed transcript and optional report + +Does not own: +- Upstream merge normalization +- Downstream normalize/trim/analyze logic + +## Config Fields Used +- `session.session_id` +- `session.campaign` +- `pipeline.workspace.root` +- `pipeline.audita.binary` +- `pipeline.audita.timeout` +- `pipeline.audita.llm_api_key_env` +- `pipeline.audita.modules` +- `pipeline.audita.base_url` +- `pipeline.audita.model` +- `pipeline.audita.transcript_description` +- `pipeline.audita.config_path` +- `pipeline.audita.output_schema` +- `pipeline.audita.work_dir_retention` +- `pipeline.audita.total_llm_concurrency` +- `pipeline.audita.proposal_llm_concurrency` +- `pipeline.audita.validation_model` +- `pipeline.audita.validation_llm_concurrency` +- `pipeline.audita.report` + +## External Adapters Used +- Audita adapter (`env.Audita.Run`). + +## State and Manifest Behavior +- Reads merged transcript from merge manifest outputs when available; falls back to canonical merged path. +- Uses run-local output/report/log/config/scratch paths when run layout is enabled. +- Promotes canonical `transcripts/processed.json` and optional report. +- Records adapter invocation metadata, credential presence signal, and output provenance in stage metadata. + +## Skip and Resume Behavior +- Runner-level skip applies when already succeeded and not forced. +- Forced rerun can stale downstream succeeded stages via runner invalidation. + +## Failure Behavior +- Fails on missing/invalid merged transcript, missing glossary, adapter error, invalid processed output shape (`segments` array required), or invalid report JSON when enabled. + +## Tests to Inspect Before Changing +- `internal/stage/polish_test.go` +- `internal/adapters/audita/subprocess_test.go` + +## Architectural Invariants +- Processed transcript must contain a top-level `segments` array. +- Report behavior is strictly config-gated. +- Stage output canonicalization always ends at `transcripts/processed.json`. diff --git a/docs/internal/stage-prepare.md b/docs/internal/stage-prepare.md new file mode 100644 index 0000000..0c77787 --- /dev/null +++ b/docs/internal/stage-prepare.md @@ -0,0 +1,74 @@ +# Stage: prepare + +## Purpose +Materialize all required session inputs into canonical local workspace paths and record input provenance in the session manifest. + +## Inputs and Outputs +Inputs: +- `session.yml` (resolved session config) +- `pipeline.resolved.yml` (materialized from resolved pipeline config) +- `speakers.yml` +- `autocorrect.yml` +- `glossary.yml` +- audio source: + - local (`session.inputs.audio_dir` or `session.inputs.audio_files`), or + - S3 (`session.inputs.audio_s3.prefix`) + +Outputs: +- `inputs/session.yml` +- `inputs/pipeline.resolved.yml` +- `inputs/speakers.yml` +- `inputs/autocorrect.yml` +- `inputs/glossary.yml` +- `audio/*.flac` in session workdir +- `manifest.Inputs` records with checksums and source metadata + +## Boundaries +Owns: +- Input path resolution and validation +- Local copy/materialization of configs and audio files +- S3 audio download to run-scoped spool, then copy into work audio dir + +Does not own: +- Transcript generation/processing +- Archive publish behavior + +## Config Fields Used +- `session.session_id` +- `session.campaign` +- `session.inputs.speakers_file` +- `session.inputs.autocorrect_file` +- `session.inputs.glossary_file` +- `session.inputs.audio_dir` +- `session.inputs.audio_files` +- `session.inputs.audio_s3.prefix` +- `pipeline.workspace.root` +- `pipeline.spool.root` +- `pipeline.storage.s3.bucket` +- `pipeline.storage.s3.root_prefix` + +## External Adapters Used +- Object storage backend (`env.ObjectStore`) for S3 audio list/download when `audio_s3` is configured. + +## State and Manifest Behavior +- Ensures workspace layout exists. +- Writes resolved config and input files to canonical `inputs/` paths. +- Records all prepared inputs into `manifest.Inputs` (sorted deterministically by kind/path). +- For S3 audio, records `S3Bucket`, `S3Key`, `S3Size`, `S3ETag`, and `SpoolPath` in each audio input record. + +## Skip and Resume Behavior +- Runner-level skip applies when stage already `succeeded` and `--force` is not set. +- Stage itself is deterministic/idempotent for unchanged inputs (`copyFileIfChanged`, `writeBytesIfChanged`). + +## Failure Behavior +- Fails on missing required files, invalid audio source combinations, no discoverable `.flac` files, duplicate audio basenames, missing object store for S3 mode, or S3 list/download failures. + +## Tests to Inspect Before Changing +- `internal/stage/prepare_test.go` +- `internal/app/session_cli_test.go` +- `internal/config/load_validate_test.go` + +## Architectural Invariants +- `audio_dir`/`audio_files` and `audio_s3` are mutually exclusive. +- Audio files must be `.flac`. +- Canonical `inputs/*` and `audio/*` paths are the durable source for downstream stages. diff --git a/docs/internal/stage-transcribe.md b/docs/internal/stage-transcribe.md new file mode 100644 index 0000000..907cdfa --- /dev/null +++ b/docs/internal/stage-transcribe.md @@ -0,0 +1,58 @@ +# Stage: transcribe + +## Purpose +Generate per-speaker raw transcripts from prepared audio using WhisperX. + +## Inputs and Outputs +Inputs: +- `audio/*.flac` prepared by `prepare` + +Outputs: +- `transcripts/raw/.json` for each input audio file + +## Boundaries +Owns: +- Discovering prepared audio inputs +- Deriving speaker ids from audio basenames +- Parallel WhisperX invocation with bounded concurrency +- Validating produced JSON and promoting run-local outputs + +Does not own: +- Transcript merge/polish/normalize/trim/analyze + +## Config Fields Used +- `session.session_id` +- `session.campaign` +- `pipeline.workspace.root` +- `pipeline.whisperx.transcribe_url` +- `pipeline.whisperx.language` +- `pipeline.whisperx.timeout` +- `pipeline.whisperx.retries` +- `pipeline.whisperx.retry_delay` +- `pipeline.whisperx.concurrency` + +## External Adapters Used +- WhisperX adapter (`env.WhisperX.Transcribe`). + +## State and Manifest Behavior +- Uses run-local output paths under `runs/{run_id}/transcribe/outputs/...` when run layout is enabled. +- Validates each generated transcript JSON before promotion. +- Promotes canonical outputs to `transcripts/raw/*.json`. +- Records per-file metadata (attempts/status/duration/output path) in stage metadata. + +## Skip and Resume Behavior +- Runner-level skip applies for previously succeeded stage unless forced. +- On forced upstream reruns, downstream succeeded stages can be marked `stale` by runner logic. + +## Failure Behavior +- Fails if no prepared audio exists, duplicate speaker basenames are detected, adapter output path mismatches expected path, any output JSON is invalid, or one worker fails. +- Cancels in-flight workers after first terminal error. + +## Tests to Inspect Before Changing +- `internal/stage/transcribe_test.go` +- `internal/app/whisperx_wiring_test.go` + +## Architectural Invariants +- Speaker identity is derived from `.flac` basename and must be unique. +- Every successful speaker output must be valid JSON before promotion. +- Canonical raw transcript set is the only supported merge input surface. diff --git a/docs/internal/stage-trim.md b/docs/internal/stage-trim.md new file mode 100644 index 0000000..378aeef --- /dev/null +++ b/docs/internal/stage-trim.md @@ -0,0 +1,75 @@ +# Stage: trim + +## Purpose +Optionally trim the normalized transcript to session bounds; always produce a durable trimmed transcript. + +## Inputs and Outputs +Inputs: +- `transcripts/normalized.json` + +Outputs: +- `transcripts/trimmed.json` (or configured trim output path) +- when trim enabled: `artifacts/session_bounds.json` + +## Boundaries +Owns: +- Trim-enabled switch behavior +- Bounds generation via Scriptorium artifact run +- Bounds validation against normalized transcript +- Keep-selector derivation and Seriatim trim invocation +- Copy-through behavior when disabled or bounds indicate unchanged transcript + +Does not own: +- Upstream normalization +- Downstream artifact analysis + +## Config Fields Used +- `session.session_id` +- `session.campaign` +- `pipeline.workspace.root` +- `pipeline.trim.enabled` +- `pipeline.trim.output_path` +- `pipeline.trim.bounds.prompt_id` +- `pipeline.trim.bounds.profile_id` +- `pipeline.trim.bounds.timeout` +- `pipeline.trim.bounds.output_path` +- `pipeline.trim.bounds.transcript_input_name` +- `pipeline.trim.bounds.render_debug` +- `pipeline.trim.bounds.render_output_path` +- `pipeline.seriatim.binary` +- `pipeline.seriatim.timeout` +- `pipeline.scriptorium.binary` +- `pipeline.scriptorium.config_path` +- `pipeline.scriptorium.timeout` + +## External Adapters Used +- Scriptorium adapter: + - optional `RenderArtifact` for bounds debug render + - `RunArtifact` for bounds output +- Seriatim adapter: + - `Trim` when bounds indicate trimming is required + +## State and Manifest Behavior +- Reads normalized transcript from normalize manifest outputs when available; falls back to canonical path. +- Uses run-local outputs/logs/reports/config/scratch paths when run layout is enabled. +- Promotes canonical trimmed transcript; promotes session bounds when trim enabled. +- Records bounds diagnostics, trim action, keep selector, and adapter metadata. + +## Skip and Resume Behavior +- Runner-level skip applies when already succeeded and not forced. +- Forced reruns can stale downstream succeeded stages. +- When `trim.enabled=false`, stage still succeeds by copying normalized to trimmed output. + +## Failure Behavior +- Fails on missing/invalid normalized transcript. +- With trim enabled, fails on missing adapters/config, bounds generation/validation errors, invalid bounds JSON, invalid range/segment ids, trim adapter failures, or invalid trimmed output. + +## Tests to Inspect Before Changing +- `internal/stage/trim_test.go` +- `internal/adapters/scriptorium/subprocess_test.go` +- `internal/adapters/seriatim/subprocess_test.go` + +## Architectural Invariants +- Trim never falls back to processed transcript; normalized transcript is required input. +- `session_bounds` output exists only for enabled trim path. +- Render-debug artifacts are diagnostics and not declared stage outputs. diff --git a/docs/internal/workspace.md b/docs/internal/workspace.md new file mode 100644 index 0000000..04001e2 --- /dev/null +++ b/docs/internal/workspace.md @@ -0,0 +1,68 @@ +# Workspace internals + +## Purpose +Define the local durable and run-local workspace model used by stages, manifests, resume, and archive. + +## Inputs and Outputs +Inputs: +- `pipeline.workspace.root` +- `session.campaign` +- `session.session_id` +- generated `run_id` + +Outputs: +- Session manifest at `{workspace.root}/work/{campaign}/{session_id}/manifest.json` +- Run manifest at `{workspace.root}/work/{campaign}/{session_id}/runs/{run_id}/manifest.json` +- Canonical durable session directories and run-local stage trees + +## Boundaries +Owns: +- Session-level path layout (`inputs/`, `audio/`, `transcripts/`, `artifacts/`, `reports/`, `logs/`, `config/`, `current/`, `runs/`) +- Run-local stage sandbox layout under `runs/{run_id}/{stage}/` +- Session lock acquisition/release (`.lock`) + +Does not own: +- Stage business logic +- Remote archive semantics (documented in `stage-archive.md`) +- CLI argument parsing + +## Config Fields Used +- `pipeline.workspace.root` +- `pipeline.workspace.cleanup_after_archive` +- `pipeline.spool.root` +- `pipeline.spool.delete_audio_after_archive` +- `session.campaign` +- `session.session_id` + +## External Adapters Used +None directly in this subsystem. Stages may use object storage adapters and then write local outputs into this layout. + +## State and Manifest Behavior +- Session state is persisted in the session manifest (`manifest.Manifest`). +- Invocation history is persisted per run in run manifests under `runs/{run_id}/manifest.json`. +- During each run, stage outputs are often written run-local first (`runs/{run_id}/{stage}/outputs/...`) and promoted to canonical session paths after stage success. +- `manifest.Artifacts` entries record `ProducerRunID` for durable outputs. +- For S3 audio sessions, `prepare` records spool/work paths and S3 provenance in `manifest.Inputs`. + +## Skip and Resume Behavior +- Skip/resume decisions are made in `internal/app` (`run_control.go`, `resume.go`) using stage status in the session manifest. +- `--force` reruns selected stages and marks downstream previously-succeeded stages as `stale`. +- Workspace layout is idempotent (`EnsureLayoutFor`) and reused across runs. + +## Failure Behavior +- Failures preserve manifests and run-local files for inspection. +- Lock conflicts fail fast via `ErrLockConflict`. +- Cleanup can fail post-archive; failure is recorded in archive stage metadata and returned by the run. + +## Tests to Inspect Before Changing +- `internal/artifacts/local_test.go` +- `internal/stage/run_local_test.go` +- `internal/app/run_control_test.go` +- `internal/app/resume_run_stage_test.go` +- `internal/app/post_archive_cleanup_test.go` + +## Architectural Invariants +- Session root is campaign-aware: `{workspace.root}/work/{campaign}/{session_id}`. +- Run roots are always nested: `runs/{run_id}` under the session root. +- Run-local output promotion must end in canonical session paths. +- Cleanup only targets run-scoped directories and must never delete configured root directories. diff --git a/docs/operations.md b/docs/operations.md new file mode 100644 index 0000000..3a9b9fc --- /dev/null +++ b/docs/operations.md @@ -0,0 +1,170 @@ +# Operations + +This guide describes the implemented operator lifecycle for Narratio. + +For field-level configuration, see [docs/config.md](./config.md). For full command/flag reference, see [docs/cli.md](./cli.md). + +## Normal workflow (S3-first path) + +1. Upload session `.flac` files to the session audio prefix in object storage: + - `{root_prefix}/campaigns/{campaign}/sessions/{session_id}/{audio_s3.prefix}` +2. Run Narratio: + +```bash +narratio run --session-id 2026-04-04 +``` + +3. Read success output: + - `narratio run: session ; executed= skipped=; manifest=` + - `manifest=` is the local session manifest path to use with `status`. + +Notes: + +- This command relies on discoverable `pipeline.yml` and `session.yml` unless `--config` and `--session` are passed explicitly. +- For S3 audio input, `session.inputs.audio_s3.prefix` must be configured and audio files must already exist remotely. + +## Local filesystem layout and state artifacts + +Session root: + +- `{workspace.root}/work/{campaign}/{session_id}/` + +Primary state: + +- `manifest.json`: session-level manifest (authoritative local stage state). +- `runs/{run_id}/manifest.json`: run-level manifest for one invocation. +- `.lock`: session lock file while a run is active. + +Canonical session directories: + +- `inputs/` +- `audio/` +- `transcripts/` +- `artifacts/` +- `reports/` +- `logs/` +- `config/` +- `current/` +- `runs/` + +Run-local stage directories: + +- `runs/{run_id}/{stage}/` +- Stage runtime files are written under deterministic run-local subdirectories such as: + - `outputs/`, `logs/`, `reports/`, `config/`, `scratch/` + +Behavior notes: + +- Layout creation is idempotent. +- Durable outputs are promoted to canonical session paths after stage success. +- Run-local artifacts remain in `runs/{run_id}/...` unless configured post-archive cleanup removes that run scope. + +## Remote archive layout and publish contract + +When archive is enabled and run upload is enabled, archive publishes to object storage under: + +- Session prefix: `{root_prefix}/campaigns/{campaign}/sessions/{session_id}/` +- Run prefix: `{session_prefix}/runs/{run_id}/` + +Archive uploads: + +- Run record files from run root (including stage subtrees and run manifest), excluding local `audio/`. +- Promoted artifacts from `archive.promote_artifacts` to session-level keys. + +Publish order (commit contract): + +1. Upload `current/manifest.json` +2. Upload `current/run_id.txt` last + +Meaning of `current/run_id.txt`: + +- It is the effective remote commit marker for published session state. +- It is written only after required run uploads and required promotions succeed. + +## Resume, retry, and safe rerun behavior + +Default skip behavior: + +- `run` and `run-stage` skip stages already marked `succeeded` unless `--force` is set. + +Resume behavior: + +- `resume` starts at the first non-`succeeded` stage in canonical stage order. +- If all stages are `succeeded`, `resume` prints that no stages remain. +- `resume --force` runs full stage order rather than starting at first non-succeeded. + +Forced rerun behavior: + +- Successful forced rerun of an upstream stage marks downstream previously `succeeded` stages as `stale`. +- `stale` stages are not treated as complete and are eligible to run in subsequent commands. + +Targeted rerun with one stage: + +```bash +narratio run-stage --force +``` + +Valid stage names: + +- `prepare`, `transcribe`, `merge`, `polish`, `normalize`, `trim`, `analyze`, `archive`, `notify` + +Safe operator pattern: + +1. Force-rerun the stage that changed. +2. Run `resume` to rebuild downstream stages in order. + +## Cleanup behavior + +Cleanup is considered only after run execution completes and only when archive stage both executed and succeeded. + +Configured cleanup toggles: + +- `pipeline.spool.delete_audio_after_archive=true` + - deletes only run-scoped spool audio directory: `{spool.root}/{campaign}/{session_id}/{run_id}/audio/` +- `pipeline.workspace.cleanup_after_archive=true` + - deletes only run-scoped local run directory: `{workspace.root}/work/{campaign}/{session_id}/runs/{run_id}/` + +Eligibility gates for cleanup: + +- archive is enabled +- archive run upload is enabled +- archive metadata indicates run record upload happened +- archive metadata indicates `current` pointer write completed (`current/run_id.txt` written) + +Cleanup does not run for: + +- failed runs +- incomplete runs +- unarchived runs +- archive-skipped runs (`archive.enabled=false` or `archive.upload_run=false`) + +## Failure and recovery playbooks + +What remains after failure: + +- Session manifest remains on disk. +- Run manifest remains under `runs/{run_id}/manifest.json`. +- Run-local stage artifacts/logs/config/reports remain under `runs/{run_id}/...`. +- Failed/incomplete runs remain local-only. +- Remote current pointer is not committed if archive prerequisite or pointer-write steps fail. + +Recommended recovery flow: + +1. Inspect current state: + +```bash +narratio status --manifest +``` + +2. Fix the root cause (config, input, credentials, adapter availability, etc.). +3. Continue with: + - `narratio resume --session-id ` for ordered continuation, or + - `narratio run-stage --force ` for targeted correction, then `resume`. + +## Operational caveats + +- `status` requires an explicit manifest path; there is no direct session-id lookup command. +- S3 audio mode and local audio mode are mutually exclusive in session config. +- Archive verifies stage prerequisites (`prepare` through `analyze`) before publishing. +- By default, archive does not upload local `audio/` into run history. +- Unknown CLI commands fail and print usage. diff --git a/docs/stages/archive.md b/docs/stages/archive.md deleted file mode 100644 index 14829cb..0000000 --- a/docs/stages/archive.md +++ /dev/null @@ -1,119 +0,0 @@ -# Archive Storage - -This document describes implemented archive-stage publish behavior. - -## S3 Paths - -Session root: - -`{root_prefix}/campaigns/{campaign}/sessions/{session_id}/` - -Run prefix: - -`{root_prefix}/campaigns/{campaign}/sessions/{session_id}/runs/{run_id}/` - -## Scope - -Implemented: - -- archive uploads successful run records to remote object storage through the storage backend abstraction. -- archive uploads configured promoted outputs to session-level keys. -- archive uploads `current/manifest.json`. -- archive uploads `current/run_id.txt` last as the effective commit marker. -- optional post-archive local cleanup: - - `pipeline.spool.delete_audio_after_archive: true` removes only the run-scoped spool audio directory - - `pipeline.workspace.cleanup_after_archive: true` removes only the run-scoped local workdir -- tests use fake storage and do not require live S3. - -Future work: - -- `notify` stage behavior -- stale detection -- optional future source-audio upload mode -- additional artifact generation beyond current implemented set - -## Prerequisites - -Archive verifies these stages succeeded before upload: - -- `prepare` -- `transcribe` -- `merge` -- `polish` -- `normalize` -- `trim` -- `analyze` - -If any prerequisite is missing or not succeeded, archive fails and does not upload. -Failed or incomplete runs remain local only. - -## Run Upload - -Archive uploads existing files from the run workdir when present: - -- `inputs/` -- `transcripts/` -- `artifacts/` -- `reports/` (optional) -- `config/` -- `logs/` -- `manifest.json` - -Relative paths are preserved under `runs/{run_id}/`. - -## Promotion Rules - -Archive applies `archive.promote_artifacts` in config order. - -Rule behavior: - -- `from`: local workdir-relative source path -- `to`: session-root-relative destination key -- `required: true`: missing source fails archive -- `required: false`: missing source is skipped and recorded - -Default promoted outputs: - -- `transcripts/trimmed.json` -- `artifacts/session_recap.md` - -## Current Pointers - -Archive writes: - -1. `current/manifest.json` (after run upload + promotions) -2. `current/run_id.txt` last - -`current/run_id.txt` contains exactly: - -- `{run_id}` plus trailing newline - -Writing `current/run_id.txt` last makes it the effective commit marker for published session state. - -If any required run upload, promotion upload, or current-manifest upload fails, archive returns failure and does not write `current/run_id.txt`. -Cleanup runs only after this commit-marker write has succeeded. - -## Audio Upload Policy - -Archive does not upload local `audio/` by default. -Original audio is expected at the session-level audio prefix and is not duplicated under `runs/{run_id}/`. - -## Config Controls - -- `archive.enabled: false` skips archive cleanly. -- `archive.upload_run: false` skips run upload cleanly. -- both skip cases also skip post-archive local cleanup. - -## Metadata - -Archive stage metadata includes non-secret upload context (for example): - -- `s3_bucket` -- `s3_run_prefix` -- run upload counts/paths -- promoted upload counts/paths -- skipped optional promotions -- `current_manifest_key` -- `current_run_id_key` -- `current_pointer_written` -- `audio_upload_skipped` diff --git a/docs/stages/prepare.md b/docs/stages/prepare.md deleted file mode 100644 index 3abfe35..0000000 --- a/docs/stages/prepare.md +++ /dev/null @@ -1,84 +0,0 @@ -# S3 Audio Input - -This document describes implemented S3 audio input behavior in `prepare`. - -## Scope - -Implemented: - -- `prepare` can acquire source audio from S3 when `session.inputs.audio_s3.prefix` is configured. -- object listing and download go through the storage backend abstraction. -- tests use fake storage; no live S3 service is required for test runs. - -Not implemented: - -- uploads of failed runs - -## Required Configuration - -`pipeline.yml`: - -- `storage.s3.bucket` must be set when S3 audio input is used. -- `storage.s3.root_prefix` defaults to `dnd`. -- `storage.s3.access_key_id_env` defaults to `OBJECT_STORAGE_KEY_ID`. -- `storage.s3.secret_access_key_env` defaults to `OBJECT_STORAGE_KEY`. -- `spool.root` defaults to `/var/spool/narratio`. - -`session.yml`: - -- configure `session.campaign` and `session.session_id`. -- configure `session.inputs.audio_s3.prefix` for S3 audio input. -- do not configure `inputs.audio_dir` or `inputs.audio_files` at the same time as `inputs.audio_s3`. - -## Prefix Shape - -Session S3 root: - -`{root_prefix}/campaigns/{campaign}/sessions/{session_id}/` - -Audio prefix: - -`{session_root}/{audio_s3.prefix}` - -Example: - -`dnd/campaigns/forsaken/sessions/2026-04-19/audio/` - -Audio files must already exist in S3 before running Narratio. - -## Prepare Behavior - -When `inputs.audio_s3.prefix` is configured, `prepare`: - -1. lists objects under the computed S3 audio prefix -2. filters to `.flac` objects -3. fails when no `.flac` objects are found -4. downloads selected objects to spool audio: - - `{spool.root}/{campaign}/{session_id}/{run_id}/audio/` -5. materializes audio into workdir audio: - - `{workspace.root}/work/{campaign}/{session_id}/runs/{run_id}/audio/` -6. records input provenance in the manifest (bucket, key, metadata, local paths, checksum) - -Notes: - -- `.flac` filtering is case-insensitive. -- ETag is recorded as provider metadata only and is not treated as a checksum. - -## Local Audio Development - -Local audio workflows remain supported: - -- `inputs.audio_dir` -- `inputs.audio_files` - -These options are mutually exclusive with `inputs.audio_s3`. - -## Archive Boundary - -Current archive behavior relevant to S3 audio input: - -- successful runs are uploaded by archive under `runs/{run_id}/` -- configured promotions are uploaded to session-level destinations -- `current/manifest.json` and `current/run_id.txt` are published -- local source audio is not re-uploaded by default -- failed or incomplete runs are not uploaded diff --git a/docs/troubleshooting.md b/docs/troubleshooting.md new file mode 100644 index 0000000..2c8d14d --- /dev/null +++ b/docs/troubleshooting.md @@ -0,0 +1,229 @@ +# Troubleshooting + +## Purpose +Canonical operator troubleshooting guide for recurring implemented Narratio failures. + +## Config file discovery failure + +Symptom: +- `run`, `plan`, `resume`, or `run-stage` fails saying config/session file was not found. + +Likely Cause: +- `pipeline.yml` or `session.yml` is missing from default search paths. +- Wrong working directory when relying on `./session.yml`. + +Diagnostics: + +```bash +pwd +ls -l ./session.yml +ls -l /usr/local/etc/narratio/pipeline.yml /etc/narratio/pipeline.yml +``` + +Safe Fix: +- Pass explicit paths with `--config` and `--session`. +- Or place files in documented discovery paths. + +Links: +- [docs/config.md](./config.md) +- [docs/cli.md](./cli.md) + +## Session template rendering failure + +Symptom: +- Load fails with unresolved template placeholder or `session_id` mismatch. + +Likely Cause: +- `session.yml` contains `{{session_id}}`/`{{ session_id }}` but `--session-id` was omitted. +- Provided `--session-id` does not match rendered `session_id`. + +Diagnostics: + +```bash +narratio plan --session ./session.yml --session-id 2026-04-04 +``` + +Safe Fix: +- Always pass `--session-id` when using template placeholders. +- Ensure rendered `session_id` equals intended run session id. + +Links: +- [docs/config.md](./config.md) + +## Strict YAML decode or validation failure + +Symptom: +- Config load fails with unknown field, missing required field, invalid duration, or invalid cross-field constraint. + +Likely Cause: +- YAML key typo or stale field name. +- Required fields missing. +- Invalid value format (for example duration/URL/env var name). + +Diagnostics: + +```bash +narratio plan --config /path/to/pipeline.yml --session /path/to/session.yml --session-id 2026-04-04 +``` + +Safe Fix: +- Correct fields/values to match canonical reference and examples. +- Validate against `docs/examples/` shapes. + +Links: +- [docs/config.md](./config.md) +- [docs/examples/](./examples/) + +## Manifest/status path failure + +Symptom: +- `status` fails because manifest path is missing, unreadable, or invalid. + +Likely Cause: +- Wrong manifest path. +- Manifest removed after cleanup. +- Trying to run `status` without `--manifest`. + +Diagnostics: + +```bash +narratio status --manifest /path/to/manifest.json +ls -l /path/to/manifest.json +``` + +Safe Fix: +- Use manifest path printed by `run`, `resume`, or `run-stage` output. +- Re-run with correct session/config if inspecting a different session. + +Links: +- [docs/cli.md](./cli.md) +- [docs/operations.md](./operations.md) + +## Session lock conflict (`.lock`) + +Symptom: +- Run fails with lock conflict indicating session workdir is already locked. + +Likely Cause: +- Another Narratio process is actively running the same session. +- Prior run exited unexpectedly and left a stale lock file. + +Diagnostics: + +```bash +ls -l {workspace.root}/work/{campaign}/{session_id}/.lock +cat {workspace.root}/work/{campaign}/{session_id}/.lock +ps aux | grep narratio +``` + +Safe Fix: +- If another run is active, wait for it to finish. +- If no process is active and lock is stale, remove only that session `.lock` file and retry. + +Links: +- [docs/operations.md](./operations.md) +- [docs/internal/workspace.md](./internal/workspace.md) + +## Secrets env-dir or credential env failure + +Symptom: +- Startup fails loading secrets directory, or a stage fails because required credential env var is missing. + +Likely Cause: +- `pipeline.secrets.env_dir` path is wrong/unreadable. +- Credential env var referenced in config is unset or empty. + +Diagnostics: + +```bash +ls -la /path/to/secrets_dir +env | grep -E 'AUDITA|OBJECT_STORAGE|AWS|SCRIPTORIUM' +``` + +Safe Fix: +- Fix `pipeline.secrets.env_dir` path/permissions. +- Ensure required env vars are set to non-empty values. +- Keep secrets out of YAML; use env references only. + +Links: +- [docs/config.md](./config.md) +- [docs/operations.md](./operations.md) + +## S3-audio prepare failure + +Symptom: +- `prepare` fails in S3 mode (no audio found, list/download failure, backend missing, path conflict). + +Likely Cause: +- Wrong `session.inputs.audio_s3.prefix`. +- No `.flac` files at expected prefix. +- Missing or invalid S3 backend credentials/config. +- Conflicting audio-source settings (`audio_s3` plus local audio fields). + +Diagnostics: + +```bash +narratio run-stage --config /path/to/pipeline.yml --session /path/to/session.yml --session-id 2026-04-04 prepare +``` + +Safe Fix: +- Ensure `audio_s3` is the only audio source configured for that session. +- Confirm `.flac` objects exist under the resolved session audio prefix. +- Fix S3 storage configuration and credentials. + +Links: +- [docs/config.md](./config.md) +- [docs/operations.md](./operations.md) +- [docs/internal/stage-prepare.md](./internal/stage-prepare.md) + +## Archive prerequisite or promotion/current-pointer failure + +Symptom: +- `archive` fails due to prerequisite stage status, missing required promotion source, or pointer write failure. + +Likely Cause: +- One or more prerequisite stages are not `succeeded`. +- Required promoted artifact does not exist. +- Remote upload failure before `current/run_id.txt` write. + +Diagnostics: + +```bash +narratio status --manifest /path/to/manifest.json +narratio run-stage --config /path/to/pipeline.yml --session /path/to/session.yml --session-id 2026-04-04 archive +``` + +Safe Fix: +- Resume or rerun failed upstream stage(s). +- Ensure required promoted artifact paths exist locally before archive. +- Retry archive after storage/connectivity issue is resolved. + +Links: +- [docs/operations.md](./operations.md) +- [docs/config.md](./config.md) +- [docs/internal/stage-archive.md](./internal/stage-archive.md) + +## `run-stage` invalid stage name or invalid flags + +Symptom: +- `run-stage` fails with unknown stage or invalid flag/argument usage. + +Likely Cause: +- Stage name typo. +- Missing positional stage argument. +- Unsupported/incorrect flag syntax. + +Diagnostics: + +```bash +narratio run-stage --config /path/to/pipeline.yml --session /path/to/session.yml --session-id 2026-04-04 normalize +``` + +Safe Fix: +- Use only supported stage names. +- Provide exactly one positional stage argument. +- Align flags to documented command reference. + +Links: +- [docs/cli.md](./cli.md) +- [docs/operations.md](./operations.md) diff --git a/examples/pipeline.audita-overrides.yml b/examples/pipeline.audita-overrides.yml deleted file mode 100644 index 635ec48..0000000 --- a/examples/pipeline.audita-overrides.yml +++ /dev/null @@ -1,26 +0,0 @@ -workspace: - root: ./tmp/narratio-workspace - -whisperx: - transcribe_url: "https://transcription.example.com/transcribe" - -seriatim: - binary: "seriatim" - timeout: "10m" - output_schema: "seriatim-intermediate" - coalesce_gap: 3.0 - -audita: - binary: "audita" - timeout: "3h" - base_url: "https://openrouter.ai/api/v1" - model: "openrouter/google/gemma-4-31b-it" - llm_api_key_env: "AUDITA_LLM_API_KEY" - modules: ["glossary", "homophones", "spoken_word", "grammar"] - output_schema: "audita-v1" - work_dir_retention: "auto" - total_llm_concurrency: 2 - proposal_llm_concurrency: 1 - validation_model: "openrouter/google/gemma-4-31b-it" - validation_llm_concurrency: 1 - report: true diff --git a/examples/pipeline.minimal.yml b/examples/pipeline.minimal.yml deleted file mode 100644 index 3130c8b..0000000 --- a/examples/pipeline.minimal.yml +++ /dev/null @@ -1,55 +0,0 @@ -workspace: - root: ./tmp/narratio-workspace - cleanup_after_archive: false - -storage: - backend: s3 - s3: - bucket: "my-dnd-archive" - root_prefix: "dnd" - region: "us-east-1" - # Optional credential env-var names (defaulted when omitted): - # access_key_id_env: "OBJECT_STORAGE_KEY_ID" - # secret_access_key_env: "OBJECT_STORAGE_KEY" - -spool: - root: "/var/spool/narratio" - delete_audio_after_archive: false - -archive: - enabled: true - upload_run: true - -whisperx: - transcribe_url: "https://transcription.example.com/transcribe" - -# Optional. When omitted entirely, Narratio defaults to seriatim binary + runtime defaults. -seriatim: {} - -# Optional runtime overrides. Model/provider can be owned by Audita runtime config. -audita: - config_path: "/usr/local/etc/audita/config.yml" - llm_api_key_env: "AUDITA_LLM_API_KEY" - -# Optional Scriptorium integration for analyze artifacts. -scriptorium: - config_path: "/usr/local/etc/scriptorium/config.yml" - artifacts: - session_recap: - enabled: true - prompt_id: "dnd.session_recap" - output_path: "artifacts/session_recap.md" - inputs: - transcript: - source: "trimmed_transcript" - required: true - previous_recap: - source: "previous_session_artifact" - artifact: "session_recap" - required: false - vars: - session_id: true - session_date: true - campaign_name: true - previous_session_id: true - output_kind: "session_recap" diff --git a/examples/session.minimal.yml b/examples/session.minimal.yml deleted file mode 100644 index 0ff8244..0000000 --- a/examples/session.minimal.yml +++ /dev/null @@ -1,13 +0,0 @@ -session_id: 2026-05-03 -campaign: sample-campaign -date: 2026-05-03 -title: Sample Session -inputs: - audio_dir: ./audio - # Optional S3 input alternative. Do not configure with audio_dir/audio_files. - # Narratio prepare lists this prefix and downloads .flac files. - # audio_s3: - # prefix: "audio/" - speakers_file: ./speakers.yml - autocorrect_file: ./autocorrect.yml - glossary_file: ./glossary.yml diff --git a/examples/session.template.yml b/examples/session.template.yml deleted file mode 100644 index b2e5ac0..0000000 --- a/examples/session.template.yml +++ /dev/null @@ -1,12 +0,0 @@ -session_id: "{{ session_id }}" -campaign: sample-campaign -date: "" -title: "" -inputs: - audio_dir: ./audio - # Optional S3 input alternative. Do not configure with audio_dir/audio_files. - # audio_s3: - # prefix: "audio/{{ session_id }}/" - speakers_file: ./speakers.yml - autocorrect_file: ./autocorrect.yml - glossary_file: ./glossary.yml diff --git a/internal/config/load_validate_test.go b/internal/config/load_validate_test.go index deb3e41..fd39437 100644 --- a/internal/config/load_validate_test.go +++ b/internal/config/load_validate_test.go @@ -866,15 +866,59 @@ func TestValidateMissingAudioSource(t *testing.T) { } func TestExamplesLoadAndValidate(t *testing.T) { - pipelinePath := filepath.Join("..", "..", "examples", "pipeline.minimal.yml") - sessionPath := filepath.Join("..", "..", "examples", "session.minimal.yml") - - cfg, err := Load(pipelinePath, sessionPath) - if err != nil { - t.Fatalf("Load(examples) error = %v", err) + examplesDir := filepath.Join("..", "..", "docs", "examples") + tests := []struct { + name string + pipelineFile string + sessionFile string + sessionOpts SessionLoadOptions + }{ + { + name: "minimal pipeline with local audio session", + pipelineFile: "pipeline.minimal.yml", + sessionFile: "session.local-audio.yml", + }, + { + name: "production pipeline with s3 audio session", + pipelineFile: "pipeline.production.yml", + sessionFile: "session.s3-audio.yml", + }, + { + name: "full annotated pipeline with local audio session", + pipelineFile: "pipeline.full.annotated.yml", + sessionFile: "session.local-audio.yml", + }, + { + name: "template session renders with session_id option", + pipelineFile: "pipeline.minimal.yml", + sessionFile: "session.template.yml", + sessionOpts: SessionLoadOptions{ + SessionID: "2026-05-03", + }, + }, } - if err := Validate(cfg); err != nil { - t.Fatalf("Validate(examples) error = %v", err) + + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + pipelinePath := filepath.Join(examplesDir, tt.pipelineFile) + sessionPath := filepath.Join(examplesDir, tt.sessionFile) + + var ( + cfg *Config + err error + ) + if strings.TrimSpace(tt.sessionOpts.SessionID) == "" { + cfg, err = Load(pipelinePath, sessionPath) + } else { + cfg, err = LoadWithSessionOptions(pipelinePath, sessionPath, tt.sessionOpts) + } + if err != nil { + t.Fatalf("load example config error = %v", err) + } + if err := Validate(cfg); err != nil { + t.Fatalf("validate example config error = %v", err) + } + }) } }