Complete documentation rebuild

2026-05-18 22:03:59 -05:00
parent 37daab7857
commit c4e87f58c7
40 changed files with 2722 additions and 2989 deletions
--- a/docs/config.md
+++ b/docs/config.md
@@ -0,0 +1,293 @@
+# Configuration
+
+## 1. Overview
+
+Narratio loads two YAML files:
+
+- `pipeline.yml`: pipeline-level runtime configuration.
+- `session.yml`: per-session metadata and input selection.
+
+These commands load and validate both files before running:
+
+- `narratio run`
+- `narratio plan`
+- `narratio resume`
+- `narratio run-stage`
+
+Configuration behavior:
+
+- strict YAML decode is enabled (`KnownFields(true)`): unknown fields fail.
+- session templates are rendered before session YAML decode.
+- defaults are applied for many optional pipeline fields.
+- validation enforces required fields, value formats, and cross-field constraints.
+
+## 2. Config file discovery
+
+Pipeline config lookup for `run`, `plan`, `resume`, and `run-stage`:
+
+- If `--config <path>` is provided, that explicit path is used.
+- If `--config` is omitted, Narratio searches in order:
+  1. `/usr/local/etc/narratio/pipeline.yml`
+  2. `/etc/narratio/pipeline.yml`
+- The first existing file wins.
+- If none exist, the command fails with a searched-paths error.
+
+## 3. Session file discovery and templating
+
+Session config lookup for `run`, `plan`, `resume`, and `run-stage`:
+
+- If `--session <path>` is provided, that explicit path is used.
+- If `--session` is omitted, Narratio searches in order:
+  1. `./session.yml`
+  2. `/usr/local/etc/narratio/session.yml`
+  3. `/etc/narratio/session.yml`
+- The first existing file wins.
+- If none exist, the command fails and asks you to pass `--session`.
+
+Session templating:
+
+- Supported placeholders:
+  - `{{session_id}}`
+  - `{{ session_id }}`
+- `--session-id <value>` supplies the template value.
+- Unresolved placeholders fail with a template-rendering error.
+- If `--session-id` is provided and rendered `session_id` differs, load fails with a mismatch error.
+- Strict YAML decode still applies after template rendering.
+
+## 4. Minimal pipeline config
+
+```yaml
+workspace:
+  root: ./tmp/narratio-workspace
+
+whisperx:
+  transcribe_url: "https://transcription.example.com/transcribe"
+```
+
+Why this is sufficient:
+
+- `workspace.root` and `whisperx.transcribe_url` are the core required pipeline fields.
+- Seriatim and Audita sections may be omitted; defaults are applied.
+- Archive, storage, spool, normalize, and other optional sections get defaults when omitted.
+
+## 5. Minimal session template
+
+```yaml
+session_id: "{{ session_id }}"
+campaign: sample-campaign
+inputs:
+  audio_dir: ./audio
+  speakers_file: ./examples/speakers.yml
+  autocorrect_file: ./examples/autocorrect.yml
+  glossary_file: ./examples/glossary.yml
+```
+
+Usage:
+
+```bash
+narratio run --config /path/to/pipeline.yml --session ./session.yml --session-id 2026-05-03
+```
+
+## 6. Production-oriented config
+
+```yaml
+workspace:
+  root: /var/lib/narratio/workspace
+  cleanup_after_archive: true
+
+storage:
+  backend: s3
+  s3:
+    bucket: my-dnd-archive
+    root_prefix: dnd
+    region: us-east-1
+    access_key_id_env: OBJECT_STORAGE_KEY_ID
+    secret_access_key_env: OBJECT_STORAGE_KEY
+
+spool:
+  root: /var/spool/narratio
+  delete_audio_after_archive: true
+
+archive:
+  enabled: true
+  upload_run: true
+  promote_artifacts:
+    - from: transcripts/trimmed.json
+      to: transcripts/trimmed.json
+      required: true
+    - from: artifacts/session_recap.md
+      to: artifacts/session_recap.md
+      required: true
+
+whisperx:
+  transcribe_url: "https://transcription.example.com/transcribe"
+```
+
+Operational notes:
+
+- `workspace.cleanup_after_archive` controls run-scoped workspace cleanup after successful archive commit.
+- `spool.delete_audio_after_archive` controls run-scoped spool-audio cleanup after successful archive commit.
+- S3 archive/session-audio workflows require `storage.s3.bucket`.
+
+## 7. Full pipeline reference
+
+Defaults listed here are effective runtime defaults after load.
+
+| Path | Type | Required | Default | Constraints / Notes |
+| --- | --- | --- | --- | --- |
+| `pipeline.workspace.root` | string | Yes | none | Must be non-empty. |
+| `pipeline.workspace.cleanup_after_archive` | bool | No | `false` | Run-scoped workdir cleanup after successful archive commit. |
+| `pipeline.secrets.env_dir` | string | Conditional | none | If `pipeline.secrets` is set, `env_dir` must be non-empty. |
+| `pipeline.storage.backend` | string | No | empty | `s3` enables S3 archive decision path checks. |
+| `pipeline.storage.bucket` | string | No | empty | Accepted by schema; compatibility field. |
+| `pipeline.storage.prefix` | string | No | empty | Accepted by schema; compatibility field. |
+| `pipeline.storage.s3.bucket` | string | Conditional | empty | Required when S3 session audio is used or S3 archive upload is enabled. |
+| `pipeline.storage.s3.root_prefix` | string | No | `dnd` | Must be relative, non-empty, no `..`. |
+| `pipeline.storage.s3.region` | string | No | empty | Optional region hint for backend wiring. |
+| `pipeline.storage.s3.endpoint` | string | No | empty | If provided, must not be all-whitespace. |
+| `pipeline.storage.s3.force_path_style` | bool | No | `false` | S3-compatible endpoint toggle. |
+| `pipeline.storage.s3.access_key_id_env` | string | No | `OBJECT_STORAGE_KEY_ID` | Must be a valid env var name. |
+| `pipeline.storage.s3.secret_access_key_env` | string | No | `OBJECT_STORAGE_KEY` | Must be a valid env var name. |
+| `pipeline.spool.root` | string | No | `/var/spool/narratio` | Local spool root. |
+| `pipeline.spool.delete_audio_after_archive` | bool | No | `false` | Cleanup toggle for run-scoped spool audio. |
+| `pipeline.archive.enabled` | bool | No | `true` | Archive stage enablement. |
+| `pipeline.archive.upload_run` | bool | No | `true` | Run-record upload toggle when archive enabled. |
+| `pipeline.archive.promote_artifacts[]` | list | No | two default rules | Defaults: `transcripts/trimmed.json` and `artifacts/session_recap.md`. |
+| `pipeline.archive.promote_artifacts[].from` | string | Yes (per rule) | none | Must be relative, non-empty, no `..`. |
+| `pipeline.archive.promote_artifacts[].to` | string | Yes (per rule) | none | Must be relative, non-empty, no `..`. |
+| `pipeline.archive.promote_artifacts[].required` | bool | No | `true` | Defaults per rule if omitted. |
+| `pipeline.whisperx.transcribe_url` | string | Yes | none | Must be a valid URL. |
+| `pipeline.whisperx.language` | string | No | `en` | Passed to WhisperX adapter. |
+| `pipeline.whisperx.timeout` | duration string | No | `30m` | Must parse as duration. |
+| `pipeline.whisperx.retries` | int | No | `3` | Must be `>= 0`. |
+| `pipeline.whisperx.retry_delay` | duration string | No | `2s` | Must parse as duration. |
+| `pipeline.whisperx.concurrency` | int | No | `2` | Must be `> 0`. |
+| `pipeline.seriatim.binary` | string | No | `seriatim` | Must be non-empty after defaults. |
+| `pipeline.seriatim.timeout` | duration string | No | `10m` | Must parse as duration. |
+| `pipeline.seriatim.output_schema` | string | No | `seriatim-intermediate` | Allowed: `seriatim-minimal`, `seriatim-intermediate`, `seriatim-full`. |
+| `pipeline.seriatim.coalesce_gap` | float | No | `3.0` | Must be `>= 0`. |
+| `pipeline.seriatim.report` | bool | No | `true` | Enables report output in Seriatim calls. |
+| `pipeline.seriatim.env.overlap_word_run_gap` | float | No | unset | If set, must be `> 0`. |
+| `pipeline.seriatim.env.overlap_word_run_reorder_window` | float | No | unset | If set, must be `> 0`. |
+| `pipeline.seriatim.env.backchannel_max_duration` | float | No | unset | If set, must be `> 0`. |
+| `pipeline.seriatim.env.filler_max_duration` | float | No | unset | If set, must be `> 0`. |
+| `pipeline.audita.binary` | string | No | `audita` | Must be non-empty after defaults. |
+| `pipeline.audita.timeout` | duration string | No | `3h` | Must parse as duration. |
+| `pipeline.audita.llm_api_key_env` | string | No | empty | Name of env var to forward to Audita. |
+| `pipeline.audita.modules[]` | list[string] | No | empty | If set, each must be one of `glossary`, `homophones`, `spoken_word`, `grammar`. |
+| `pipeline.audita.base_url` | string | No | empty | If non-empty, must be a valid URL. |
+| `pipeline.audita.model` | string | No | empty | Optional model override passed to Audita. |
+| `pipeline.audita.total_llm_concurrency` | int | No | unset | If set, must be `> 0`. |
+| `pipeline.audita.proposal_llm_concurrency` | int | No | unset | If set, must be `> 0`. |
+| `pipeline.audita.validation_model` | string | No | empty | Optional validation model override. |
+| `pipeline.audita.validation_llm_concurrency` | int | No | unset | If set, must be `> 0`. |
+| `pipeline.audita.transcript_description` | string | No | empty | If provided, must not be all-whitespace. |
+| `pipeline.audita.config_path` | string | No | empty | If provided, must not be all-whitespace. |
+| `pipeline.audita.output_schema` | string | No | empty | Allowed: empty, `bare-segments`, `audita-v1`. |
+| `pipeline.audita.work_dir_retention` | string | No | empty | Allowed: empty, `always`, `auto`, `never`. |
+| `pipeline.audita.report` | bool | No | `true` | Enables Audita report output. |
+| `pipeline.normalize.output_path` | string | No | `transcripts/normalized.json` | Must be non-empty after defaults. |
+| `pipeline.normalize.output_schema` | string | No | `seriatim-intermediate` | Allowed: `seriatim-minimal`, `seriatim-intermediate`, `seriatim-full`. |
+| `pipeline.normalize.report` | bool | No | `true` | Enables normalize report output. |
+| `pipeline.trim.enabled` | bool | No | `false` | When `false`, trim bounds fields are not required. |
+| `pipeline.trim.output_path` | string | Conditional | none | Required when `pipeline.trim.enabled=true`. |
+| `pipeline.trim.bounds.prompt_id` | string | Conditional | none | Required when `pipeline.trim.enabled=true`. |
+| `pipeline.trim.bounds.profile_id` | string | No | empty | Optional profile override. |
+| `pipeline.trim.bounds.transcript_input_name` | string | Conditional | none | Required when `pipeline.trim.enabled=true`. |
+| `pipeline.trim.bounds.output_path` | string | Conditional | none | Required when `pipeline.trim.enabled=true`. |
+| `pipeline.trim.bounds.timeout` | duration string | No | `10m` | Must parse as duration when set. |
+| `pipeline.trim.bounds.render_debug` | bool | No | `false` | Enables render-debug output for bounds prompt. |
+| `pipeline.trim.bounds.render_output_path` | string | Conditional | none | Required when `render_debug=true`. |
+| `pipeline.trim.seriatim.report` | bool | No | `false` | Trim-stage Seriatim report toggle. |
+| `pipeline.scriptorium.binary` | string | No | `scriptorium` | Required only when `pipeline.scriptorium` is configured. |
+| `pipeline.scriptorium.config_path` | string | No | empty | If provided, must not be all-whitespace. |
+| `pipeline.scriptorium.timeout` | duration string | No | `10m` | Must parse as duration when set. |
+| `pipeline.scriptorium.render_debug` | bool | No | `false` | Global render-debug toggle for Scriptorium adapter usage. |
+| `pipeline.scriptorium.artifacts` | map | No | empty | Artifact-generation map keyed by artifact name. |
+| `pipeline.scriptorium.artifacts.<name>.enabled` | bool | No | `false` | If `true`, `prompt_id` and `output_path` are required. |
+| `pipeline.scriptorium.artifacts.<name>.render_debug` | bool | No | unset | Per-artifact render-debug override. |
+| `pipeline.scriptorium.artifacts.<name>.prompt_id` | string | Conditional | none | Required when artifact is enabled. |
+| `pipeline.scriptorium.artifacts.<name>.profile_id` | string | No | empty | Optional profile override. |
+| `pipeline.scriptorium.artifacts.<name>.output_path` | string | Conditional | none | Required when artifact is enabled. |
+| `pipeline.scriptorium.artifacts.<name>.timeout` | duration string | No | empty | If set, must parse as duration. |
+| `pipeline.scriptorium.artifacts.<name>.inputs.<key>.source` | string | Conditional | none | Required when input is present; allowed values listed below. |
+| `pipeline.scriptorium.artifacts.<name>.inputs.<key>.artifact` | string | No | empty | Used by `previous_session_artifact` source. |
+| `pipeline.scriptorium.artifacts.<name>.inputs.<key>.path` | string | No | empty | Optional explicit path metadata. |
+| `pipeline.scriptorium.artifacts.<name>.inputs.<key>.required` | bool | No | `false` | Input requirement flag for artifact generation. |
+| `pipeline.scriptorium.artifacts.<name>.vars.<key>` | map value | No | empty | Value must be string or boolean. |
+| `pipeline.analyzer.binary_path` | string | No | empty | Optional analyzer binary override. |
+| `pipeline.analyzer.timeout` | duration string | No | empty | If set, must parse as duration. |
+| `pipeline.analyzer.artifacts.output_dir` | string | No | empty | Optional analyzer output directory hint. |
+| `pipeline.analyzer.artifacts.types[]` | list[string] | No | empty | Optional analyzer artifact type list. |
+| `pipeline.notification.backend` | string | No | empty | Optional notifier backend selector. |
+| `pipeline.notification.recipient` | string | No | empty | Optional notification recipient target. |
+| `pipeline.notification.timeout` | duration string | No | empty | If set, must parse as duration. |
+
+Allowed `pipeline.scriptorium.artifacts.<name>.inputs.<key>.source` values:
+
+- `previous_session_artifact`
+- `processed_transcript`
+- `normalized_transcript`
+- `trimmed_transcript`
+- `narratio.transcript.merged`
+- `narratio.transcript.polished`
+- `narratio.transcript.full`
+- `narratio.transcript.trimmed`
+- `narratio.bounds.session`
+- `narratio.artifact.session_recap`
+
+## 8. Full session reference
+
+| Path | Type | Required | Default | Constraints / Notes |
+| --- | --- | --- | --- | --- |
+| `session.session_id` | string | Yes | none | Must be non-empty after template rendering. |
+| `session.campaign` | string | Yes | none | Must be non-empty. Used in local/remote path modeling. |
+| `session.date` | string | No | empty | Optional session metadata for prompts/artifacts. |
+| `session.title` | string | No | empty | Optional session metadata for prompts/artifacts. |
+| `session.inputs.audio_dir` | string | Conditional | empty | One audio-source option. Mutually exclusive with `audio_s3`. |
+| `session.inputs.audio_files[]` | list[string] | Conditional | empty | One audio-source option. At least one entry can satisfy audio-source requirement. Mutually exclusive with `audio_s3`. |
+| `session.inputs.audio_s3.prefix` | string | Conditional | none | Required if `audio_s3` object is present; must be relative, non-empty, no `..`. Mutually exclusive with `audio_dir` and `audio_files`. |
+| `session.inputs.speakers_file` | string | Yes | none | Must be non-empty. |
+| `session.inputs.autocorrect_file` | string | Yes | none | Must be non-empty. |
+| `session.inputs.glossary_file` | string | Yes | none | Must be non-empty. |
+
+Audio-source rule:
+
+- You must configure exactly one audio source mode:
+  - `audio_dir`, or
+  - `audio_files` (at least one), or
+  - `audio_s3.prefix`
+- `audio_s3` cannot be combined with `audio_dir` or `audio_files`.
+
+## 9. Secrets
+
+Narratio supports filesystem-based secret injection via `pipeline.secrets.env_dir`.
+
+Behavior:
+
+- `env_dir` may be absolute or relative.
+- Relative `env_dir` is resolved from Narratio’s current working directory.
+- Each top-level file with a valid env-var filename (`[A-Za-z_][A-Za-z0-9_]*`) is loaded.
+- File contents become env-var values, with trailing `\n` / `\r\n` trimmed.
+- Existing process environment variables are preserved and not overwritten.
+- Invalid names and directories inside `env_dir` are skipped.
+- Missing/unreadable `env_dir` fails command execution.
+
+Guidance:
+
+- Store secret values in secret files or pre-set environment variables.
+- Do not put secret values directly in `pipeline.yml` or `session.yml`.
+- Use config fields like `llm_api_key_env` and S3 credential env names to reference secret variable names, not secret data.
+
+## 10. Examples
+
+Maintained config examples:
+
+- `docs/examples/pipeline.minimal.yml`
+- `docs/examples/pipeline.production.yml`
+- `docs/examples/pipeline.full.annotated.yml`
+- `docs/examples/session.template.yml`
+- `docs/examples/session.local-audio.yml`
+- `docs/examples/session.s3-audio.yml`
+
+These examples are covered by configuration load/validate tests in `internal/config`.