Documentation update in advance of building the scriptorium integration

2026-05-06 16:03:52 -05:00
parent 15037c6d80
commit 26e9746133
4 changed files with 1088 additions and 266 deletions
--- a/docs/integrations/audita.md
+++ b/docs/integrations/audita.md
@@ -0,0 +1,147 @@
+# Audita
+
+Audita is a framework-first transcript correction application. The public `audita` package provides:
+
+- deterministic transcript normalization
+- token-batched module orchestration
+- concrete `glossary`, `homophones`, `spoken_word`, and `grammar` modules built on reusable proposal / validator contracts
+- structured run reporting and work-dir diagnostics
+
+The previous working implementation has been preserved as `audita_prototype` inside this repository. Its full regression suite lives under `tests/audita_prototype`.
+
+## Development
+
+This project is set up for `uv`.
+
+```sh
+uv sync --extra dev
+uv run pytest
+```
+
+## Usage
+
+Process a transcript with the current framework implementation:
+
+```sh
+uv run audita process transcript.json --glossary glossary.yaml --output corrected.json
+```
+
+The framework currently runs this default module sequence:
+
+1. `glossary`
+2. `homophones`
+3. `glossary`
+4. `spoken_word`
+5. `grammar`
+
+Resolved run instance names are auto-numbered for repeats, so the default report pipeline is:
+
+1. `glossary_1`
+2. `homophones`
+3. `glossary_2`
+4. `spoken_word`
+5. `grammar`
+
+The default module sequence is fully implemented today:
+
+- `glossary` proposes glossary-supported acoustic corrections
+- `homophones` proposes conservative homophone and mistranscription corrections
+- `spoken_word` proposes conservative dysfluency cleanup
+- `grammar` proposes punctuation, capitalization, and spacing cleanup only
+
+To run a custom module sequence, pass `--modules`:
+
+```sh
+uv run audita process transcript.json --glossary glossary.yaml --modules grammar --output corrected.json
+```
+
+To also write a structured JSON report:
+
+```sh
+uv run audita process transcript.json --glossary glossary.yaml --output corrected.json --report-json report.json
+```
+
+From a checked-out repository, you can also use the root launcher:
+
+```sh
+./audita process transcript.json --glossary glossary.yaml --output corrected.json
+```
+
+For a system-wide command, install the source tree under `/usr/local/src/audita`, sync dependencies there, and symlink the root launcher into your `PATH`:
+
+```sh
+cd /usr/local/src/audita
+uv sync --extra dev
+ln -s /usr/local/src/audita/audita /usr/local/bin/audita
+audita process transcript.json --glossary glossary.yaml --output corrected.json
+```
+
+Without `--output`, Audita writes the corrected transcript JSON to stdout and progress logs to stderr.
+`--report-json` writes a separate machine-readable run report and never mixes report data into stdout.
+
+Useful configuration can be supplied by CLI flag or environment variable. CLI flags take precedence over environment variables. Normal runs now require LLM API credentials, because the `glossary`, `homophones`, `spoken_word`, and `grammar` modules make real LLM calls. `AUDITA_LLM_API_KEY` and `--llm-api-key` are the preferred provider-neutral credential surfaces, while `OPENROUTER_API_KEY` remains supported as a backward-compatible fallback.
+
+| Environment variable | CLI flag | Default | Purpose |
+| --- | --- | --- | --- |
+| `AUDITA_MODULES` | `--modules` | `glossary,homophones,glossary,spoken_word,grammar` | Comma-separated logical module keys to run; CLI overrides the environment value |
+| `AUDITA_LLM_API_KEY` | `--llm-api-key` | unset | Preferred provider-neutral LLM API credential; CLI overrides both environment-key variants |
+| `AUDITA_VALIDATION_LLM_API_KEY` | `--validation-llm-api-key` | unset | Validation-phase LLM API credential; defaults to the primary LLM API key |
+| `AUDITA_MODEL` | `--model` | `openrouter/google/gemma-4-31b-it` | LLM model name sent to the configured OpenAI-compatible endpoint |
+| `AUDITA_VALIDATION_MODEL` | `--validation-model` | unset | Validation-phase LLM model; defaults to `AUDITA_MODEL` |
+| `AUDITA_BASE_URL` | `--base-url` | `https://openrouter.ai/api/v1` | OpenAI-compatible API base URL |
+| `AUDITA_VALIDATION_BASE_URL` | `--validation-base-url` | unset | Validation-phase OpenAI-compatible API base URL; defaults to `AUDITA_BASE_URL` |
+| `AUDITA_LLM_TIMEOUT_SECONDS` | `--llm-timeout-seconds` | `600` | Per-request timeout in seconds for LLM calls to the configured OpenAI-compatible endpoint |
+| `AUDITA_VALIDATION_LLM_TIMEOUT_SECONDS` | `--validation-llm-timeout-seconds` | unset | Validation-phase per-request timeout in seconds; defaults to `AUDITA_LLM_TIMEOUT_SECONDS` |
+| `AUDITA_VALIDATION_MAX_PROMPT_TOKENS` | `--validation-max-prompt-tokens` | `2048` | Maximum estimated tokens per validation-phase LLM prompt batch |
+| `AUDITA_TARGET_SECTIONS` | `--target-sections` | unset | Exact number of contiguous proposal-stage transcript sections; errors if min/max token bounds cannot be satisfied |
+| `AUDITA_MAX_RETRIES` | `--max-retries` | `3` | Maximum Instructor retries for structured responses |
+| `AUDITA_VALIDATION_MAX_RETRIES` | `--validation-max-retries` | unset | Validation-phase structured-output retries; defaults to `AUDITA_MAX_RETRIES` |
+| `AUDITA_VALIDATION_LLM_CONCURRENCY` | `--validation-llm-concurrency` | unset | Validation-phase LLM concurrency; defaults to `AUDITA_LLM_CONCURRENCY` |
+| `AUDITA_MAX_SECTION_TOKENS` | `--max-section-tokens` | `8192` | Maximum estimated tokens per proposal-stage transcript section |
+| `AUDITA_MIN_SECTION_TOKENS` | `--min-section-tokens` | `2048` | Minimum estimated tokens per proposal-stage transcript section when balancing for concurrency |
+| `AUDITA_GLOSSARY_CONFIDENCE_THRESHOLD` | `--glossary-confidence-threshold` | `0.8` | Minimum confidence required for glossary proposals to survive validation |
+| `AUDITA_GRAMMAR_CONFIDENCE_THRESHOLD` | `--grammar-confidence-threshold` | `0.8` | Minimum confidence required for grammar proposals to survive validation |
+| `AUDITA_HOMOPHONES_CONFIDENCE_THRESHOLD` | `--homophones-confidence-threshold` | `0.8` | Minimum confidence required for homophone proposals to survive validation |
+| `AUDITA_SPOKEN_WORD_CONFIDENCE_THRESHOLD` | `--spoken-word-confidence-threshold` | `0.8` | Minimum confidence required for spoken-word proposals to survive validation |
+| `AUDITA_NORMALIZE_MAX_SEGMENT_GAP` | `--normalize-max-segment-gap` | `4.0` | Same-speaker gaps eligible for deterministic merging |
+| `AUDITA_NORMALIZE_ELLIPSIS_GAP` | `--normalize-ellipsis-gap` | `3.5` | Same-speaker gaps above this value are joined with ` ... ` |
+| `AUDITA_NORMALIZE_MAX_SEGMENT_DURATION` | `--normalize-max-segment-duration` | `60.0` | Maximum merged segment duration |
+| `AUDITA_NORMALIZE_MAX_SEGMENT_TOKENS` | `--normalize-max-segment-tokens` | `2048` | Maximum merged segment prompt payload size |
+| `AUDITA_WORK_DIR` | `--work-dir` | `/tmp/audita` | Per-run scratch diagnostics directory |
+| `AUDITA_WORK_DIR_RETENTION` | `--work-dir-retention` | `auto` | Whether to retain the per-run work directory: `auto`, `always`, or `never` |
+
+Set `AUDITA_MODULES=grammar` to run only the grammar module by default, or override it per command with `--modules`.
+
+Validation-phase LLM settings inherit from the primary `AUDITA_*` LLM settings by default. Set any of the `AUDITA_VALIDATION_*` values only when you want LLM-backed validators to use a different model, endpoint, credential, timeout, retry budget, or concurrency level.
+
+OpenRouter remains the default out of the box:
+
+```sh
+export AUDITA_LLM_API_KEY=your-openrouter-key
+audita process transcript.json --glossary glossary.yaml --output corrected.json
+```
+
+You can point Audita at any OpenAI-compatible endpoint by changing `AUDITA_BASE_URL` and, if needed, `AUDITA_MODEL`. For example, a local vLLM server:
+
+```sh
+export AUDITA_LLM_API_KEY=local-dev-key
+export AUDITA_BASE_URL=http://localhost:8000/v1
+export AUDITA_MODEL=meta-llama/Llama-3.1-8B-Instruct
+audita process transcript.json --glossary glossary.yaml --output corrected.json
+```
+
+Or the actual OpenAI API:
+
+```sh
+export AUDITA_LLM_API_KEY=your-openai-key
+export AUDITA_BASE_URL=https://api.openai.com/v1
+export AUDITA_MODEL=gpt-4.1-mini
+audita process transcript.json --glossary glossary.yaml --output corrected.json
+```
+
+`AUDITA_WORK_DIR` stores per-run diagnostics while processing. Under the default `AUDITA_WORK_DIR_RETENTION=auto`, clean successful runs are removed, while failed runs and successful runs with final skipped corrections are preserved. Use `always` to keep every run directory and `never` to remove successful run directories even when skips remain.
+Failed runs always preserve the run directory and include an authoritative `report.json` alongside normalization and prompt/response diagnostics.
+
+## Prototype Archive
+
+The archived prototype remains importable as `audita_prototype` and is still covered by its original regression suite. This is intentional: the new `audita` package is a framework-oriented rewrite, not a thin wrapper around the old code.
--- a/docs/integrations/scriptorium.md
+++ b/docs/integrations/scriptorium.md
@@ -0,0 +1,322 @@
+# Narratio -> Scriptorium CLI Integration
+
+## 1. Purpose
+
+This document defines how Narratio should invoke Scriptorium through the **public CLI**.
+
+This is a **subprocess integration contract**, not an internal Go API contract.
+
+## 2. Assumptions
+
+- `scriptorium` is installed and available on `PATH`.
+- Scriptorium is configured with `config.yml`.
+- `config.yml` provides `prompt_dir`, `profile_dir`, and `schema_dir` as needed.
+- Prompt and profile libraries are already deployed for the environment.
+- Narratio provides prepared artifact files (for example polished transcript, glossary, previous recap, campaign notes).
+- Initial integration is synchronous subprocess execution.
+- Narratio remains the orchestrator.
+
+In normal operation, Narratio does not need to pass `--prompt-dir` and `--profile-dir` if they are supplied by Scriptorium config.
+
+Narratio may pass `--config <PATH>` when it must use a non-default Scriptorium config file.
+
+## 3. Core Commands Narratio May Call
+
+Primary commands for subprocess integration:
+
+- `scriptorium run`
+- `scriptorium render`
+
+For production generation, use `scriptorium run`.
+
+`scriptorium render` is for debugging, dry-runs, test assertions, and validating command construction without LLM execution.
+
+Note: `scriptorium serve` and HTTP API exist, but they are not the initial integration path.
+
+## 4. Command Selection Guidance
+
+- Use `run` to generate an output artifact.
+- Use `render` to inspect the prepared prompt and effective settings without calling the LLM.
+- Use `render --format json` when Narratio/tests need structured prepare output.
+
+## 5. Recommended `run` Invocation Shape
+
+Production shape:
+
+```bash
+scriptorium run \
+  --prompt <prompt_id> \
+  --input transcript=<processed-transcript-path> \
+  --out <output-artifact-path>
+```
+
+Common optional additions:
+
+- `--config <path>`: use a specific Scriptorium config file.
+- `--profile <profile_id>`: override prompt default profile.
+- `--var name=value` (repeatable): small metadata values.
+- `--input name=path` (repeatable): additional named artifacts.
+- `--timeout <duration>`: per-run timeout override.
+- Runtime model override flags (`--llm-base-url`, `--model`, etc.) only for exceptional/operator-directed cases.
+
+## 6. Recommended `render` Invocation Shape
+
+Human-readable debug shape:
+
+```bash
+scriptorium render \
+  --prompt <prompt_id> \
+  --input transcript=<processed-transcript-path> \
+  --format text
+```
+
+Structured debug/test shape:
+
+```bash
+scriptorium render \
+  --prompt <prompt_id> \
+  --input transcript=<processed-transcript-path> \
+  --format json \
+  --out <render-debug-path>
+```
+
+`render` does **not** call the LLM, does **not** validate model output, and does **not** perform repair.
+
+## 7. Inputs
+
+- Pass inputs as repeated `--input name=path` flags.
+- `name` must match the Prompt Definition input name.
+- Prefer absolute paths, or paths relative to a working directory controlled by Narratio.
+- Pass Audita output as the primary transcript input.
+- Additional inputs may include glossary, previous recap, campaign notes, event logs, final state maps, or other prompt-specific artifacts.
+- Scriptorium reads input files directly; Narratio does not need to inline file content for CLI use.
+
+## 8. Variables
+
+Use repeated `--var name=value` for small metadata values.
+
+Typical examples:
+
+- `session_date`
+- `session_id`
+- `campaign_name`
+- `previous_session_id`
+- `output_kind`
+
+Large content belongs in input files, not `--var` values.
+
+## 9. Prompt IDs and Output Artifact Types
+
+Narratio should treat prompt IDs as configuration, not hardcoded business logic.
+
+Narratio config may map stage/output names to prompt IDs, for example:
+
+- session recap prompt
+- structured event extraction prompt
+- glossary suggestion prompt
+- player-facing summary prompt
+
+Prompt IDs used by Narratio should come from the deployed Scriptorium prompt library.
+
+## 10. Profiles
+
+- Prompts may declare `default_profile`.
+- Narratio may omit `--profile` to use prompt default profile.
+- Narratio may pass `--profile` to force profile selection.
+- This enables environment/profile selection like `local-fast`, `local-quality`, `frontier`, `batch`, or test profiles.
+- Profile names should generally be Narratio configuration values.
+
+## 11. Runtime Overrides
+
+Supported runtime override flags:
+
+- `--llm-base-url`
+- `--model`
+- `--api-key-env`
+- `--temperature`
+- `--max-tokens`
+- `--top-p`
+- `--timeout`
+
+Guidance:
+
+- Keep normal model/runtime settings in Execution Profiles.
+- Use runtime overrides only for explicit per-run exceptions, tests, or operator overrides.
+- Never pass raw API keys on the command line.
+- `--api-key-env` names an environment variable; Narratio must ensure that variable is set in subprocess environment.
+
+## 12. Config Behavior
+
+- Default config path: `/etc/scriptorium/config.yml`.
+- `--config <PATH>` overrides default path.
+- Missing default config is allowed by Scriptorium.
+- If `--config` is provided explicitly, the file must exist and be valid.
+- CLI flags override `config.yml`.
+- `config.yml` overrides built-in application defaults.
+
+Narratio can either:
+
+- rely on system default config path, or
+- carry an explicit config path and pass `--config`.
+
+## 13. Environment Handling
+
+Subprocess environment recommendations:
+
+- Pass through required API-key environment variables referenced by `api_key_env`.
+- Do not pass raw API keys as CLI arguments.
+- Avoid logging full environment dumps.
+- Capture stdout and stderr separately.
+- Use a controlled working directory.
+- Prefer absolute artifact paths.
+
+## 14. Output Handling
+
+For `scriptorium run`:
+
+- Use `--out` when Narratio needs durable artifact files.
+- Without `--out`, artifact content is written to stdout.
+- Preferred orchestration pattern: always use `--out`, then treat the file as stage output artifact.
+- Capture stderr for diagnostics.
+
+For `scriptorium render`:
+
+- Use `--out` to store render diagnostics.
+- Use `--format json` when tests need to inspect selected profile, effective runtime settings, input hashes, prompt hash, and rendered messages.
+
+## 15. Exit Status and Errors
+
+Current CLI behavior (verified from implementation/tests):
+
+- `0`: success.
+- `1`: runtime/parse/config/load/render/generation/IO error.
+- `2`: run completed but output validation failed (`ValidationFailed`).
+
+Additional details:
+
+- On `run`, output artifact write happens before exit code selection. If validation fails, artifact may still be written and exit code is `2`.
+- `stderr` carries both errors and normal run summary output; non-empty stderr alone does not imply failure.
+- `render` returns `0` on success and `1` on failures.
+
+Narratio should treat non-zero exit codes as failed stage execution, but may record generated artifact paths if a run exited `2` and output file exists.
+
+## 16. Recommended Narratio Integration Pattern
+
+1. Build CLI args from Narratio stage configuration.
+2. Use subprocess context cancellation/timeout.
+3. Pass absolute input paths.
+4. Pass `--out` to a session-scoped artifact path.
+5. Add `--var` metadata values.
+6. Optionally add `--config`.
+7. Optionally add `--profile`.
+8. Ensure required API-key env vars are present.
+9. Run subprocess synchronously.
+10. Capture stdout/stderr separately.
+11. On success, store output artifact path and invocation metadata in stage artifacts.
+12. On failure, store exit code and stderr diagnostics in stage status.
+
+## 17. Suggested Narratio Configuration Shape
+
+Illustrative (not required schema):
+
+```yaml
+scriptorium:
+  config_path: /etc/scriptorium/config.yml
+  stages:
+    session_recap:
+      prompt_id: dnd.session_recap
+      profile_id: local-quality   # optional
+      inputs: [transcript, glossary, previous_recap]
+      vars: [session_id, session_date, campaign_name]
+      output_path_template: artifacts/{session_id}/session_recap.md
+      timeout: 2m
+      render_debug: false
+```
+
+The key idea: map Narratio stage/artifact names to prompt ID, optional profile, expected inputs, and output destination.
+
+## 18. Testing Strategy for Narratio Integration
+
+- Use `scriptorium render --format json` to verify command construction without LLM calls.
+- Use dedicated test prompt/profile libraries for integration tests.
+- Use small fixture transcripts.
+- Verify missing-input failure behavior.
+- Verify prompt `default_profile` behavior.
+- Verify explicit `--profile` override behavior.
+- Verify `--config` behavior (default and explicit).
+- Verify output file creation when `--out` is used.
+- Verify stderr capture on failures.
+- Avoid real API keys in tests.
+
+## 19. Security and Privacy Notes
+
+- Never pass raw API keys on command line.
+- Do not log full rendered prompts by default; transcripts may contain sensitive content.
+- Avoid logging prompt content unless explicit debug mode is enabled.
+- Treat generated artifacts as potentially sensitive.
+- Use session-scoped, access-controlled output paths.
+- `api_key_env` names should come from environment management, not embedded secrets.
+
+## 20. Initial D&D Artifact Generation Examples
+
+These are examples only. Use prompt IDs from the deployed prompt library.
+
+Session recap:
+
+```bash
+scriptorium run \
+  --prompt dnd.session_recap \
+  --input transcript=/work/session-42/transcript.polished.md \
+  --input glossary=/work/session-42/glossary.yml \
+  --out /work/session-42/artifacts/session_recap.md
+```
+
+Structured events:
+
+```bash
+scriptorium run \
+  --prompt dnd.structured_events \
+  --input transcript=/work/session-42/transcript.polished.md \
+  --out /work/session-42/artifacts/structured_events.json
+```
+
+Glossary suggestions:
+
+```bash
+scriptorium run \
+  --prompt dnd.glossary_suggestions \
+  --input transcript=/work/session-42/transcript.polished.md \
+  --input previous_recap=/work/session-41/artifacts/session_recap.md \
+  --out /work/session-42/artifacts/glossary_suggestions.md
+```
+
+Player-facing summary:
+
+```bash
+scriptorium run \
+  --prompt dnd.player_summary \
+  --input transcript=/work/session-42/transcript.polished.md \
+  --input structured_events=/work/session-42/artifacts/structured_events.json \
+  --out /work/session-42/artifacts/player_summary.md
+```
+
+## 21. Non-Goals
+
+Initial Narratio integration should not:
+
+- call Scriptorium internal Go packages
+- use HTTP API as the primary path
+- expect Scriptorium to read S3 refs directly
+- make Scriptorium responsible for Narratio stage state
+- make Scriptorium responsible for notification
+- require Scriptorium to understand D&D workflow semantics beyond prompt definitions
+
+## 22. Future Extension Notes
+
+Possible later extensions:
+
+- HTTP API integration
+- S3 artifact references if Scriptorium adds S3 reader support
+- storing render diagnostics alongside generated artifacts
+- token budgeting/prompt-size checks
+- batch execution if Scriptorium later adds batch support
--- a/docs/integrations/seriatim.md
+++ b/docs/integrations/seriatim.md
@@ -0,0 +1,403 @@
+# seriatim
+
+`seriatim` merges per-speaker WhisperX-style JSON transcripts into a single JSON transcript that preserves speaker identity and chronological order.
+
+The current implementation supports the `merge` command. It reads one or more input JSON files, optionally maps each input file to a canonical speaker using `speakers.yml`, sorts all segments by timestamp, detects and resolves overlaps when word-level timing is available, assigns consecutive numeric `id` values, and writes a merged JSON artifact.
+
+## Usage
+
+Run from source:
+
+```sh
+go run ./cmd/seriatim merge \
+  --input-file samples/raw/2026-04-19-Eric_Rakestraw.json \
+  --input-file samples/raw/2026-04-19-Mike_Brown.json \
+  --output-file merged.json
+```
+
+Optional report output:
+
+```sh
+go run ./cmd/seriatim merge \
+  --input-file eric.json \
+  --input-file mike.json \
+  --output-file merged.json \
+  --report-file report.json
+```
+
+## CLI
+
+```text
+seriatim merge [flags]
+```
+
+Global flags:
+
+| Flag | Description |
+| --- | --- |
+| `--help` | Show command help. |
+| `--version` | Show application version. Local builds default to `dev`; release builds inject the release version. |
+
+`merge` flags:
+
+| Flag | Required | Default | Description |
+| --- | --- | --- | --- |
+| `--input-file` | Yes | none | Input transcript JSON file. Repeat once per speaker/input file. |
+| `--output-file` | Yes | none | Merged transcript JSON output path. |
+| `--report-file` | No | none | Optional report JSON output path. |
+| `--speakers` | No | none | Speaker map YAML file. When omitted, input file basenames are used as speaker labels. |
+| `--autocorrect` | No | none | Autocorrect rules YAML file. When omitted, the default `autocorrect` module leaves text unchanged. |
+| `--input-reader` | No | `json-files` | Input reader module. |
+| `--output-modules` | No | `json` | Comma-separated output modules. |
+| `--output-schema` | No | `seriatim-intermediate` | JSON output contract. Allowed values are `seriatim-minimal`, `seriatim-intermediate`, and `seriatim-full`. If omitted, the runtime default is used; consumers that depend on a specific shape should set this explicitly. |
+| `--preprocessing-modules` | No | `validate-raw,normalize-speakers,trim-text` | Comma-separated preprocessing modules, evaluated in order. |
+| `--postprocessing-modules` | No | `detect-overlaps,resolve-overlaps,backchannel,filler,resolve-danglers,coalesce,detect-overlaps,autocorrect,assign-ids,validate-output` | Comma-separated postprocessing modules, evaluated in order. |
+| `--coalesce-gap` | No | `3.0` | Maximum same-speaker gap in seconds for `coalesce`; also used as the `resolve-overlaps` context window. Must be a non-negative float. |
+
+Environment variables:
+
+| Environment Variable | Default | Description |
+| --- | --- | --- |
+| `SERIATIM_OUTPUT_SCHEMA` | `seriatim-intermediate` | Output schema used when `--output-schema` is not explicitly provided. Allowed values are `seriatim-minimal`, `seriatim-intermediate`, and `seriatim-full`. The CLI flag takes precedence. |
+| `SERIATIM_OVERLAP_WORD_RUN_GAP` | `1.0` | Maximum gap in seconds between adjacent timed words when `resolve-overlaps` builds word-run replacement segments. Must be a positive float. |
+| `SERIATIM_OVERLAP_WORD_RUN_REORDER_WINDOW` | `1.0` | Near-start window in seconds for ordering replacement word runs shortest-first. Must be a positive float. |
+| `SERIATIM_BACKCHANNEL_MAX_DURATION` | `2.0` | Maximum duration in seconds for `backchannel` classification. Must be a positive float. |
+| `SERIATIM_FILLER_MAX_DURATION` | `1.25` | Maximum duration in seconds for `filler` classification. Must be a positive float. |
+
+## Input JSON Format
+
+Each input file must be valid JSON with a top-level `segments` array. The current parser accepts the WhisperX segment subset needed for merging:
+
+```json
+{
+  "segments": [
+    {
+      "start": 1.25,
+      "end": 3.5,
+      "text": "Hello there.",
+      "words": [
+        {"word": "Hello", "start": 1.25, "end": 1.55, "score": 0.98},
+        {"word": "there.", "start": 1.7, "end": 2.0}
+      ]
+    }
+  ]
+}
+```
+
+Required segment fields:
+
+- `start`: number, must be `>= 0`.
+- `end`: number, must be `>= start`.
+- `text`: string.
+
+Optional word fields:
+
+- `words`: array of word timing objects.
+- `words[].word`: string.
+- `words[].start`: optional number, must be `>= 0` when present.
+- `words[].end`: optional number, must be `>= start` when present with `start`.
+- `words[].score`: optional number.
+- `words[].speaker`: optional raw speaker label string.
+
+Word-level timing is preserved internally for overlap resolution. If a word is missing `start` or `end`, seriatim keeps the word text, emits a warning in the optional report, and does not use that word as a timing anchor. Word timing is not emitted in the final JSON artifact.
+
+## Speaker Map Format
+
+`speakers.yml` maps input files to canonical speaker names using ordered substring rules:
+
+This file is optional. If `--speakers` is omitted, `seriatim` uses each input file basename as the segment speaker label.
+
+```yaml
+match:
+  - speaker: "Eric Rakestraw"
+    match:
+      - "Eric_Rakestraw"
+      - "Eric"
+
+  - speaker: "Mike Brown"
+    match:
+      - "Mike_Brown"
+      - "mb"
+```
+
+For each `--input-file`, `seriatim` takes the file basename and evaluates the rules in order. The first rule with a matching substring wins, and no later rules are evaluated.
+
+For example, this input:
+
+```text
+samples/raw/2026-04-19-Eric_Rakestraw.json
+```
+
+matches this rule because the basename contains `Eric_Rakestraw`:
+
+```yaml
+- speaker: "Eric Rakestraw"
+  match:
+    - "Eric_Rakestraw"
+```
+
+Important details:
+
+- Matching is against the input file basename, not the full path.
+- Matching is case-insensitive.
+- Rules are evaluated from first to last.
+- Each rule must have a non-empty `speaker`.
+- Each rule must have at least one non-empty `match` string.
+- Duplicate speaker names are invalid.
+- Every input file must match at least one rule or the command fails.
+
+Deprecated old format:
+
+```yaml
+inputs:
+  eric.json:
+    speaker: "Eric Rakestraw"
+```
+
+The old `inputs:` direct mapping format is no longer supported.
+
+## Output JSON Format
+
+`--output-modules json` controls the writer. `--output-schema` controls the JSON contract that writer serializes.
+
+The named schemas are stable public contracts. If a consumer depends on a specific shape, it should request that schema explicitly at runtime. The runtime default selection may change in a future release.
+
+The `seriatim-intermediate` schema is the current default selection when neither `--output-schema` nor `SERIATIM_OUTPUT_SCHEMA` is set. It stays close to the minimal schema, but adds optional `categories` on each segment:
+
+```json
+{
+  "metadata": {
+    "application": "seriatim",
+    "version": "dev",
+    "output_schema": "seriatim-intermediate"
+  },
+  "segments": [
+    {
+      "id": 1,
+      "start": 1.25,
+      "end": 3.5,
+      "speaker": "Eric Rakestraw",
+      "text": "Hello there.",
+      "categories": ["backchannel"]
+    }
+  ]
+}
+```
+
+The `seriatim-full` schema uses the full seriatim envelope:
+
+```json
+{
+  "metadata": {
+    "application": "seriatim",
+    "version": "dev",
+    "input_reader": "json-files",
+    "input_files": ["eric.json", "mike.json"],
+    "preprocessing_modules": ["validate-raw", "normalize-speakers", "trim-text"],
+    "postprocessing_modules": ["detect-overlaps", "resolve-overlaps", "backchannel", "filler", "resolve-danglers", "coalesce", "detect-overlaps", "autocorrect", "assign-ids", "validate-output"],
+    "output_modules": ["json"]
+  },
+  "segments": [
+    {
+      "id": 1,
+      "source": "eric.json",
+      "source_segment_index": 0,
+      "speaker": "Eric Rakestraw",
+      "start": 1.25,
+      "end": 3.5,
+      "text": "Hello there.",
+      "overlap_group_id": 1
+    },
+    {
+      "id": 2,
+      "source": "eric.json",
+      "source_ref": "word-run:1:1:1",
+      "derived_from": ["eric.json#0"],
+      "speaker": "Eric Rakestraw",
+      "start": 2.0,
+      "end": 2.5,
+      "text": "Resolved word run",
+      "categories": ["backchannel"]
+    }
+  ],
+  "overlap_groups": [
+    {
+      "id": 1,
+      "start": 1.25,
+      "end": 4.0,
+      "segments": ["eric.json#0", "mike.json#0"],
+      "speakers": ["Eric Rakestraw", "Mike Brown"],
+      "class": "unknown",
+      "resolution": "unresolved"
+    }
+  ]
+}
+```
+
+The `seriatim-minimal` schema emits minimal metadata and compact ordered segments:
+
+```json
+{
+  "metadata": {
+    "application": "seriatim",
+    "version": "dev",
+    "output_schema": "seriatim-minimal"
+  },
+  "segments": [
+    {
+      "id": 1,
+      "start": 1.25,
+      "end": 3.5,
+      "speaker": "Eric Rakestraw",
+      "text": "Hello there."
+    }
+  ]
+}
+```
+
+Minimal output intentionally omits categories, overlap groups, source/provenance fields, and pipeline configuration metadata.
+
+Intermediate output intentionally omits overlap groups and source/provenance fields, but keeps optional `categories` and minimal metadata.
+
+Segments are sorted deterministically by:
+
+```text
+(start, end, source, source_segment_index/source_ref, speaker)
+```
+
+Final segment IDs are assigned after sorting and start at `1`.
+
+The public Go output contract is available from:
+
+```go
+import "gitea.maximumdirect.net/eric/seriatim/schema"
+```
+
+The same package embeds machine-readable JSON Schemas in `schema/full-output.schema.json`, `schema/intermediate-output.schema.json`, and `schema/minimal-output.schema.json`. The default `validate-output` postprocessor validates the selected output shape and verifies final segment IDs are present, sequential, and start at `1`.
+
+## Overlap Detection
+
+The default postprocessing pipeline detects overlapping segment groups.
+
+Overlap behavior:
+
+- A strict timing overlap is required: `next.start < current_group_end`.
+- Segments that only touch at a boundary are not grouped.
+- Groups require at least two distinct speakers.
+- Transitive overlaps are grouped together.
+- Segments in detected groups receive `overlap_group_id`.
+- `overlap_groups[].segments` contains stable references in `source#source_segment_index` format.
+- `class` is currently `unknown`.
+- `resolution` is `unresolved` until `resolve-overlaps` replaces the group.
+
+## Overlap Resolution
+
+The default postprocessing pipeline runs `detect-overlaps`, then `resolve-overlaps`, then `backchannel`, then `filler`, then `resolve-danglers`, then `coalesce`, then a second `detect-overlaps` pass.
+
+For each detected overlap group, `resolve-overlaps` uses preserved WhisperX word timing to build smaller word-run replacement segments:
+
+- The resolution window expands the detected overlap group by `--coalesce-gap` seconds on both sides.
+- Nearby same-speaker context segments are included when they intersect the expanded window and their start or end is within `--coalesce-gap` of the original overlap boundary.
+- Once a segment is selected for replacement, all timed words from that segment participate in word-run construction; the window controls segment selection, not per-word clipping.
+- Context segments that are part of another detected overlap group are not pulled into the current group.
+- Untimed words are included in replacement text in original word order when nearby timed words create a replacement run.
+- Untimed words do not affect replacement segment start/end times or word-run gap splitting.
+- Words for the same speaker are merged into one run when the gap between adjacent words is no greater than `SERIATIM_OVERLAP_WORD_RUN_GAP`.
+- The default word-run gap is `1.0` seconds.
+- Set `SERIATIM_OVERLAP_WORD_RUN_GAP` to a positive number of seconds to override the default.
+- Near-start replacement word runs are reordered so shorter segments come first when adjacent starts are within `SERIATIM_OVERLAP_WORD_RUN_REORDER_WINDOW`.
+- The default word-run reorder window is `1.0` seconds.
+- Set `SERIATIM_OVERLAP_WORD_RUN_REORDER_WINDOW` to a positive number of seconds to override the default.
+- Replacement segment text is built by joining word text with single spaces.
+- Replacement segments include `source_ref` and `derived_from`.
+- Replacement segments omit `source_segment_index` because they are derived from one or more original segments.
+- Resolved overlap groups are removed before the second detection pass.
+- Replacement segments are left without `overlap_group_id` until the second detection pass annotates any remaining overlap.
+- If a speaker has no usable word timing in a group, that speaker's original segment is kept.
+- If no speakers in a group have usable word timing, the original group and annotations remain unchanged.
+
+## Backchannels
+
+The default pipeline runs `backchannel` before `coalesce`. It tags short acknowledgement segments with:
+
+```json
+"categories": ["backchannel"]
+```
+
+Backchannel matching is case-insensitive, ignores punctuation for matching and word-count purposes, trims surrounding whitespace, and requires a matching acknowledgement phrase, no more than three whitespace-delimited words, and duration no greater than `SERIATIM_BACKCHANNEL_MAX_DURATION` seconds. The default maximum duration is `2.0` seconds.
+
+## Fillers
+
+The default pipeline runs `filler` after `backchannel` and before `coalesce`. It tags short filler utterances with:
+
+```json
+"categories": ["filler"]
+```
+
+Filler matching is case-insensitive, ignores punctuation for matching and word-count purposes, trims surrounding whitespace, and requires only filler tokens such as `um`, `uh`, `er`, `erm`, `ah`, `eh`, `hmm`, `mm`, or repeated combinations of those tokens. Matching segments must contain no more than three whitespace-delimited words and have duration no greater than `SERIATIM_FILLER_MAX_DURATION` seconds. The default maximum duration is `1.25` seconds.
+
+## Dangler Resolution
+
+The default pipeline runs `resolve-danglers` before `coalesce` and before the second overlap detection pass. It repairs short derived fragments when they share provenance with a nearby segment:
+
+- Dangling-end fragments have no more than two words and end in punctuation.
+- Dangling-start fragments have no more than two words.
+- Matching uses same-speaker segments with any shared `derived_from` value.
+- Merged segments use `source_ref` values such as `resolve-danglers:1`, keep the target segment's transcript position, and union `derived_from`.
+
+## Coalescing
+
+The default pipeline runs `coalesce` after `resolve-danglers` and before the second overlap detection pass. It merges adjacent same-speaker segments in the transcript's current order when `next.start - current.end <= --coalesce-gap`.
+
+Coalesced segments use `source_ref` values such as `coalesce:1`, include `derived_from`, and omit `source_segment_index`.
+
+Different-speaker backchannel and filler segments do not block coalescing of surrounding same-speaker segments. Same-speaker backchannel and filler segments are merged normally when they are within `--coalesce-gap`. When same-speaker segments are coalesced, any `backchannel` or `filler` category from the merged inputs is dropped from the coalesced segment.
+
+## Autocorrect
+
+Autocorrect is included in the default postprocessing pipeline. If `--autocorrect` is omitted, the module leaves transcript text unchanged and records a skip event in the optional report.
+
+Enable corrections by passing `--autocorrect`:
+
+```sh
+go run ./cmd/seriatim merge \
+  --input-file input.json \
+  --autocorrect autocorrect.yml \
+  --output-file merged.json
+```
+
+`autocorrect.yml` format:
+
+```yaml
+autocorrect:
+  - target: "Hrank"
+    match:
+      - "hrank"
+      - "Frank"
+
+  - target: "Mike Brown"
+    match:
+      - "Mike Pat"
+```
+
+Matching behavior:
+
+- Matching is case-sensitive.
+- Matches apply only to whole tokens, not substrings inside larger words.
+- Punctuation and whitespace can surround a match.
+- Multi-word and hyphenated matches are supported.
+- Duplicate match strings are invalid, including duplicates across separate rules.
+
+## Current Limitations
+
+- Only JSON input is supported.
+- Overlap resolution depends on WhisperX word timing; groups without usable word timing remain unresolved.
+- Alternate output formats are not implemented yet.
+
+## Release Builds
+
+Local builds record version metadata as `dev`. Release builds should inject the release version with `ldflags`:
+
+```sh
+go build -ldflags "-X gitea.maximumdirect.net/eric/seriatim/internal/buildinfo.Version=v1.0.0" ./cmd/seriatim
+```