Document Audita polish integration

architecture.md

@@ -4,7 +4,7 @@
 
 `narratio` is a Go orchestrator for D&D session processing. It coordinates a stage-based pipeline from recorded audio through transcript and artifact generation, while preserving durable run state for skip/rerun/resume behavior.
 
-The repository currently implements the orchestration scaffold, local state model, and stage framework, including real WhisperX and Seriatim adapters plus real `prepare`/`transcribe`/`merge` stages. It intentionally does **not** yet implement real Audita, analyzer, remote archive, or notification integrations.
+The repository currently implements the orchestration scaffold, local state model, and stage framework, including real WhisperX/Seriatim/Audita adapters plus real `prepare`/`transcribe`/`merge`/`polish` stages. It intentionally does **not** yet implement real analyzer, remote archive, or notification integrations.
 
 ## 2. Design Goals
 
@@ -37,16 +37,17 @@ Implemented now:
 - Real `prepare` stage (input resolution/materialization/provenance).
 - Real `transcribe` stage (prepared-audio discovery, WhisperX adapter execution, JSON transcript validation, provenance metadata).
 - Real `merge` stage (raw-transcript discovery, Seriatim adapter execution, merged/report JSON validation, provenance metadata).
+- Real `polish` stage (merged-transcript discovery, Audita adapter execution, processed/report JSON validation, provenance metadata).
 - Real WhisperX HTTP adapter implementation (multipart POST + retries + timeout + atomic output writes).
 - Real Seriatim subprocess adapter implementation (deterministic CLI/env construction + output/report JSON validation).
-- Placeholder downstream stages (`normalize`, `polish`, `analyze`, `archive`, `notify`) with adapter contract calls.
+- Real Audita subprocess adapter implementation (deterministic CLI/env construction + credential env handling + output/report JSON validation).
+- Placeholder downstream stages (`normalize`, `analyze`, `archive`, `notify`) with adapter contract calls.
 - Adapter interfaces and fake/no-op implementations for all external boundaries.
 - Reusable subprocess helper and generated YAML/config writing helper.
 - Test coverage across config, manifest, artifacts, planning, runner control, and adapters.
 
 Still planned/future:
 
-- Real Audita adapter.
 - Real analyzer integration.
 - Real remote archive/storage backend (S3/SFTP/etc).
 - Real notification backend.
@@ -70,7 +71,8 @@ Execution status:
 - `prepare` is implemented with real local filesystem behavior.
 - `transcribe` is implemented and validates raw transcript JSON outputs.
 - `merge` is implemented and validates merged transcript/report JSON outputs.
-- `normalize`/`polish`/`analyze`/`archive`/`notify` remain placeholders.
+- `polish` is implemented and validates processed transcript/report JSON outputs.
+- `normalize`/`analyze`/`archive`/`notify` remain placeholders.
 
 ## 6. CLI Commands
 
@@ -126,6 +128,24 @@ Seriatim config keys:
 - `pipeline.seriatim.report` (default: `true`)
 - optional tuning under `pipeline.seriatim.env.*` (must be `> 0` when provided)
 
+Audita config keys:
+
+- `pipeline.audita.binary` (required)
+- `pipeline.audita.timeout` (default: `3h`)
+- `pipeline.audita.llm_api_key_env` (required; default: `AUDITA_LLM_API_KEY`)
+- `pipeline.audita.modules` (default sequence: `glossary,homophones,glossary,spoken_word,grammar,homophones,glossary`)
+- `pipeline.audita.base_url` (default: `https://openrouter.ai/api/v1`)
+- `pipeline.audita.model` (default: `openrouter/google/gemma-4-31b-it`)
+- `pipeline.audita.llm_concurrency` (default: `1`, must be `> 0`)
+- `pipeline.audita.validation_model` (default: empty string)
+- `pipeline.audita.validation_llm_concurrency` (default: `1`, must be `> 0`)
+- `pipeline.audita.report` (default: `true`)
+
+Audita secret-handling policy:
+
+- `llm_api_key_env` stores only the environment variable **name**.
+- API key values are read from the process environment at runtime and are not stored in `pipeline.yml`, manifest metadata, generated configs, or logs.
+
 Validation currently enforces:
 
 - `pipeline.workspace.root` is required.
@@ -138,6 +158,14 @@ Validation currently enforces:
 - `pipeline.seriatim.output_schema` must be one of `seriatim-minimal|seriatim-intermediate|seriatim-full`.
 - `pipeline.seriatim.coalesce_gap` must be `>= 0`.
 - optional `pipeline.seriatim.env.*` values must be `> 0` when provided.
+- `pipeline.audita.binary` is required.
+- `pipeline.audita.timeout` must parse as Go duration.
+- `pipeline.audita.llm_api_key_env` is required.
+- `pipeline.audita.modules` must be non-empty and each entry must be one of `glossary|homophones|spoken_word|grammar`.
+- `pipeline.audita.base_url` must be a valid URL when provided.
+- `pipeline.audita.model` is required.
+- `pipeline.audita.llm_concurrency` must be `> 0`.
+- `pipeline.audita.validation_llm_concurrency` must be `> 0`.
 - `session.session_id` is required.
 - `session.inputs.speakers_file`, `autocorrect_file`, `glossary_file` are required.
 - At least one audio source: `session.inputs.audio_dir` or non-empty `session.inputs.audio_files`.
@@ -258,7 +286,19 @@ Current `Declares` role:
   - Invokes `seriatim.Runner` using canonical merged/report/log/generated-config paths under the session workdir.
   - Validates merged transcript output JSON and (when enabled) report JSON before stage success.
   - Records merged/report output refs plus stage metadata (input paths/count, Seriatim settings, output paths, adapter duration/exit metadata).
-- `normalize`/`polish`/`analyze`/`archive`/`notify` (placeholder):
+- `polish` (real):
+  - Discovers merged transcript input from `manifest.stages.merge.outputs` (`kind=transcript_merged`) or fallback `work/.../transcripts/merged.json`.
+  - Validates merged transcript as JSON and requires prepared `inputs/glossary.yml`.
+  - Invokes `audita.Runner` using canonical paths:
+    - `transcripts/processed.json`
+    - `artifacts/audita.report.json` (when enabled)
+    - `artifacts/audita-work`
+    - `logs/audita.stdout.log`
+    - `logs/audita.stderr.log`
+    - `config/audita.generated.yml`
+  - Validates processed transcript output JSON with required top-level `segments` array and validates report JSON when enabled.
+  - Records processed/report outputs plus non-secret provenance metadata (module/model/base-url/concurrency settings, paths, adapter duration/exit metadata, credential env-var name/presence).
+- `normalize`/`analyze`/`archive`/`notify` (placeholder):
   - Return placeholder metadata.
   - Optionally call adapters using expected request/result contract shapes.
 
@@ -283,7 +323,7 @@ Adapter boundaries (`internal/adapters/*`):
 - `storage.Backend`
 - `notify.Sender`
 
-All adapters currently have fake/no-op implementations for tests/scaffold execution. WhisperX has a real HTTP adapter (`internal/adapters/whisperx/http.go`) and Seriatim has a real subprocess adapter (`internal/adapters/seriatim/subprocess.go`).
+All adapters currently have fake/no-op implementations for tests/scaffold execution. WhisperX has a real HTTP adapter (`internal/adapters/whisperx/http.go`), Seriatim has a real subprocess adapter (`internal/adapters/seriatim/subprocess.go`), and Audita has a real subprocess adapter (`internal/adapters/audita/subprocess.go`).
 
 ### Subprocess helper
 
@@ -318,11 +358,11 @@ Stale detection is intentionally TODO (`run_control.go`) pending checksum-based
 
 - Structured logger is initialized via `internal/logging` (`slog` text handler).
 - Runner emits concise stage lifecycle logs (skip/start/success/fail + manifest save points).
-- Real WhisperX/Seriatim stages already use explicit output/log/config paths; placeholder downstream subprocess adapters keep that same boundary pattern.
+- Real WhisperX/Seriatim/Audita stages already use explicit output/log/config paths; placeholder downstream subprocess adapters keep that same boundary pattern.
 
 ### Long-running stage expectations
 
-The architecture expects long-running stages (especially WhisperX, Seriatim, and Audita). WhisperX and Seriatim already use context timeout/cancellation in their adapters (with WhisperX retry logic); other long-running integrations are still pending.
+The architecture expects long-running stages (especially WhisperX, Seriatim, and Audita). WhisperX/Seriatim/Audita already use context timeout/cancellation in their adapters (with WhisperX retry logic); other long-running integrations are still pending.
 
 ## 13. Testing Strategy
 
@@ -336,9 +376,11 @@ Current tests verify scaffold behavior without real external services:
 - Prepare-stage input materialization/provenance/idempotency.
 - Real transcribe-stage audio discovery/concurrency/failure handling/output validation/provenance.
 - Real merge-stage input discovery/validation, Seriatim adapter failure handling, merged/report output validation, and provenance recording.
+- Real polish-stage merged-input discovery/validation, Audita adapter failure handling, processed/report output validation, and provenance recording.
 - Adapter fake behavior and error propagation.
 - WhisperX HTTP adapter behavior (request shape, retry policy, timeout/cancel, JSON validation, atomic writes).
 - Seriatim subprocess adapter behavior (arg/env construction, timeout/failure handling, output/report JSON validation).
+- Audita subprocess adapter behavior (arg/env construction, credential handling, timeout/failure handling, output/report JSON validation).
 - Subprocess helper behavior (success/failure/timeout/log capture).
 
 Tests intentionally avoid hardcoding arbitrary operational default values.
@@ -348,12 +390,11 @@ Tests intentionally avoid hardcoding arbitrary operational default values.
 Recommended implementation sequence (one focused boundary at a time):
 
 1. Implement real `normalize` transcript transformation/validation.
-2. Implement real `polish` using Audita adapter + checkpoint/log handling.
-3. Implement real `analyze` adapter integration and artifact validation.
-4. Implement real `archive` remote backend behavior.
-5. Implement real `notify` backend.
-6. Add checksum-based stale detection and stale status transitions.
-7. Add selective parallelism where architecturally safe (`transcribe` fan-out and/or downstream-safe boundaries).
+2. Implement real `analyze` adapter integration and artifact validation.
+3. Implement real `archive` remote backend behavior.
+4. Implement real `notify` backend.
+5. Add checksum-based stale detection and stale status transitions.
+6. Add selective parallelism where architecturally safe (`transcribe` fan-out and/or downstream-safe boundaries).
 
 Each step must preserve existing package boundaries and manifest-based control flow.