Complete Phase 19 Go rollout documentation

2026-05-12 13:46:14 +00:00
parent 68e2d9b549
commit fb59cb21b9
5 changed files with 345 additions and 65 deletions
--- a/README.md
+++ b/README.md
@@ -1,61 +1,235 @@
-# Audita (Go Rewrite)
+# Audita (Go)

-Audita is a transcript polishing CLI. The Go implementation at the repository root is canonical going forward, and is being delivered compatibility-first against the frozen Python reference in `python/`.
+Audita is a transcript polishing CLI.

-## Current Status: Phase 3 Deterministic Foundation
+The Go implementation in this repository is the active implementation. It runs a full default correction pipeline over transcript JSON using glossary context, LLM-backed proposal generation, validator chains, deterministic proposal application, and structured reports/diagnostics.

-The current `audita process` implementation includes deterministic input handling and foundation behavior:
+## What Audita does

- typed transcript parsing for both accepted top-level forms (`[]` and `{ "segments": [...] }`);
- typed glossary YAML parsing;
- transcript and glossary schema validation with actionable errors;
- deterministic same-speaker normalization;
- sequential normalized segment IDs starting at `1`;
- deterministic token estimation;
- contiguous transcript chunking and section summaries;
- normalized transcript JSON output;
- minimal structured `--report-json` output including normalization and chunking summary fields;
- minimal normalization diagnostics artifacts in per-run work directories;
- chunking diagnostics artifacts in per-run work directories;
- subprocess-safe stdout/stderr behavior.
+`audita process` performs:
+- transcript/glossary schema validation;
+- deterministic normalization and chunking;
+- default module sequence:
+  - `glossary`
+  - `homophones`
+  - `glossary`
+  - `spoken_word`
+  - `grammar`
+- glossary-backed domain/acoustic corrections;
+- conservative homophone and likely mistranscription corrections;
+- conservative spoken-word cleanup (dysfluencies/fillers) with semantic guardrails;
+- grammar/punctuation/capitalization/formatting cleanup;
+- machine-readable process and module reports;
+- per-run diagnostics artifacts with secret redaction.

-The repository also now contains deterministic proposal foundation packages (models, preview, application, skip reporting, and framework contracts) with fixture/golden tests.
+## Build and install

-Still not implemented in the Go CLI runtime path:
-
- real module execution pipeline;
- real `glossary`, `homophones`, `spoken_word`, or `grammar` module behavior;
- LLM proposal generation;
- validator-chain execution;
- concrete structured LLM client implementation;
- final end-to-end transcript polishing behavior.
-
-## Quick Start (Go)
-
-Run tests:
+Build a local binary:

 ```sh
-go test ./...
+go build -o ./bin/audita ./cmd/audita
+```
+
+Install into your Go bin directory:
+
+```sh
+go install ./cmd/audita
 ```

 Run help:

 ```sh
-go run ./cmd/audita --help
-go run ./cmd/audita process --help
+audita --help
+audita process --help
 ```

-Run current deterministic flow:
+## Test
+
+Run the full test suite:

 ```sh
-go run ./cmd/audita process transcript.json --glossary glossary.yaml --output corrected.json
+go test ./...
 ```

-Without `--output`, transcript JSON is written to stdout.
+Normal tests are deterministic and do not require real LLM credentials or Python dependencies.

-## Repository Notes
+## Basic usage

- `python/` contains the frozen Python implementation used as behavioral reference during the Go port.
- Architecture and rewrite guidance:
-  - `docs/architecture.md`
-  - `docs/rewrite-notes.md`
+Required inputs:
+- transcript JSON path (positional argument)
+- `--glossary <glossary.yaml>`
+
+Default full pipeline (recommended local example):
+
+```sh
+audita process transcript.json \
+  --glossary glossary.yaml \
+  --output corrected.json \
+  --report-json report.json
+```
+
+Explicit module override:
+
+```sh
+audita process transcript.json \
+  --glossary glossary.yaml \
+  --modules glossary,homophones,grammar \
+  --output corrected.json \
+  --report-json report.json
+```
+
+Emit transcript JSON to stdout (no `--output`):
+
+```sh
+audita process transcript.json --glossary glossary.yaml
+```
+
+Diagnostics/work-dir control:
+
+```sh
+audita process transcript.json \
+  --glossary glossary.yaml \
+  --work-dir /tmp/audita \
+  --work-dir-retention auto \
+  --output corrected.json \
+  --report-json report.json
+```
+
+## Stdout/stderr and orchestration behavior
+
+- With `--output`, stdout should be empty on success.
+- Without `--output`, stdout contains transcript JSON only on success.
+- `--report-json` writes report JSON to file; report JSON is never printed to stdout.
+- stderr is for human-readable warnings/errors.
+
+For parent-process integration guidance, see:
+- [`docs/subprocess-operations.md`](docs/subprocess-operations.md)
+
+For orchestrated runs, use both `--output` and `--report-json`.
+
+## Configuration
+
+Precedence:
+1. defaults
+2. environment (`AUDITA_*`)
+3. CLI flags
+
+### Module sequence
+
+- `AUDITA_MODULES` (CSV, e.g. `glossary,homophones,glossary,spoken_word,grammar`)
+- CLI override: `--modules`
+
+### Primary LLM settings
+
+- `AUDITA_LLM_API_KEY` (or `OPENROUTER_API_KEY` fallback)
+- `AUDITA_MODEL`
+- `AUDITA_BASE_URL`
+- `AUDITA_LLM_TIMEOUT_SECONDS`
+- `AUDITA_MAX_RETRIES`
+- `AUDITA_LLM_CONCURRENCY`
+
+CLI overrides:
+- `--llm-api-key`
+- `--model`
+- `--base-url`
+- `--llm-timeout-seconds`
+- `--max-retries`
+
+### Validation LLM settings
+
+- `AUDITA_VALIDATION_LLM_API_KEY`
+- `AUDITA_VALIDATION_MODEL`
+- `AUDITA_VALIDATION_BASE_URL`
+- `AUDITA_VALIDATION_LLM_TIMEOUT_SECONDS`
+- `AUDITA_VALIDATION_MAX_RETRIES`
+- `AUDITA_VALIDATION_LLM_CONCURRENCY`
+- `AUDITA_VALIDATION_MAX_PROMPT_TOKENS`
+
+CLI overrides:
+- `--validation-llm-api-key`
+- `--validation-model`
+- `--validation-base-url`
+- `--validation-llm-timeout-seconds`
+- `--validation-max-retries`
+- `--validation-llm-concurrency`
+- `--validation-max-prompt-tokens`
+
+Validation LLM inheritance behavior:
+- unset validation fields inherit from primary LLM config;
+- set validation fields override primary values for validation calls only.
+
+### Confidence thresholds
+
+- `AUDITA_GLOSSARY_CONFIDENCE_THRESHOLD`
+- `AUDITA_HOMOPHONES_CONFIDENCE_THRESHOLD`
+- `AUDITA_SPOKEN_WORD_CONFIDENCE_THRESHOLD`
+- `AUDITA_GRAMMAR_CONFIDENCE_THRESHOLD`
+
+CLI overrides:
+- `--glossary-confidence-threshold`
+- `--homophones-confidence-threshold`
+- `--spoken-word-confidence-threshold`
+- `--grammar-confidence-threshold`
+
+### Normalization and chunking
+
+- `AUDITA_NORMALIZE_MAX_SEGMENT_GAP`
+- `AUDITA_NORMALIZE_ELLIPSIS_GAP`
+- `AUDITA_NORMALIZE_MAX_SEGMENT_DURATION`
+- `AUDITA_NORMALIZE_MAX_SEGMENT_TOKENS`
+- `AUDITA_MAX_SECTION_TOKENS`
+- `AUDITA_MIN_SECTION_TOKENS`
+- `AUDITA_TARGET_SECTIONS`
+
+CLI overrides:
+- `--normalize-max-segment-gap`
+- `--normalize-ellipsis-gap`
+- `--normalize-max-segment-duration`
+- `--normalize-max-segment-tokens`
+- `--max-section-tokens`
+- `--min-section-tokens`
+- `--target-sections`
+
+### Work-dir and retention
+
+- `AUDITA_WORK_DIR`
+- `AUDITA_WORK_DIR_RETENTION` (`auto`, `always`, `never`)
+
+CLI overrides:
+- `--work-dir`
+- `--work-dir-retention`
+
+Retention summary:
+- `always`: keep all run directories.
+- `never`: keep successful run directories.
+- `auto`: keep failed runs and successful runs with skipped/rejected corrections.
+
+## Report and diagnostics
+
+Per-run diagnostics include:
+- source transcript artifacts;
+- normalized transcript artifact;
+- normalization summary;
+- chunking summary;
+- invocation metadata;
+- redacted effective config;
+- prompt/response diagnostics for module and validator LLM interactions;
+- `report.json`;
+- `error.log` on failure.
+
+Optional external report output:
+- `--report-json <path>`
+
+## Legacy Python reference
+
+The original Python implementation is preserved in [`python/`](python/) as a legacy/reference implementation for parity history and migration context.
+For migration guidance, see [`docs/migration-from-python.md`](docs/migration-from-python.md).
+
+Parity fixture notes and intentional differences:
+- [`docs/python-parity.md`](docs/python-parity.md)
+
+## Additional docs
+
+- Architecture: [`docs/architecture.md`](docs/architecture.md)
+- Rewrite history and phase notes: [`docs/rewrite-notes.md`](docs/rewrite-notes.md)
+- Migration from Python to Go: [`docs/migration-from-python.md`](docs/migration-from-python.md)
--- a/docs/architecture.md
+++ b/docs/architecture.md
@@ -2,10 +2,10 @@

 ## Scope and intent
 This document describes:
- the current implemented Go architecture; and
- the intended final architecture for later rewrite phases.
+- the implemented Go architecture used in production today; and
+- historical rewrite-phase context that explains how the architecture was delivered.

-Status labels are explicit so future engineers and LLM agents do not assume unimplemented behavior exists.
+Status labels are explicit so future engineers and LLM agents do not infer missing behavior that is not actually missing.

 ## Current implementation status
 Implemented today:
@@ -42,9 +42,6 @@ Implemented today:
 - Explicit runtime support for `--modules homophones` through the production runner path.
 - Explicit runtime support for `--modules spoken_word` through the production runner path.

-Not implemented in CLI runtime path today:
- Rollout and Python retirement work (Phase 19).
-
 Current reality:
 - all production modules exist and are wired into the default runtime path.
 - a normal `audita process` run without `--modules` now executes the full sequence:
@@ -66,7 +63,7 @@ Phase sequencing note:
 - Phase 16 default full pipeline integration is complete;
 - Phase 17 parity fixture suite is complete;
 - Phase 18 operational hardening and subprocess integration are complete;
- next recommended phase is Phase 19 (documentation, rollout, and Python retirement).
+- Phase 19 documentation/rollout completion is complete.

 ## Actual Go package layout

@@ -468,8 +465,8 @@ Current runtime note:
 - default non-explicit runs usually have no module-level skipped corrections, so `auto` commonly removes clean successful run directories.
 - explicit grammar/glossary/homophones/spoken_word runs can produce validator rejections and application skips, which are reflected in reports and retention input.

-Intentionally deferred to module/LLM phases:
- real domain proposal prompts and production module implementations remain tied to later module phases.
+Phase note:
+- Earlier-phase documentation deferred module and LLM prompt work; those deferred items were completed by Phases 12-16.

 ## Current tests and quality posture
 Implemented tests currently cover:
@@ -507,8 +504,9 @@ The runtime now includes hardened subprocess behavior for parent-process callers

 Operational caller guidance is documented in [`docs/subprocess-operations.md`](docs/subprocess-operations.md).

-## Remaining work (Phase 19)
-Remaining rewrite work is documentation/rollout/retirement-oriented:
- repository-level rollout documentation and migration guidance;
- Python retirement/archival decisions and related documentation cleanup;
- final release-facing communication of Go-primary operational guidance.
+## Final status (Phase 19 complete)
+- Go Audita is the active implementation.
+- Default full module-sequence runtime is implemented and tested.
+- Parity fixtures and operational hardening coverage are in place.
+- Python is preserved as a legacy/reference implementation in `python/` and is no longer the primary operational path.
+- Migration and rollout guidance is documented in [`docs/migration-from-python.md`](docs/migration-from-python.md).
--- a/docs/migration-from-python.md
+++ b/docs/migration-from-python.md
@@ -0,0 +1,84 @@
+# Migrating from Python Audita to Go Audita
+
+This guide describes the operational migration from the legacy Python Audita implementation to the Go Audita implementation in this repository.
+
+## Status summary
+
+- The Go CLI is now the primary Audita implementation.
+- The Go `audita process` command is intended to replace the Python CLI for normal operation.
+- The Python implementation under [`python/`](../python/) is preserved as a legacy/reference implementation for parity history and troubleshooting context.
+
+## What changes for operators
+
+Use the Go binary as the integration target in orchestrators and parent processes.
+
+Default Go runtime behavior executes the full module sequence:
+1. `glossary`
+2. `homophones`
+3. `glossary`
+4. `spoken_word`
+5. `grammar`
+
+Repeated stages are resolved deterministically in reports (for example `glossary_1`, `glossary_2`).
+
+## Recommended invocation pattern
+
+For orchestrated runs, use explicit output files:
+
+```sh
+audita process <transcript.json> \
+  --glossary <glossary.yaml> \
+  --output <corrected-transcript.json> \
+  --report-json <report.json>
+```
+
+Why:
+- `--output` keeps stdout empty on success, simplifying subprocess integration.
+- `--report-json` provides machine-readable run metadata independent of stderr.
+
+Without `--output`, stdout contains transcript JSON only on successful runs.
+
+## Diagnostics and retention behavior
+
+- Each run creates a diagnostics run directory when initialization succeeds.
+- Failed runs retain diagnostics and include `error.log`.
+- Retention mode is controlled by `AUDITA_WORK_DIR_RETENTION` / `--work-dir-retention`:
+  - `always`: keep all run directories.
+  - `never`: keep successful run directories.
+  - `auto`: keep failed runs and successful runs with skipped/rejected corrections.
+
+For subprocess behavior and pipe-handling guidance, see:
+- [`docs/subprocess-operations.md`](subprocess-operations.md)
+
+## Parity notes and known differences
+
+Python-vs-Go parity fixtures and intentional differences are documented in:
+- [`docs/python-parity.md`](python-parity.md)
+
+Known open parity gaps are tracked there and should be treated as real gaps, not intentional differences. Current documented gaps include:
+- broader direct import/use of Python fixture corpus;
+- a repository-standard Python+Go side-by-side runner command;
+- wider transcript/glossary corpus coverage.
+
+## Testing expectations
+
+- Normal `go test ./...` does not require real LLM credentials.
+- Normal `go test ./...` does not require Python dependencies.
+- Deterministic fake-LLM fixtures are used for routine CI-friendly testing.
+
+## Legacy Python status
+
+The Python implementation remains in-repo as a legacy/reference baseline. It is not the primary operational path.
+
+Do not route new production orchestration to Python unless you are doing explicit parity/debug work.
+
+## Rollout checklist
+
+Use this checklist when switching an environment from Python invocation to Go invocation:
+
+1. Run `go test ./...`.
+2. Build the Go binary (`go build -o ./bin/audita ./cmd/audita`).
+3. Run one representative fixture through `audita process`.
+4. Verify `--report-json` output is written and machine-readable.
+5. Verify failure runs print diagnostics path to stderr and retain diagnostics with `error.log`.
+6. Update orchestrator configuration to call the Go binary and pass `--output` and `--report-json`.
--- a/docs/rewrite-notes.md
+++ b/docs/rewrite-notes.md
@@ -29,9 +29,19 @@ The Go rewrite is complete when both of the following are true:
   - Strict stdout/stderr discipline for subprocess callers.
   - No hidden service dependency; the CLI remains the primary integration surface.

+## Final status (Phase 19 complete)
+
+The Go rewrite definition of done is met, with documented parity caveats tracked in `docs/python-parity.md`.
+
+- Phase 19 is complete.
+- Go Audita is the active implementation.
+- The default full module pipeline is implemented (`glossary,homophones,glossary,spoken_word,grammar`).
+- Parity fixtures and operational hardening coverage exist in normal `go test ./...`.
+- Python is retained as a legacy/reference implementation and is not the primary operational path.
+
 ## Current implementation status

-The Go rewrite now has feature-complete runtime behavior for the module pipeline plus Phase 18 operational hardening coverage.
+The Go rewrite is complete and feature-complete for the intended CLI runtime architecture.

 Implemented:
 - CLI command surface for `audita process`.
@@ -97,8 +107,11 @@ Implemented:
  - validation override behavior when validation fields are set.
 - Generic JSON diagnostics primitives for LLM interactions (request metadata, request payload, response payload, optional error payload) with secret redaction.

-Not yet implemented:
- Phase 19 rollout and Python retirement/documentation transition work.
+Phase 19 documentation and rollout items are implemented:
+- Go-first README/build/test/usage guidance.
+- subprocess/orchestrator operational guidance.
+- Python-to-Go migration guidance.
+- explicit legacy/reference Python status documentation.

 ## Completed phases

@@ -234,10 +247,6 @@ Not implemented in Phase 8 (by design):
 - Prompt/response diagnostics runtime wiring.
 - End-to-end transcript polishing.

-## Remaining work plan
-
-Next recommended phase: **Phase 19 (documentation, rollout, and Python retirement)**.
-
 ## Phase 9: Structured LLM client and scheduler infrastructure

 ### Status
@@ -656,8 +665,8 @@ Implemented:
  - permit release on error/cancellation.
 - Added focused subprocess-caller operational guidance in `docs/subprocess-operations.md`.

-Intentionally deferred:
- Phase 19 documentation/rollout/Python-retirement transitions.
+Historical note:
+- Phase 19 documentation/rollout/Python-retirement transitions were deferred at the end of Phase 18 and completed in Phase 19.

 ## Phase 19: Documentation, rollout, and Python retirement

@@ -689,9 +698,23 @@ The repository clearly presents the Go implementation as the active Audita imple
 - Users can build, test, and run the Go implementation from docs alone.
 - `go test ./...` passes.

+### Phase 19 completion status
+
+Completed.
+
+Implemented:
+- README is Go-first and documents active full-pipeline runtime behavior.
+- Build/install/test usage guidance is Go-primary and operationally oriented.
+- Subprocess/orchestrator behavior is documented in `docs/subprocess-operations.md`.
+- Migration guidance from Python to Go is documented in `docs/migration-from-python.md`.
+- Python parity notes and open parity gaps remain documented in `docs/python-parity.md`.
+- Python implementation status is explicitly legacy/reference in primary docs and `python/README.md`.
+- Documentation no longer routes normal operations to Python.
+- Normal `go test ./...` remains independent of real LLM credentials and Python dependencies.
+
 ## Cross-phase compatibility requirements

-These constraints apply to every remaining phase:
+These constraints were used across implementation phases and remain useful maintenance checks:

 - Preserve subprocess-safe stdout/stderr behavior.
 - Preserve CLI-over-env precedence.
--- a/python/README.md
+++ b/python/README.md
@@ -4,7 +4,8 @@
 This directory contains the frozen Python implementation of Audita.
 It is retained as the behavioral reference for the Go port.

-The canonical implementation is moving to the Go code at the repository root.
+The canonical implementation is now the Go code at the repository root.
+This Python implementation is legacy/reference and is no longer the primary operational path.
 Do not add new features here except to fix reference-test issues needed for port validation.
 ```