audita/docs/rewrite-notes.md

# Audita Go Rewrite Notes

## Definition of done for the Go rewrite

The Go rewrite is complete when both of the following are true:

1. Feature parity with the initial Python implementation:
   - `audita process` performs end-to-end transcript polishing, not only deterministic preprocessing.
   - The default module sequence is implemented and active in the runtime path:
     - `glossary`
     - `homophones`
     - `glossary`
     - `spoken_word`
     - `grammar`
   - Repeated module instances are resolved deterministically, for example `glossary_1` and `glossary_2`.
   - Real LLM-backed proposal generation is implemented through an OpenAI-compatible structured-output client.
   - Deterministic validators and LLM-backed validators are implemented and enforced in the runtime path.
   - Proposal application preserves the safety-first semantics of the Python implementation.
   - Structured run reports and diagnostics are sufficient for debugging successful runs, skipped corrections, and failures.
   - The Go CLI can replace the Python CLI in the surrounding orchestration pipeline without downstream contract surprises.

2. Adherence to the intended application architecture in `docs/architecture.md`:
   - Sequential module pipeline over a mutable working transcript.
   - Bounded intra-module concurrency only.
   - Provider-neutral LLM abstraction.
   - Separate proposal and validation LLM settings.
   - Prompt/response diagnostics for LLM stages.
   - Module-level run reports with applied and skipped changes.
   - Strict stdout/stderr discipline for subprocess callers.
   - No hidden service dependency; the CLI remains the primary integration surface.

## Current implementation status

The Go rewrite is currently in a deterministic foundation stage.

Implemented:
- CLI command surface for `audita process`.
- Config/env/flag loading and validation.
- Transcript/glossary schema parsing and validation.
- Deterministic normalization with summary stats.
- Deterministic chunking with summary stats.
- Per-run diagnostics directory plus source/normalization/chunking artifacts.
- Redacted invocation/effective-config diagnostics metadata artifacts.
- Process report output through `--report-json` and run-dir `report.json`.
- Report-level diagnostics artifact references.
- Framework foundation packages for contracts and proposal preview/apply semantics.
- Production runner orchestration over a mutable working transcript.
- Module-level report structures and run-level module summaries.
- CLI runner integration point via injectable module factory/registry (used by deterministic tests).
- Runtime validator models and deterministic validator implementations.
- Validator cardinality enforcement (missing/duplicate/unknown proposal index errors).
- Deterministic validator-chain execution in the production runner.
- Module reports including validator decisions and validator rejections.
- LLM-backed validator request/response models and prompt builders.
- LLM validator batching by validation prompt-token budget.
- LLM validator runtime integration through structured LLM client abstraction and scheduler hooks.
- LLM validator prompt/response diagnostics artifact wiring with secret redaction.
- Shared LLM proposal-generation helper with structured correction-set parsing.
- Deterministic proposal-index assignment for shared proposal generation.
- Proposal-generation prompt/response diagnostics artifact wiring with secret redaction.
- Production module-registry scaffolding with known key recognition and explicit unsupported/unimplemented errors.
- Production grammar module package with Python-aligned prompt intent and guardrails.
- Explicit `--modules grammar` runtime path through runner, shared proposal generation, validators, application, reporting, and diagnostics.
- Broad deterministic and CLI/subprocess test coverage for implemented phases through `go test ./...`.
- Internal typed structured LLM contract (`StructuredLLMClient.CompleteStructured(ctx, req, out)`).
- `internal/framework/llm` instructor-go-backed adapter with:
  - configurable base URL/model/retries/mode/timeout
  - optional API key support for local-compatible endpoints
  - API-key redaction in returned errors
  - typed structured decode into caller-provided outputs.
- LLM scheduler/semaphore infrastructure for bounded concurrency with context-aware acquisition and reliable release.
- LLM effective-config resolution helpers:
  - primary config resolution
  - validation config inheritance from primary when validation fields are unset
  - validation override behavior when validation fields are set.
- Generic JSON diagnostics primitives for LLM interactions (request metadata, request payload, response payload, optional error payload) with secret redaction.

Not yet implemented in runtime pipeline:
- Real correction modules for `glossary`, `homophones`, and `spoken_word`.
- Domain proposal prompts for remaining real modules.
- End-to-end transcript polishing behavior.

## Completed phases

### Phase 1: Go CLI skeleton

Completed.

Implemented:
- Go module and `cmd/audita` entrypoint.
- `audita process` command surface.
- Core flags and config wiring.
- Subprocess-safe command behavior foundation.

### Phase 2: Schemas and file I/O

Completed.

Implemented:
- Transcript parsing for bare arrays and `{ "segments": [...] }` input.
- Glossary YAML parsing.
- Transcript and glossary validation.
- Canonical transcript output serialization.

### Phase 3: Deterministic normalization

Completed.

Implemented:
- Chronological sorting.
- Same-speaker segment merging.
- Gap-sensitive join behavior.
- Duration and token-budget merge constraints.
- Sequential normalized segment IDs.
- Normalization summary stats.

### Phase 4: Chunking and token estimation

Completed.

Implemented:
- Deterministic heuristic token estimation.
- Contiguous transcript sectioning.
- Min/max section token behavior.
- Optional target section handling.
- Chunking summaries and diagnostics artifacts.

### Phase 5: Proposal model and application semantics

Completed.

Implemented:
- Correction proposal and enriched proposal models.
- Replacement policies:
  - `require_unique`
  - `replace_all`
- Safe preview logic.
- Deterministic proposal application.
- Applied/skipped change records.
- Stable skip reasons.

### Phase 6: Reports and diagnostics

Completed for the current deterministic runtime scope.

Implemented:
- Per-run diagnostics directory creation.
- Source transcript artifacts.
- Parsed source transcript artifacts.
- Normalized transcript artifact.
- Normalization summary artifact.
- Chunking summary artifact.
- Redacted invocation metadata artifact.
- Redacted effective-config artifact.
- Run-dir `report.json`.
- Optional external `--report-json`.
- Failure `error.log`.
- Report-level diagnostics artifact references.
- Retention decision model with future skipped-correction hook.

Intentionally deferred:
- Module prompt/response diagnostics artifacts are not produced yet because module execution and LLM calls are not implemented in the runtime path.

### Phase 7: Pipeline runner with deterministic test modules

Completed.

Implemented:
- `internal/framework/runner` production package with sequential module orchestration.
- Deterministic module run-spec resolution and repeated instance naming (`glossary_1`, `glossary_2`, etc.).
- Mutable working transcript handoff across module instances.
- Proposal application through `internal/framework/proposals`.
- Per-module applied/skipped change capture and module status/timing metadata.
- Partial-progress return on module failure, with pipeline stop on first failure.
- Process report support for module-level results and run-level module summaries.
- CLI runtime integration point via injectable module factory/registry, exercised by deterministic fake-module tests.

Not implemented in Phase 7 (by design):
- Real `glossary`, `homophones`, `spoken_word`, `grammar` production modules.
- Validators (Phase 8).
- Structured LLM calls or scheduler behavior.
- Prompt/response diagnostics.
- End-to-end transcript polishing.

Current runtime behavior note:
- Default user-facing CLI behavior remains deterministic normalization/chunking output unless test-only module injection is used during tests.

## Phase 8: Runtime validator framework and deterministic validators
Completed.

Implemented:
- Runtime validator request/result models in `internal/framework/validators`.
- Deterministic validator reason codes for stable reporting.
- Validator cardinality enforcement:
  - one decision per candidate proposal index
  - missing indexes are errors
  - duplicate indexes are errors
  - unknown indexes are errors
- Deterministic validators:
  - confidence threshold
  - original-text presence against working transcript
  - non-empty correction
  - identical/no-effect rejection
  - conservative protected glossary-term guard
- Ordered validator-chain execution in the production runner.
- Runner behavior where only validator-approved proposals proceed to proposal application.
- Module-level reporting of validator decisions and validator rejections, distinct from application-level skips.
- Deterministic fake-module tests covering approvals, rejections, validator order/filtering, and cardinality failure pipeline-stop behavior.

Not implemented in Phase 8 (by design):
- LLM-backed validators (Phase 10).
- Structured LLM runtime wiring (Phase 9 follow-up).
- Real correction modules.
- Prompt/response diagnostics runtime wiring.
- End-to-end transcript polishing.

## Remaining work plan

Next recommended phase: **Phase 13 (glossary module and protected-term behavior)**.

## Phase 9: Structured LLM client and scheduler infrastructure

### Status

Completed for Phase 9 infrastructure scope.

Implemented in this phase so far:
- Added internal structured LLM contract support for caller-provided typed outputs.
- Added `internal/framework/llm` adapter backed by `github.com/jxnl/instructor-go`.
- Confirmed OpenAI-compatible base URL support through the adapter path.
- Added adapter unit tests for model/base URL handling, retries, context cancellation, optional API key behavior, and error redaction.

Explicitly deferred from Phase 9 into later phases:
- Runtime wiring in runner/module infrastructure (without introducing real modules yet).
- Wiring prompt/response diagnostics primitives into future module/validator call sites.
- Wiring effective primary/validation LLM config resolution into runtime LLM call sites.

### Purpose

Implement the provider-neutral LLM infrastructure needed by both proposal generation and LLM-backed validators, without yet implementing real modules.

### Scope completed in this phase

Implemented:
- Provider-neutral internal structured LLM contract with caller-provided typed output decoding.
- OpenAI-compatible structured-output adapter (`instructor-go`) with:
  - configurable base URL and model
  - optional API key behavior
  - retry budget
  - timeout-aware HTTP client handling
  - context cancellation propagation
  - structured response decoding into caller-provided typed targets
  - redacted error surfaces.
- Scheduler/semaphore infrastructure for bounded backend concurrency with context-aware acquisition and reliable permit release.
- Effective LLM config resolution helpers for:
  - primary LLM settings
  - validation inheritance from primary when unset
  - validation overrides when set.
- Generic JSON diagnostics primitives for request metadata, request payload, response payload, and optional error payload, with secret redaction.
- Unit tests covering adapter behavior, scheduler behavior, effective config resolution, and diagnostics redaction/JSON validity.

Do not implement:
- Real correction modules.
- LLM-backed validators.
- Prompt text for domain modules.
- End-to-end transcript polishing.

### Expected behavior at end of phase

At the end of Phase 9, the codebase had tested LLM infrastructure primitives, while default CLI runtime behavior remained deterministic preprocessing/reporting because real modules were not implemented yet.

### Definition of done status

Met:
- Structured LLM client infrastructure is implemented and tested.
- OpenAI-compatible structured-output client exists and is tested.
- Scheduler enforces configured concurrency and is tested.
- Primary and validation effective config resolution exists and is tested.
- Prompt/response diagnostics primitives exist and are tested.
- API keys are redacted in LLM adapter errors and diagnostics artifacts; config/report diagnostics redaction remains in place.
- No real module behavior was introduced.
- `go test ./...` passes.

## Phase 10: LLM-backed validators

Completed.

Implemented:
- LLM-backed validator request/response models in `internal/framework/validators`.
- Prompt builders for:
  - spoken-form plausibility
  - meaning reversal detection
  - editorial review
  - grammar review
  - spoken-word review
- Deterministic batching by `validation_max_prompt_tokens` with stable ordering and no drop/dup behavior.
- LLM validator execution through the internal structured client abstraction (no direct provider calls in validator code).
- Scheduler/concurrency hooks for LLM validator calls.
- Prompt/response diagnostics artifact writing for LLM validator batches using Phase 9 diagnostics primitives.
- Secret redaction in validator LLM diagnostics artifacts.
- Strict structured-response safety and cardinality checks (missing/duplicate/unknown indexes fail closed).
- Runner/report integration so LLM validator decisions and rejections appear in module reports.
- Fake-module and fake-client tests for approval/rejection, malformed output, cardinality errors, batching, scheduler usage, and diagnostics redaction.

Not implemented in Phase 10 (by design):
- Real correction modules (`glossary`, `homophones`, `spoken_word`, `grammar`).
- Real module implementation and full runtime wiring (Phase 12+).
- Domain proposal prompts.
- Default CLI end-to-end transcript polishing behavior.

## Phase 11: Shared LLM proposal generation framework and module registry

Completed.

Implemented:
- Shared proposal-generation package `internal/framework/proposal_generation`.
- Reusable request model for proposal generation including:
  - module key/instance
  - replacement policy
  - working transcript context
  - optional section metadata
  - glossary/config context
  - diagnostics context
  - injected structured LLM client/scheduler dependencies.
- Structured correction-set response model and parsing into existing proposal models:
  - `proposals.CorrectionProposal`
  - `proposals.EnrichedCorrectionProposal`.
- Deterministic proposal-index assignment via caller-provided start index.
- Proposal-generation diagnostics artifact writing using generic LLM diagnostics primitives with secret redaction.
- Scheduler-aware proposal generation through the internal LLM scheduler interface.
- Production module-registry scaffolding in `internal/framework/modules` with:
  - known module-key recognition for `glossary`, `homophones`, `spoken_word`, `grammar`
  - constructor registration and dependency-injection path
  - explicit unsupported and recognized-but-unimplemented module errors.
- Runner/CLI injection-path tests showing shared proposal generation can flow through runner validation/application semantics using fake modules/clients.

Not implemented in Phase 11 (by design):
- Real production `glossary`, `homophones`, and `spoken_word` modules.
- Domain proposal prompts for production modules.
- Default CLI end-to-end transcript polishing behavior.

## Phase 12: Grammar module

Completed.

Implemented:
- Production grammar module package in `internal/modules/grammar`.
- Grammar prompt builder aligned to Python intent and constrained to punctuation/capitalization/spacing/article cleanup.
- Grammar proposal generation through shared `internal/framework/proposal_generation` using `contracts.StructuredLLMClient`.
- Scheduler-aware grammar proposal generation through existing scheduler hooks.
- Grammar replacement policy `require_unique` (matching Python behavior).
- Grammar validator chain using existing deterministic and LLM-backed validator infrastructure.
- Grammar confidence threshold enforcement through existing config + confidence-threshold validator behavior.
- Explicit runtime support for `--modules grammar` through normalization, chunking, runner, proposal generation, validation, application, and reporting.
- Prompt/response diagnostics artifacts for grammar proposal + validator interactions with secret redaction.
- Module-level reports for grammar including validator decisions/rejections, applied changes, and skipped changes.
- CLI/runtime fake-client tests for approved proposals, validator rejection, application skips, diagnostics, failure/error.log behavior, and report outputs (`--report-json` and run-dir `report.json`).

Not implemented in Phase 12 (by design):
- Production `glossary`, `homophones`, and `spoken_word` modules.
- Full default module sequence execution as a feature-complete claim.

## Phase 13: Glossary module and protected-term behavior

### Purpose

Implement the glossary correction module and the glossary-derived protection behavior needed by downstream modules.

### Scope

Implement:
- `glossary` module package.
- Glossary prompt builder ported from Python.
- Glossary structured response model.
- Glossary replacement policy.
- Glossary confidence threshold handling.
- Glossary validator chain.
- Protected-term extraction from parsed glossary.
- Protected-term validator behavior used by other modules where applicable.
- Prompt/response diagnostics.
- CLI support for `--modules glossary`.
- Fake LLM tests.
- Tests for repeated glossary stages using `glossary,glossary`.

Do not implement:
- Homophones module.
- Spoken-word module.
- Default full pipeline parity claim.

### Expected behavior at end of phase

Running `audita process ... --modules glossary` should perform real glossary-supported corrections. Repeated glossary stages should work and be reported as separate module instances.

### Definition of done

- Glossary module runs in the production runner.
- Glossary terms and aliases are used in prompts and validators.
- Protected-term behavior is implemented and tested.
- Repeated glossary module instances are reported correctly.
- Applied/skipped glossary changes appear in reports.
- Prompt/response diagnostics are written.
- `go test ./...` passes without requiring external LLM credentials.

## Phase 14: Homophones module

### Purpose

Implement conservative homophone and mistranscription correction behavior.

### Scope

Implement:
- `homophones` module package.
- Homophones prompt builder ported from Python.
- Homophones structured response model.
- Homophones replacement policy.
- Homophones confidence threshold handling.
- Homophones validator chain.
- Prompt/response diagnostics.
- CLI support for `--modules homophones`.
- Fake LLM tests.
- Tests for interaction with glossary/protected terms where relevant.

Do not implement:
- Spoken-word module.
- Default full pipeline parity claim unless spoken-word is already complete.

### Expected behavior at end of phase

Running `audita process ... --modules homophones` should perform real conservative homophone/mistranscription corrections using the configured LLM endpoint.

### Definition of done

- Homophones module runs in the production runner.
- Homophones proposals are generated through structured LLM calls.
- Validator chain is enforced.
- Protected-term behavior is respected where applicable.
- Applied/skipped homophone changes appear in reports.
- Prompt/response diagnostics are written.
- `go test ./...` passes without requiring external LLM credentials.

## Phase 15: Spoken-word module

### Purpose

Implement conservative dysfluency cleanup while preserving substantive meaning.

### Scope

Implement:
- `spoken_word` module package.
- Spoken-word prompt builder ported from Python.
- Spoken-word structured response model.
- Spoken-word replacement policy.
- Spoken-word confidence threshold handling.
- Spoken-word validator chain.
- Strong semantic guardrails using LLM-backed validators.
- Prompt/response diagnostics.
- CLI support for `--modules spoken_word`.
- Fake LLM tests.
- Tests for rejection of meaning-changing cleanup.

Do not implement:
- Prompt redesign beyond faithful porting.
- New stylistic rewriting behavior not present in the Python implementation.

### Expected behavior at end of phase

Running `audita process ... --modules spoken_word` should perform real conservative dysfluency cleanup, with guardrails against semantic changes.

### Definition of done

- Spoken-word module runs in the production runner.
- Spoken-word proposals are generated through structured LLM calls.
- Semantic validators reject meaning-changing proposals.
- Applied/skipped spoken-word changes appear in reports.
- Prompt/response diagnostics are written.
- `go test ./...` passes without requiring external LLM credentials.

## Phase 16: Default full pipeline integration

### Purpose

Enable and harden the full default module sequence in the Go runtime path.

### Scope

Implement:
- Default runtime sequence:
  - `glossary`
  - `homophones`
  - `glossary`
  - `spoken_word`
  - `grammar`
- End-to-end execution through all real modules.
- Accurate resolved module instance names.
- Module-level report aggregation.
- Full run-level applied/skipped summaries.
- Skip-aware retention behavior using actual skipped correction data.
- Failure behavior with partial module progress.
- PipelineRunError or equivalent partial-progress error type if not already implemented.
- Diagnostics for each module instance.
- Tests for successful full pipeline using fake LLM.
- Tests for mid-pipeline failure preserving partial report and diagnostics.

Do not implement:
- Python archive removal.
- Side-by-side rollout tooling beyond what is needed for tests.

### Expected behavior at end of phase

Running `audita process transcript.json --glossary glossary.yaml --output corrected.json` should execute the full Go module pipeline and produce a corrected transcript.

### Definition of done

- Default module sequence runs end-to-end.
- All real modules participate in runtime path.
- Reports include all module instances.
- Applied/skipped changes are aggregated at run level.
- Failed runs retain useful partial reports and diagnostics.
- `auto` retention keeps successful runs with skipped corrections.
- CLI stdout/stderr behavior remains subprocess-safe.
- `go test ./...` passes without requiring external LLM credentials.

## Phase 17: Python parity fixture suite

### Purpose

Establish confidence that the Go implementation matches the behavior and safety posture of the initial Python implementation.

### Scope

Implement:
- Parity fixtures based on representative Python-era inputs and expected behaviors.
- Fake LLM response fixtures where exact deterministic behavior is required.
- Golden tests for:
  - transcript schema handling
  - glossary schema handling
  - normalization
  - chunking
  - proposal application
  - validator behavior
  - module reports
  - diagnostics artifacts
  - default pipeline shape
- Comparison tests or scripts that can run Python and Go side by side where practical.
- Documentation of intentional differences between Python and Go.

Do not require:
- Real LLM credentials for normal automated tests.
- Exact nondeterministic natural-language output equivalence across Python and Go.

### Expected behavior at end of phase

The repository has a durable test suite demonstrating that the Go implementation preserves the functional contract of the Python implementation.

### Definition of done

- Representative parity fixtures exist.
- Golden tests cover deterministic behavior.
- Fake LLM tests cover full pipeline behavior.
- Intentional differences from Python are documented.
- Unintentional compatibility breaks are fixed.
- `go test ./...` passes.

## Phase 18: Operational hardening and subprocess integration

### Purpose

Harden the Go binary for use as the production Audita implementation in the surrounding application suite.

### Scope

Implement:
- Additional subprocess tests for large transcripts.
- Timeout/cancellation tests.
- Failure-mode tests for unreadable inputs, unwritable outputs, malformed LLM responses, and backend failures.
- Clear stderr summaries pointing to diagnostics.
- Review of all output paths and file-close behavior.
- Review of all secret redaction paths.
- Review of LLM retry and timeout behavior.
- Documentation for production use from orchestrators such as Narratio.

### Expected behavior at end of phase

The Go binary should be safe to call from other Go applications and should not reproduce the Python subprocess/stdio integration problems.

### Definition of done

- Subprocess tests cover success, failure, large inputs, and cancellation.
- stdout contains machine output only.
- stderr contains human-readable logs/errors only.
- API keys do not appear in reports, diagnostics, logs, or tests.
- Failed runs always preserve diagnostics.
- Operational docs are accurate.
- `go test ./...` passes.

## Phase 19: Documentation, rollout, and Python retirement

### Purpose

Make the Go implementation the documented active implementation and preserve the Python implementation only as historical reference if desired.

### Scope

Implement:
- README updates.
- Architecture updates.
- Rewrite notes updates.
- Installation/build instructions for Go binary.
- Migration notes from Python to Go.
- Documentation of any intentionally changed behavior.
- Removal or archival of Python-specific operational instructions from the primary path.
- Clear statement that the Go implementation is now feature-complete.

### Expected behavior at end of phase

The repository clearly presents the Go implementation as the active Audita implementation.

### Definition of done

- Documentation no longer describes Go as only a deterministic foundation.
- Documentation accurately describes full transcript polishing behavior.
- Python implementation is marked archived/prototype/reference, or removed if that is the chosen repository policy.
- Users can build, test, and run the Go implementation from docs alone.
- `go test ./...` passes.

## Cross-phase compatibility requirements

These constraints apply to every remaining phase:

- Preserve subprocess-safe stdout/stderr behavior.
- Preserve CLI-over-env precedence.
- Preserve existing accepted transcript input forms.
- Preserve glossary YAML compatibility unless a deliberate migration is documented.
- Preserve deterministic safety-first proposal application semantics.
- Preserve run-dir diagnostics and report writing.
- Preserve API key redaction.
- Do not require real LLM credentials for normal `go test ./...`.
- Do not claim a feature is implemented until it is in the runtime path.
- Keep default behavior honest in docs and CLI help.
- Prefer fake LLMs and golden fixtures for automated tests.
- Keep real LLM smoke tests opt-in.

## Guidance for Codex-style prompts

When requesting implementation work, use one phase at a time.

Good prompt shape:
- Name the exact phase.
- State what is in scope.
- State what is explicitly out of scope.
- Require tests.
- Require `go test ./...`.
- Require documentation updates only when the phase changes user-visible or architectural status.
- Require that no later-phase features be implemented opportunistically.

Avoid broad prompts such as:
- “finish the rewrite”
- “make Audita work”
- “port the Python app”
- “implement all modules”

Those prompts blur phase boundaries and make review difficult.

## Suggested review checklist for every phase

Before accepting a phase implementation, verify:

- Does the implementation match the phase scope?
- Did it avoid implementing unrelated later-phase behavior?
- Does `go test ./...` pass?
- Are stdout and stderr still clean for subprocess callers?
- Are API keys redacted everywhere?
- Are reports machine-readable?
- Are diagnostics sufficient for debugging?
- Are fake LLM tests used instead of requiring real credentials?
- Are new public behaviors documented?
- Does the code follow the architecture in `docs/architecture.md`?
- Does the implementation move Audita closer to Python feature parity?