audita/docs/rewrite-notes.md

Audita Go Rewrite Notes

Definition of done for the Go rewrite

The Go rewrite is complete when both of the following are true:

1. Feature parity with the initial Python implementation:

   • audita process performs end-to-end transcript polishing, not only deterministic preprocessing.
   • The default module sequence is implemented and active in the runtime path:
     • glossary
     • homophones
     • glossary
     • spoken_word
     • grammar
   • Repeated module instances are resolved deterministically, for example glossary_1 and glossary_2.
   • Real LLM-backed proposal generation is implemented through an OpenAI-compatible structured-output client.
   • Deterministic validators and LLM-backed validators are implemented and enforced in the runtime path.
   • Proposal application preserves the safety-first semantics of the Python implementation.
   • Structured run reports and diagnostics are sufficient for debugging successful runs, skipped corrections, and failures.
   • The Go CLI can replace the Python CLI in the surrounding orchestration pipeline without downstream contract surprises.

2. Adherence to the intended application architecture in docs/architecture.md:

   • Sequential module pipeline over a mutable working transcript.
   • Bounded intra-module concurrency only.
   • Provider-neutral LLM abstraction.
   • Separate proposal and validation LLM settings.
   • Prompt/response diagnostics for LLM stages.
   • Module-level run reports with applied and skipped changes.
   • Strict stdout/stderr discipline for subprocess callers.
   • No hidden service dependency; the CLI remains the primary integration surface.

Current implementation status

The Go rewrite is currently in a deterministic foundation stage.

Implemented:

• CLI command surface for audita process.
• Config/env/flag loading and validation.
• Transcript/glossary schema parsing and validation.
• Deterministic normalization with summary stats.
• Deterministic chunking with summary stats.
• Per-run diagnostics directory plus source/normalization/chunking artifacts.
• Redacted invocation/effective-config diagnostics metadata artifacts.
• Process report output through --report-json and run-dir report.json.
• Report-level diagnostics artifact references.
• Framework foundation packages for contracts and proposal preview/apply semantics.
• Production runner orchestration over a mutable working transcript.
• Module-level report structures and run-level module summaries.
• CLI runner integration point via injectable module factory/registry (used by deterministic tests).
• Runtime validator models and deterministic validator implementations.
• Validator cardinality enforcement (missing/duplicate/unknown proposal index errors).
• Deterministic validator-chain execution in the production runner.
• Module reports including validator decisions and validator rejections.
• LLM-backed validator request/response models and prompt builders.
• LLM validator batching by validation prompt-token budget.
• LLM validator runtime integration through structured LLM client abstraction and scheduler hooks.
• LLM validator prompt/response diagnostics artifact wiring with secret redaction.
• Broad deterministic and CLI/subprocess test coverage for implemented phases through go test ./....
• Internal typed structured LLM contract (StructuredLLMClient.CompleteStructured(ctx, req, out)).
• internal/framework/llm instructor-go-backed adapter with:
  • configurable base URL/model/retries/mode/timeout
  • optional API key support for local-compatible endpoints
  • API-key redaction in returned errors
  • typed structured decode into caller-provided outputs.
• LLM scheduler/semaphore infrastructure for bounded concurrency with context-aware acquisition and reliable release.
• LLM effective-config resolution helpers:
  • primary config resolution
  • validation config inheritance from primary when validation fields are unset
  • validation override behavior when validation fields are set.
• Generic JSON diagnostics primitives for LLM interactions (request metadata, request payload, response payload, optional error payload) with secret redaction.

Not yet implemented in runtime pipeline:

• Real correction modules.
• Shared module proposal generation and module registry wiring.
• End-to-end transcript polishing behavior.

Completed phases

Phase 1: Go CLI skeleton

Completed.

Implemented:

• Go module and cmd/audita entrypoint.
• audita process command surface.
• Core flags and config wiring.
• Subprocess-safe command behavior foundation.

Phase 2: Schemas and file I/O

Completed.

Implemented:

• Transcript parsing for bare arrays and { "segments": [...] } input.
• Glossary YAML parsing.
• Transcript and glossary validation.
• Canonical transcript output serialization.

Phase 3: Deterministic normalization

Completed.

Implemented:

• Chronological sorting.
• Same-speaker segment merging.
• Gap-sensitive join behavior.
• Duration and token-budget merge constraints.
• Sequential normalized segment IDs.
• Normalization summary stats.

Phase 4: Chunking and token estimation

Completed.

Implemented:

• Deterministic heuristic token estimation.
• Contiguous transcript sectioning.
• Min/max section token behavior.
• Optional target section handling.
• Chunking summaries and diagnostics artifacts.

Phase 5: Proposal model and application semantics

Completed.

Implemented:

• Correction proposal and enriched proposal models.
• Replacement policies:
  • require_unique
  • replace_all
• Safe preview logic.
• Deterministic proposal application.
• Applied/skipped change records.
• Stable skip reasons.

Phase 6: Reports and diagnostics

Completed for the current deterministic runtime scope.

Implemented:

• Per-run diagnostics directory creation.
• Source transcript artifacts.
• Parsed source transcript artifacts.
• Normalized transcript artifact.
• Normalization summary artifact.
• Chunking summary artifact.
• Redacted invocation metadata artifact.
• Redacted effective-config artifact.
• Run-dir report.json.
• Optional external --report-json.
• Failure error.log.
• Report-level diagnostics artifact references.
• Retention decision model with future skipped-correction hook.

Intentionally deferred:

• Module prompt/response diagnostics artifacts are not produced yet because module execution and LLM calls are not implemented in the runtime path.

Phase 7: Pipeline runner with deterministic test modules

Completed.

Implemented:

• internal/framework/runner production package with sequential module orchestration.
• Deterministic module run-spec resolution and repeated instance naming (glossary_1, glossary_2, etc.).
• Mutable working transcript handoff across module instances.
• Proposal application through internal/framework/proposals.
• Per-module applied/skipped change capture and module status/timing metadata.
• Partial-progress return on module failure, with pipeline stop on first failure.
• Process report support for module-level results and run-level module summaries.
• CLI runtime integration point via injectable module factory/registry, exercised by deterministic fake-module tests.

Not implemented in Phase 7 (by design):

• Real glossary, homophones, spoken_word, grammar production modules.
• Validators (Phase 8).
• Structured LLM calls or scheduler behavior.
• Prompt/response diagnostics.
• End-to-end transcript polishing.

Current runtime behavior note:

• Default user-facing CLI behavior remains deterministic normalization/chunking output unless test-only module injection is used during tests.

Phase 8: Runtime validator framework and deterministic validators

Completed.

Implemented:

• Runtime validator request/result models in internal/framework/validators.
• Deterministic validator reason codes for stable reporting.
• Validator cardinality enforcement:
  • one decision per candidate proposal index
  • missing indexes are errors
  • duplicate indexes are errors
  • unknown indexes are errors
• Deterministic validators:
  • confidence threshold
  • original-text presence against working transcript
  • non-empty correction
  • identical/no-effect rejection
  • conservative protected glossary-term guard
• Ordered validator-chain execution in the production runner.
• Runner behavior where only validator-approved proposals proceed to proposal application.
• Module-level reporting of validator decisions and validator rejections, distinct from application-level skips.
• Deterministic fake-module tests covering approvals, rejections, validator order/filtering, and cardinality failure pipeline-stop behavior.

Not implemented in Phase 8 (by design):

• LLM-backed validators (Phase 10).
• Structured LLM runtime wiring (Phase 9 follow-up).
• Real correction modules.
• Prompt/response diagnostics runtime wiring.
• End-to-end transcript polishing.

Remaining work plan

Next recommended phase: Phase 11 (shared LLM proposal generation framework and module registry).

Phase 9: Structured LLM client and scheduler infrastructure

Status

Completed for Phase 9 infrastructure scope.

Implemented in this phase so far:

• Added internal structured LLM contract support for caller-provided typed outputs.
• Added internal/framework/llm adapter backed by github.com/jxnl/instructor-go.
• Confirmed OpenAI-compatible base URL support through the adapter path.
• Added adapter unit tests for model/base URL handling, retries, context cancellation, optional API key behavior, and error redaction.

Explicitly deferred from Phase 9 into later phases:

• Runtime wiring in runner/module infrastructure (without introducing real modules yet).
• Wiring prompt/response diagnostics primitives into future module/validator call sites.
• Wiring effective primary/validation LLM config resolution into runtime LLM call sites.

Purpose

Implement the provider-neutral LLM infrastructure needed by both proposal generation and LLM-backed validators, without yet implementing real modules.

Scope completed in this phase

Implemented:

• Provider-neutral internal structured LLM contract with caller-provided typed output decoding.
• OpenAI-compatible structured-output adapter (instructor-go) with:
  • configurable base URL and model
  • optional API key behavior
  • retry budget
  • timeout-aware HTTP client handling
  • context cancellation propagation
  • structured response decoding into caller-provided typed targets
  • redacted error surfaces.
• Scheduler/semaphore infrastructure for bounded backend concurrency with context-aware acquisition and reliable permit release.
• Effective LLM config resolution helpers for:
  • primary LLM settings
  • validation inheritance from primary when unset
  • validation overrides when set.
• Generic JSON diagnostics primitives for request metadata, request payload, response payload, and optional error payload, with secret redaction.
• Unit tests covering adapter behavior, scheduler behavior, effective config resolution, and diagnostics redaction/JSON validity.

Do not implement:

• Real correction modules.
• LLM-backed validators.
• Prompt text for domain modules.
• End-to-end transcript polishing.

Expected behavior at end of phase

The codebase has tested Phase 9 LLM infrastructure, but default CLI runtime behavior remains deterministic preprocessing/reporting because real modules and LLM-backed validators are not implemented.

Definition of done status

Met:

• Structured LLM client infrastructure is implemented and tested.
• OpenAI-compatible structured-output client exists and is tested.
• Scheduler enforces configured concurrency and is tested.
• Primary and validation effective config resolution exists and is tested.
• Prompt/response diagnostics primitives exist and are tested.
• API keys are redacted in LLM adapter errors and diagnostics artifacts; config/report diagnostics redaction remains in place.
• No real module behavior was introduced.
• go test ./... passes.

Phase 10: LLM-backed validators

Completed.

Implemented:

• LLM-backed validator request/response models in internal/framework/validators.
• Prompt builders for:
  • spoken-form plausibility
  • meaning reversal detection
  • editorial review
  • grammar review
  • spoken-word review
• Deterministic batching by validation_max_prompt_tokens with stable ordering and no drop/dup behavior.
• LLM validator execution through the internal structured client abstraction (no direct provider calls in validator code).
• Scheduler/concurrency hooks for LLM validator calls.
• Prompt/response diagnostics artifact writing for LLM validator batches using Phase 9 diagnostics primitives.
• Secret redaction in validator LLM diagnostics artifacts.
• Strict structured-response safety and cardinality checks (missing/duplicate/unknown indexes fail closed).
• Runner/report integration so LLM validator decisions and rejections appear in module reports.
• Fake-module and fake-client tests for approval/rejection, malformed output, cardinality errors, batching, scheduler usage, and diagnostics redaction.

Not implemented in Phase 10 (by design):

• Real correction modules (glossary, homophones, spoken_word, grammar).
• Shared module proposal generation and module registry work (Phase 11).
• Domain proposal prompts.
• Default CLI end-to-end transcript polishing behavior.

Phase 11: Shared LLM proposal generation framework and module registry

Purpose

Create the reusable proposal-generation layer used by all real modules, and establish the real module registry without yet requiring all modules to be fully implemented.

Scope

Implement:

• Shared LLM proposal generation helper.
• Proposal prompt request/response models.
• Structured correction set response parsing.
• Proposal index assignment.
• Module prompt/response diagnostics.
• Module registry package for real module keys.
• Module construction from run specs.
• Clean unsupported-module behavior.
• Shared module test harness using fake LLM responses.
• One minimal real module may be implemented as a proof of the proposal-generation path if that keeps the phase coherent, but only if it does not blur scope.

Do not implement:

• All real modules.
• Full default pipeline parity.
• Prompt improvements beyond faithful porting of Python behavior.

Expected behavior at end of phase

The framework can support real LLM proposal generation, and modules can be registered and instantiated consistently. At least the infrastructure for real modules exists, even if most modules are implemented in later phases.

Definition of done

• Shared proposal-generation helper exists.
• Proposal prompt/response diagnostics are written for module proposal calls.
• Module registry resolves known module keys deterministically.
• Unsupported modules fail cleanly.
• Fake module tests exercise shared proposal-generation behavior.
• Module reports include proposal-generation failures where applicable.
• go test ./... passes.

Phase 12: Grammar module

Purpose

Implement the first production module in the Go runtime path. Grammar is a good first real module because it exercises LLM proposal generation and validator chains while remaining constrained to punctuation, capitalization, and spacing cleanup.

Scope

Implement:

• grammar module package.
• Grammar prompt builder ported from the Python implementation.
• Grammar structured response model.
• Grammar replacement policy.
• Grammar validator chain.
• Grammar confidence threshold handling.
• Prompt/response diagnostics.
• Module-level report integration.
• CLI support for --modules grammar.
• Fake LLM tests.
• Optional real LLM smoke test gated so normal go test ./... does not require credentials.

Do not implement:

• Glossary module.
• Homophones module.
• Spoken-word module.
• Default full module sequence as active parity claim.

Expected behavior at end of phase

Running audita process ... --modules grammar should perform real grammar-stage transcript polishing using the configured LLM endpoint.

Definition of done

• Grammar module runs in the production runner.
• Grammar module generates structured proposals through the LLM client.
• Grammar validator chain runs.
• Approved grammar proposals are applied.
• Applied/skipped grammar changes appear in reports.
• Prompt/response diagnostics are written.
• --modules grammar works end-to-end.
• Default pipeline is not yet claimed complete.
• go test ./... passes without requiring external LLM credentials.

Phase 13: Glossary module and protected-term behavior

Purpose

Implement the glossary correction module and the glossary-derived protection behavior needed by downstream modules.

Scope

Implement:

• glossary module package.
• Glossary prompt builder ported from Python.
• Glossary structured response model.
• Glossary replacement policy.
• Glossary confidence threshold handling.
• Glossary validator chain.
• Protected-term extraction from parsed glossary.
• Protected-term validator behavior used by other modules where applicable.
• Prompt/response diagnostics.
• CLI support for --modules glossary.
• Fake LLM tests.
• Tests for repeated glossary stages using glossary,glossary.

Do not implement:

• Homophones module.
• Spoken-word module.
• Default full pipeline parity claim.

Expected behavior at end of phase

Running audita process ... --modules glossary should perform real glossary-supported corrections. Repeated glossary stages should work and be reported as separate module instances.

Definition of done

• Glossary module runs in the production runner.
• Glossary terms and aliases are used in prompts and validators.
• Protected-term behavior is implemented and tested.
• Repeated glossary module instances are reported correctly.
• Applied/skipped glossary changes appear in reports.
• Prompt/response diagnostics are written.
• go test ./... passes without requiring external LLM credentials.

Phase 14: Homophones module

Purpose

Implement conservative homophone and mistranscription correction behavior.

Scope

Implement:

• homophones module package.
• Homophones prompt builder ported from Python.
• Homophones structured response model.
• Homophones replacement policy.
• Homophones confidence threshold handling.
• Homophones validator chain.
• Prompt/response diagnostics.
• CLI support for --modules homophones.
• Fake LLM tests.
• Tests for interaction with glossary/protected terms where relevant.

Do not implement:

• Spoken-word module.
• Default full pipeline parity claim unless spoken-word is already complete.

Expected behavior at end of phase

Running audita process ... --modules homophones should perform real conservative homophone/mistranscription corrections using the configured LLM endpoint.

Definition of done

• Homophones module runs in the production runner.
• Homophones proposals are generated through structured LLM calls.
• Validator chain is enforced.
• Protected-term behavior is respected where applicable.
• Applied/skipped homophone changes appear in reports.
• Prompt/response diagnostics are written.
• go test ./... passes without requiring external LLM credentials.

Phase 15: Spoken-word module

Purpose

Implement conservative dysfluency cleanup while preserving substantive meaning.

Scope

Implement:

• spoken_word module package.
• Spoken-word prompt builder ported from Python.
• Spoken-word structured response model.
• Spoken-word replacement policy.
• Spoken-word confidence threshold handling.
• Spoken-word validator chain.
• Strong semantic guardrails using LLM-backed validators.
• Prompt/response diagnostics.
• CLI support for --modules spoken_word.
• Fake LLM tests.
• Tests for rejection of meaning-changing cleanup.

Do not implement:

• Prompt redesign beyond faithful porting.
• New stylistic rewriting behavior not present in the Python implementation.

Expected behavior at end of phase

Running audita process ... --modules spoken_word should perform real conservative dysfluency cleanup, with guardrails against semantic changes.

Definition of done

• Spoken-word module runs in the production runner.
• Spoken-word proposals are generated through structured LLM calls.
• Semantic validators reject meaning-changing proposals.
• Applied/skipped spoken-word changes appear in reports.
• Prompt/response diagnostics are written.
• go test ./... passes without requiring external LLM credentials.

Phase 16: Default full pipeline integration

Purpose

Enable and harden the full default module sequence in the Go runtime path.

Scope

Implement:

• Default runtime sequence:
  • glossary
  • homophones
  • glossary
  • spoken_word
  • grammar
• End-to-end execution through all real modules.
• Accurate resolved module instance names.
• Module-level report aggregation.
• Full run-level applied/skipped summaries.
• Skip-aware retention behavior using actual skipped correction data.
• Failure behavior with partial module progress.
• PipelineRunError or equivalent partial-progress error type if not already implemented.
• Diagnostics for each module instance.
• Tests for successful full pipeline using fake LLM.
• Tests for mid-pipeline failure preserving partial report and diagnostics.

Do not implement:

• Python archive removal.
• Side-by-side rollout tooling beyond what is needed for tests.

Expected behavior at end of phase

Running audita process transcript.json --glossary glossary.yaml --output corrected.json should execute the full Go module pipeline and produce a corrected transcript.

Definition of done

• Default module sequence runs end-to-end.
• All real modules participate in runtime path.
• Reports include all module instances.
• Applied/skipped changes are aggregated at run level.
• Failed runs retain useful partial reports and diagnostics.
• auto retention keeps successful runs with skipped corrections.
• CLI stdout/stderr behavior remains subprocess-safe.
• go test ./... passes without requiring external LLM credentials.

Phase 17: Python parity fixture suite

Purpose

Establish confidence that the Go implementation matches the behavior and safety posture of the initial Python implementation.

Scope

Implement:

• Parity fixtures based on representative Python-era inputs and expected behaviors.
• Fake LLM response fixtures where exact deterministic behavior is required.
• Golden tests for:
  • transcript schema handling
  • glossary schema handling
  • normalization
  • chunking
  • proposal application
  • validator behavior
  • module reports
  • diagnostics artifacts
  • default pipeline shape
• Comparison tests or scripts that can run Python and Go side by side where practical.
• Documentation of intentional differences between Python and Go.

Do not require:

• Real LLM credentials for normal automated tests.
• Exact nondeterministic natural-language output equivalence across Python and Go.

Expected behavior at end of phase

The repository has a durable test suite demonstrating that the Go implementation preserves the functional contract of the Python implementation.

Definition of done

• Representative parity fixtures exist.
• Golden tests cover deterministic behavior.
• Fake LLM tests cover full pipeline behavior.
• Intentional differences from Python are documented.
• Unintentional compatibility breaks are fixed.
• go test ./... passes.

Phase 18: Operational hardening and subprocess integration

Purpose

Harden the Go binary for use as the production Audita implementation in the surrounding application suite.

Scope

Implement:

• Additional subprocess tests for large transcripts.
• Timeout/cancellation tests.
• Failure-mode tests for unreadable inputs, unwritable outputs, malformed LLM responses, and backend failures.
• Clear stderr summaries pointing to diagnostics.
• Review of all output paths and file-close behavior.
• Review of all secret redaction paths.
• Review of LLM retry and timeout behavior.
• Documentation for production use from orchestrators such as Narratio.

Expected behavior at end of phase

The Go binary should be safe to call from other Go applications and should not reproduce the Python subprocess/stdio integration problems.

Definition of done

• Subprocess tests cover success, failure, large inputs, and cancellation.
• stdout contains machine output only.
• stderr contains human-readable logs/errors only.
• API keys do not appear in reports, diagnostics, logs, or tests.
• Failed runs always preserve diagnostics.
• Operational docs are accurate.
• go test ./... passes.

Phase 19: Documentation, rollout, and Python retirement

Purpose

Make the Go implementation the documented active implementation and preserve the Python implementation only as historical reference if desired.

Scope

Implement:

• README updates.
• Architecture updates.
• Rewrite notes updates.
• Installation/build instructions for Go binary.
• Migration notes from Python to Go.
• Documentation of any intentionally changed behavior.
• Removal or archival of Python-specific operational instructions from the primary path.
• Clear statement that the Go implementation is now feature-complete.

Expected behavior at end of phase

The repository clearly presents the Go implementation as the active Audita implementation.

Definition of done

• Documentation no longer describes Go as only a deterministic foundation.
• Documentation accurately describes full transcript polishing behavior.
• Python implementation is marked archived/prototype/reference, or removed if that is the chosen repository policy.
• Users can build, test, and run the Go implementation from docs alone.
• go test ./... passes.

Cross-phase compatibility requirements

These constraints apply to every remaining phase:

• Preserve subprocess-safe stdout/stderr behavior.
• Preserve CLI-over-env precedence.
• Preserve existing accepted transcript input forms.
• Preserve glossary YAML compatibility unless a deliberate migration is documented.
• Preserve deterministic safety-first proposal application semantics.
• Preserve run-dir diagnostics and report writing.
• Preserve API key redaction.
• Do not require real LLM credentials for normal go test ./....
• Do not claim a feature is implemented until it is in the runtime path.
• Keep default behavior honest in docs and CLI help.
• Prefer fake LLMs and golden fixtures for automated tests.
• Keep real LLM smoke tests opt-in.

Guidance for Codex-style prompts

When requesting implementation work, use one phase at a time.

Good prompt shape:

• Name the exact phase.
• State what is in scope.
• State what is explicitly out of scope.
• Require tests.
• Require go test ./....
• Require documentation updates only when the phase changes user-visible or architectural status.
• Require that no later-phase features be implemented opportunistically.

Avoid broad prompts such as:

• “finish the rewrite”
• “make Audita work”
• “port the Python app”
• “implement all modules”

Those prompts blur phase boundaries and make review difficult.

Suggested review checklist for every phase

Before accepting a phase implementation, verify:

• Does the implementation match the phase scope?
• Did it avoid implementing unrelated later-phase behavior?
• Does go test ./... pass?
• Are stdout and stderr still clean for subprocess callers?
• Are API keys redacted everywhere?
• Are reports machine-readable?
• Are diagnostics sufficient for debugging?
• Are fake LLM tests used instead of requiring real credentials?
• Are new public behaviors documented?
• Does the code follow the architecture in docs/architecture.md?
• Does the implementation move Audita closer to Python feature parity?