audita/docs/rewrite-notes.md

Audita Go Rewrite Notes

Definition of done for the Go rewrite

The Go rewrite is complete when both of the following are true:

1. Feature parity with the initial Python implementation:

   • audita process performs end-to-end transcript polishing, not only deterministic preprocessing.
   • The default module sequence is implemented and active in the runtime path:
     • glossary
     • homophones
     • glossary
     • spoken_word
     • grammar
   • Repeated module instances are resolved deterministically, for example glossary_1 and glossary_2.
   • Real LLM-backed proposal generation is implemented through an OpenAI-compatible structured-output client.
   • Deterministic validators and LLM-backed validators are implemented and enforced in the runtime path.
   • Proposal application preserves the safety-first semantics of the Python implementation.
   • Structured run reports and diagnostics are sufficient for debugging successful runs, skipped corrections, and failures.
   • The Go CLI can replace the Python CLI in the surrounding orchestration pipeline without downstream contract surprises.

2. Adherence to the intended application architecture in docs/architecture.md:

   • Sequential module pipeline over a mutable working transcript.
   • Bounded intra-module concurrency only.
   • Provider-neutral LLM abstraction.
   • Separate proposal and validation LLM settings.
   • Prompt/response diagnostics for LLM stages.
   • Module-level run reports with applied and skipped changes.
   • Strict stdout/stderr discipline for subprocess callers.
   • No hidden service dependency; the CLI remains the primary integration surface.

Current implementation status

The Go rewrite is currently in a deterministic foundation stage.

Implemented:

• CLI command surface for audita process.
• Config/env/flag loading and validation.
• Transcript/glossary schema parsing and validation.
• Deterministic normalization with summary stats.
• Deterministic chunking with summary stats.
• Per-run diagnostics directory plus source/normalization/chunking artifacts.
• Redacted invocation/effective-config diagnostics metadata artifacts.
• Process report output through --report-json and run-dir report.json.
• Report-level diagnostics artifact references.
• Framework foundation packages for contracts and proposal preview/apply semantics.
• Production runner orchestration over a mutable working transcript.
• Module-level report structures and run-level module summaries.
• CLI runner integration point via injectable module factory/registry (used by deterministic tests).
• Runtime validator models and deterministic validator implementations.
• Validator cardinality enforcement (missing/duplicate/unknown proposal index errors).
• Deterministic validator-chain execution in the production runner.
• Module reports including validator decisions and validator rejections.
• Broad deterministic and CLI/subprocess test coverage for implemented phases through go test ./....
• Internal typed structured LLM contract (StructuredLLMClient.CompleteStructured(ctx, req, out)).
• internal/framework/llm instructor-go-backed adapter with:
  • configurable base URL/model/retries/mode/timeout
  • optional API key support for local-compatible endpoints
  • API-key redaction in returned errors
  • typed structured decode into caller-provided outputs.

Not yet implemented in runtime pipeline:

• Real correction modules.
• Structured LLM scheduler/concurrency orchestration.
• Runtime wiring from production runner/modules into the structured LLM adapter.
• LLM-backed validators.
• Prompt/response diagnostics for LLM calls.
• End-to-end transcript polishing behavior.

Completed phases

Phase 1: Go CLI skeleton

Completed.

Implemented:

• Go module and cmd/audita entrypoint.
• audita process command surface.
• Core flags and config wiring.
• Subprocess-safe command behavior foundation.

Phase 2: Schemas and file I/O

Completed.

Implemented:

• Transcript parsing for bare arrays and { "segments": [...] } input.
• Glossary YAML parsing.
• Transcript and glossary validation.
• Canonical transcript output serialization.

Phase 3: Deterministic normalization

Completed.

Implemented:

• Chronological sorting.
• Same-speaker segment merging.
• Gap-sensitive join behavior.
• Duration and token-budget merge constraints.
• Sequential normalized segment IDs.
• Normalization summary stats.

Phase 4: Chunking and token estimation

Completed.

Implemented:

• Deterministic heuristic token estimation.
• Contiguous transcript sectioning.
• Min/max section token behavior.
• Optional target section handling.
• Chunking summaries and diagnostics artifacts.

Phase 5: Proposal model and application semantics

Completed.

Implemented:

• Correction proposal and enriched proposal models.
• Replacement policies:
  • require_unique
  • replace_all
• Safe preview logic.
• Deterministic proposal application.
• Applied/skipped change records.
• Stable skip reasons.

Phase 6: Reports and diagnostics

Completed for the current deterministic runtime scope.

Implemented:

• Per-run diagnostics directory creation.
• Source transcript artifacts.
• Parsed source transcript artifacts.
• Normalized transcript artifact.
• Normalization summary artifact.
• Chunking summary artifact.
• Redacted invocation metadata artifact.
• Redacted effective-config artifact.
• Run-dir report.json.
• Optional external --report-json.
• Failure error.log.
• Report-level diagnostics artifact references.
• Retention decision model with future skipped-correction hook.

Intentionally deferred:

• Module prompt/response diagnostics artifacts are not produced yet because module execution and LLM calls are not implemented in the runtime path.

Phase 7: Pipeline runner with deterministic test modules

Completed.

Implemented:

• internal/framework/runner production package with sequential module orchestration.
• Deterministic module run-spec resolution and repeated instance naming (glossary_1, glossary_2, etc.).
• Mutable working transcript handoff across module instances.
• Proposal application through internal/framework/proposals.
• Per-module applied/skipped change capture and module status/timing metadata.
• Partial-progress return on module failure, with pipeline stop on first failure.
• Process report support for module-level results and run-level module summaries.
• CLI runtime integration point via injectable module factory/registry, exercised by deterministic fake-module tests.

Not implemented in Phase 7 (by design):

• Real glossary, homophones, spoken_word, grammar production modules.
• Validators (Phase 8).
• Structured LLM calls or scheduler behavior.
• Prompt/response diagnostics.
• End-to-end transcript polishing.

Current runtime behavior note:

• Default user-facing CLI behavior remains deterministic normalization/chunking output unless test-only module injection is used during tests.

Phase 8: Runtime validator framework and deterministic validators

Completed.

Implemented:

• Runtime validator request/result models in internal/framework/validators.
• Deterministic validator reason codes for stable reporting.
• Validator cardinality enforcement:
  • one decision per candidate proposal index
  • missing indexes are errors
  • duplicate indexes are errors
  • unknown indexes are errors
• Deterministic validators:
  • confidence threshold
  • original-text presence against working transcript
  • non-empty correction
  • identical/no-effect rejection
  • conservative protected glossary-term guard
• Ordered validator-chain execution in the production runner.
• Runner behavior where only validator-approved proposals proceed to proposal application.
• Module-level reporting of validator decisions and validator rejections, distinct from application-level skips.
• Deterministic fake-module tests covering approvals, rejections, validator order/filtering, and cardinality failure pipeline-stop behavior.

Not implemented in Phase 8 (by design):

• LLM-backed validators (Phase 10).
• Structured LLM scheduler behavior or runtime wiring (Phase 9 follow-up).
• Real correction modules.
• Prompt/response diagnostics.
• End-to-end transcript polishing.

Remaining work plan

Next recommended phase: Phase 9 follow-up (scheduler + runtime LLM wiring, still no real modules).

Phase 9: Structured LLM client and scheduler infrastructure

Status

Partially completed.

Implemented in this phase so far:

• Added internal structured LLM contract support for caller-provided typed outputs.
• Added internal/framework/llm adapter backed by github.com/jxnl/instructor-go.
• Confirmed OpenAI-compatible base URL support through the adapter path.
• Added adapter unit tests for model/base URL handling, retries, context cancellation, optional API key behavior, and error redaction.

Still pending in Phase 9:

• Scheduler/semaphore behavior for bounded concurrency.
• Runtime wiring in runner/module infrastructure (without introducing real modules yet).
• Prompt/response diagnostics writer primitives for LLM call artifacts.
• Full primary vs validation LLM config-resolution plumbing into runtime LLM call sites.

Purpose

Implement the provider-neutral LLM infrastructure needed by both proposal generation and LLM-backed validators, without yet implementing real modules.

Scope

Implement (remaining):

• Primary LLM config resolution.
• Validation LLM config resolution and inheritance from primary settings.
• Redaction of credentials in all diagnostics and reports.
• LLM scheduler/semaphore for bounded backend concurrency.
• Prompt/response diagnostics writer primitives that can later be used by modules and validators.

Do not implement:

• Real correction modules.
• LLM-backed validators.
• Prompt text for domain modules.
• End-to-end transcript polishing.

Expected behavior at end of phase

The codebase has a tested OpenAI-compatible structured-output client adapter, but scheduler and runtime wiring remain before this phase is fully complete. The CLI still does not perform real LLM polishing.

Definition of done

Remaining checklist to close Phase 9:

• Scheduler enforces configured concurrency.
• Primary and validation LLM settings resolve correctly in runtime wiring.
• Prompt/response diagnostic primitives exist.
• API keys are not leaked.
• No real module behavior is introduced.
• go test ./... passes.

Phase 10: LLM-backed validators

Purpose

Implement the LLM-backed validator layer used by the Python implementation, and wire it into the runtime validator framework.

Scope

Implement:

• LLM validator request and response models.
• Shared batching logic for validation prompts using validation token limits.
• Prompt builders for LLM validators.
• LLM-backed validation categories needed for parity, such as:
  • spoken-form plausibility
  • meaning reversal detection
  • editorial review
  • grammar review
  • spoken-word review
• Validator prompt/response diagnostics.
• Validation LLM scheduler usage.
• Validator error handling and report integration.
• Fake LLM tests for approval, rejection, malformed output, missing decision, duplicate decision, and retry cases.

Do not implement:

• Real correction modules, except for minimal fake/test modules needed to exercise validators.
• Full default pipeline behavior.
• Domain proposal prompts.

Expected behavior at end of phase

The runner can execute a mixed deterministic + LLM validator chain against proposals produced by fake modules. LLM validators use the structured LLM client and write diagnostics.

Definition of done

• LLM-backed validators are implemented.
• Validator batching respects configured token limits.
• Validator cardinality rules are enforced for LLM validator output.
• Prompt/response diagnostics are written for LLM validator calls.
• Validator results appear in module reports.
• Fake LLM tests cover success, rejection, malformed output, and retry behavior.
• go test ./... passes.

Phase 11: Shared LLM proposal generation framework and module registry

Purpose

Create the reusable proposal-generation layer used by all real modules, and establish the real module registry without yet requiring all modules to be fully implemented.

Scope

Implement:

• Shared LLM proposal generation helper.
• Proposal prompt request/response models.
• Structured correction set response parsing.
• Proposal index assignment.
• Module prompt/response diagnostics.
• Module registry package for real module keys.
• Module construction from run specs.
• Clean unsupported-module behavior.
• Shared module test harness using fake LLM responses.
• One minimal real module may be implemented as a proof of the proposal-generation path if that keeps the phase coherent, but only if it does not blur scope.

Do not implement:

• All real modules.
• Full default pipeline parity.
• Prompt improvements beyond faithful porting of Python behavior.

Expected behavior at end of phase

The framework can support real LLM proposal generation, and modules can be registered and instantiated consistently. At least the infrastructure for real modules exists, even if most modules are implemented in later phases.

Definition of done

• Shared proposal-generation helper exists.
• Proposal prompt/response diagnostics are written for module proposal calls.
• Module registry resolves known module keys deterministically.
• Unsupported modules fail cleanly.
• Fake module tests exercise shared proposal-generation behavior.
• Module reports include proposal-generation failures where applicable.
• go test ./... passes.

Phase 12: Grammar module

Purpose

Implement the first production module in the Go runtime path. Grammar is a good first real module because it exercises LLM proposal generation and validator chains while remaining constrained to punctuation, capitalization, and spacing cleanup.

Scope

Implement:

• grammar module package.
• Grammar prompt builder ported from the Python implementation.
• Grammar structured response model.
• Grammar replacement policy.
• Grammar validator chain.
• Grammar confidence threshold handling.
• Prompt/response diagnostics.
• Module-level report integration.
• CLI support for --modules grammar.
• Fake LLM tests.
• Optional real LLM smoke test gated so normal go test ./... does not require credentials.

Do not implement:

• Glossary module.
• Homophones module.
• Spoken-word module.
• Default full module sequence as active parity claim.

Expected behavior at end of phase

Running audita process ... --modules grammar should perform real grammar-stage transcript polishing using the configured LLM endpoint.

Definition of done

• Grammar module runs in the production runner.
• Grammar module generates structured proposals through the LLM client.
• Grammar validator chain runs.
• Approved grammar proposals are applied.
• Applied/skipped grammar changes appear in reports.
• Prompt/response diagnostics are written.
• --modules grammar works end-to-end.
• Default pipeline is not yet claimed complete.
• go test ./... passes without requiring external LLM credentials.

Phase 13: Glossary module and protected-term behavior

Purpose

Implement the glossary correction module and the glossary-derived protection behavior needed by downstream modules.

Scope

Implement:

• glossary module package.
• Glossary prompt builder ported from Python.
• Glossary structured response model.
• Glossary replacement policy.
• Glossary confidence threshold handling.
• Glossary validator chain.
• Protected-term extraction from parsed glossary.
• Protected-term validator behavior used by other modules where applicable.
• Prompt/response diagnostics.
• CLI support for --modules glossary.
• Fake LLM tests.
• Tests for repeated glossary stages using glossary,glossary.

Do not implement:

• Homophones module.
• Spoken-word module.
• Default full pipeline parity claim.

Expected behavior at end of phase

Running audita process ... --modules glossary should perform real glossary-supported corrections. Repeated glossary stages should work and be reported as separate module instances.

Definition of done

• Glossary module runs in the production runner.
• Glossary terms and aliases are used in prompts and validators.
• Protected-term behavior is implemented and tested.
• Repeated glossary module instances are reported correctly.
• Applied/skipped glossary changes appear in reports.
• Prompt/response diagnostics are written.
• go test ./... passes without requiring external LLM credentials.

Phase 14: Homophones module

Purpose

Implement conservative homophone and mistranscription correction behavior.

Scope

Implement:

• homophones module package.
• Homophones prompt builder ported from Python.
• Homophones structured response model.
• Homophones replacement policy.
• Homophones confidence threshold handling.
• Homophones validator chain.
• Prompt/response diagnostics.
• CLI support for --modules homophones.
• Fake LLM tests.
• Tests for interaction with glossary/protected terms where relevant.

Do not implement:

• Spoken-word module.
• Default full pipeline parity claim unless spoken-word is already complete.

Expected behavior at end of phase

Running audita process ... --modules homophones should perform real conservative homophone/mistranscription corrections using the configured LLM endpoint.

Definition of done

• Homophones module runs in the production runner.
• Homophones proposals are generated through structured LLM calls.
• Validator chain is enforced.
• Protected-term behavior is respected where applicable.
• Applied/skipped homophone changes appear in reports.
• Prompt/response diagnostics are written.
• go test ./... passes without requiring external LLM credentials.

Phase 15: Spoken-word module

Purpose

Implement conservative dysfluency cleanup while preserving substantive meaning.

Scope

Implement:

• spoken_word module package.
• Spoken-word prompt builder ported from Python.
• Spoken-word structured response model.
• Spoken-word replacement policy.
• Spoken-word confidence threshold handling.
• Spoken-word validator chain.
• Strong semantic guardrails using LLM-backed validators.
• Prompt/response diagnostics.
• CLI support for --modules spoken_word.
• Fake LLM tests.
• Tests for rejection of meaning-changing cleanup.

Do not implement:

• Prompt redesign beyond faithful porting.
• New stylistic rewriting behavior not present in the Python implementation.

Expected behavior at end of phase

Running audita process ... --modules spoken_word should perform real conservative dysfluency cleanup, with guardrails against semantic changes.

Definition of done

• Spoken-word module runs in the production runner.
• Spoken-word proposals are generated through structured LLM calls.
• Semantic validators reject meaning-changing proposals.
• Applied/skipped spoken-word changes appear in reports.
• Prompt/response diagnostics are written.
• go test ./... passes without requiring external LLM credentials.

Phase 16: Default full pipeline integration

Purpose

Enable and harden the full default module sequence in the Go runtime path.

Scope

Implement:

• Default runtime sequence:
  • glossary
  • homophones
  • glossary
  • spoken_word
  • grammar
• End-to-end execution through all real modules.
• Accurate resolved module instance names.
• Module-level report aggregation.
• Full run-level applied/skipped summaries.
• Skip-aware retention behavior using actual skipped correction data.
• Failure behavior with partial module progress.
• PipelineRunError or equivalent partial-progress error type if not already implemented.
• Diagnostics for each module instance.
• Tests for successful full pipeline using fake LLM.
• Tests for mid-pipeline failure preserving partial report and diagnostics.

Do not implement:

• Python archive removal.
• Side-by-side rollout tooling beyond what is needed for tests.

Expected behavior at end of phase

Running audita process transcript.json --glossary glossary.yaml --output corrected.json should execute the full Go module pipeline and produce a corrected transcript.

Definition of done

• Default module sequence runs end-to-end.
• All real modules participate in runtime path.
• Reports include all module instances.
• Applied/skipped changes are aggregated at run level.
• Failed runs retain useful partial reports and diagnostics.
• auto retention keeps successful runs with skipped corrections.
• CLI stdout/stderr behavior remains subprocess-safe.
• go test ./... passes without requiring external LLM credentials.

Phase 17: Python parity fixture suite

Purpose

Establish confidence that the Go implementation matches the behavior and safety posture of the initial Python implementation.

Scope

Implement:

• Parity fixtures based on representative Python-era inputs and expected behaviors.
• Fake LLM response fixtures where exact deterministic behavior is required.
• Golden tests for:
  • transcript schema handling
  • glossary schema handling
  • normalization
  • chunking
  • proposal application
  • validator behavior
  • module reports
  • diagnostics artifacts
  • default pipeline shape
• Comparison tests or scripts that can run Python and Go side by side where practical.
• Documentation of intentional differences between Python and Go.

Do not require:

• Real LLM credentials for normal automated tests.
• Exact nondeterministic natural-language output equivalence across Python and Go.

Expected behavior at end of phase

The repository has a durable test suite demonstrating that the Go implementation preserves the functional contract of the Python implementation.

Definition of done

• Representative parity fixtures exist.
• Golden tests cover deterministic behavior.
• Fake LLM tests cover full pipeline behavior.
• Intentional differences from Python are documented.
• Unintentional compatibility breaks are fixed.
• go test ./... passes.

Phase 18: Operational hardening and subprocess integration

Purpose

Harden the Go binary for use as the production Audita implementation in the surrounding application suite.

Scope

Implement:

• Additional subprocess tests for large transcripts.
• Timeout/cancellation tests.
• Failure-mode tests for unreadable inputs, unwritable outputs, malformed LLM responses, and backend failures.
• Clear stderr summaries pointing to diagnostics.
• Review of all output paths and file-close behavior.
• Review of all secret redaction paths.
• Review of LLM retry and timeout behavior.
• Documentation for production use from orchestrators such as Narratio.

Expected behavior at end of phase

The Go binary should be safe to call from other Go applications and should not reproduce the Python subprocess/stdio integration problems.

Definition of done

• Subprocess tests cover success, failure, large inputs, and cancellation.
• stdout contains machine output only.
• stderr contains human-readable logs/errors only.
• API keys do not appear in reports, diagnostics, logs, or tests.
• Failed runs always preserve diagnostics.
• Operational docs are accurate.
• go test ./... passes.

Phase 19: Documentation, rollout, and Python retirement

Purpose

Make the Go implementation the documented active implementation and preserve the Python implementation only as historical reference if desired.

Scope

Implement:

• README updates.
• Architecture updates.
• Rewrite notes updates.
• Installation/build instructions for Go binary.
• Migration notes from Python to Go.
• Documentation of any intentionally changed behavior.
• Removal or archival of Python-specific operational instructions from the primary path.
• Clear statement that the Go implementation is now feature-complete.

Expected behavior at end of phase

The repository clearly presents the Go implementation as the active Audita implementation.

Definition of done

• Documentation no longer describes Go as only a deterministic foundation.
• Documentation accurately describes full transcript polishing behavior.
• Python implementation is marked archived/prototype/reference, or removed if that is the chosen repository policy.
• Users can build, test, and run the Go implementation from docs alone.
• go test ./... passes.

Cross-phase compatibility requirements

These constraints apply to every remaining phase:

• Preserve subprocess-safe stdout/stderr behavior.
• Preserve CLI-over-env precedence.
• Preserve existing accepted transcript input forms.
• Preserve glossary YAML compatibility unless a deliberate migration is documented.
• Preserve deterministic safety-first proposal application semantics.
• Preserve run-dir diagnostics and report writing.
• Preserve API key redaction.
• Do not require real LLM credentials for normal go test ./....
• Do not claim a feature is implemented until it is in the runtime path.
• Keep default behavior honest in docs and CLI help.
• Prefer fake LLMs and golden fixtures for automated tests.
• Keep real LLM smoke tests opt-in.

Guidance for Codex-style prompts

When requesting implementation work, use one phase at a time.

Good prompt shape:

• Name the exact phase.
• State what is in scope.
• State what is explicitly out of scope.
• Require tests.
• Require go test ./....
• Require documentation updates only when the phase changes user-visible or architectural status.
• Require that no later-phase features be implemented opportunistically.

Avoid broad prompts such as:

• “finish the rewrite”
• “make Audita work”
• “port the Python app”
• “implement all modules”

Those prompts blur phase boundaries and make review difficult.

Suggested review checklist for every phase

Before accepting a phase implementation, verify:

• Does the implementation match the phase scope?
• Did it avoid implementing unrelated later-phase behavior?
• Does go test ./... pass?
• Are stdout and stderr still clean for subprocess callers?
• Are API keys redacted everywhere?
• Are reports machine-readable?
• Are diagnostics sufficient for debugging?
• Are fake LLM tests used instead of requiring real credentials?
• Are new public behaviors documented?
• Does the code follow the architecture in docs/architecture.md?
• Does the implementation move Audita closer to Python feature parity?