audita/docs/rewrite-notes.md

Audita Go Rewrite Notes

Rewrite strategy

The Go rewrite should be compatibility-first. The existing Python implementation should be treated as the executable specification for public behavior, pipeline semantics, diagnostics, and safety rules.

The rewrite should not begin as a redesign. The first production-capable Go version should be able to stand in for the Python CLI in the surrounding transcript pipeline.

The initial goal is:

same public contract
same pipeline semantics
same safety posture
same diagnostic philosophy
idiomatic Go implementation
reliable subprocess behavior

Exact LLM output parity is not required because LLM calls are nondeterministic and may vary by backend, prompt formatting, model, or structured-output implementation. Deterministic framework behavior should be held to a much stricter compatibility standard.

Core principles

1. Preserve behavior before improving behavior.
2. Port deterministic layers before LLM layers.
3. Test with fake LLMs before testing with real LLMs.
4. Keep module stages sequential until correctness is established.
5. Add bounded concurrency only after the sequential implementation is correct.
6. Keep the CLI contract stable for downstream callers.
7. Keep stdout/stderr behavior clean and predictable.
8. Treat diagnostics and reports as part of the product, not as afterthoughts.
9. Avoid prompt improvements during the port.
10. Prefer explicit Go structs and validation over reflection-heavy abstractions.

Compatibility targets

The Go implementation should preserve the following public behaviors where practical:

• audita process command shape;
• transcript input support for bare segment arrays and { "segments": [...] } objects;
• glossary YAML support;
• default module sequence;
• repeated module instance naming;
• environment variable and CLI flag configuration concepts;
• CLI-over-env precedence;
• output file behavior;
• stdout behavior when --output is omitted;
• stderr logging behavior;
• separate --report-json machine-readable report behavior;
• work-dir diagnostics behavior;
• retention modes auto, always, and never;
• failed-run diagnostics preservation;
• skipped-change reporting instead of crashing on stale or unsafe proposal application.

Phase 0: Freeze the Python implementation as the reference

Before writing substantial Go code, preserve the behavior of the current Python implementation.

Tasks:

1. Create or identify a stable branch/tag representing the Python reference implementation.
2. Preserve the existing README and architecture notes.
3. Preserve the Python regression suite.
4. Collect representative transcript/glossary fixtures.
5. Capture reference outputs for deterministic behaviors.
6. Capture several real end-to-end run artifacts for qualitative comparison.

Representative fixtures should include:

• tiny valid transcript;
• transcript with multiple speakers;
• transcript accepted as a bare array;
• transcript accepted as an object with segments;
• glossary-supported correction case;
• homophone/mistranscription case;
• spoken-word dysfluency case;
• grammar/punctuation/capitalization case;
• duplicate original_text case;
• missing original_text case;
• skipped-change case;
• malformed input case;
• failed LLM response case.

Definition of done:

• A developer or LLM agent can run the Python test suite.
• Reference fixtures are committed.
• Reference reports or golden artifacts exist for deterministic comparisons.
• The Python implementation can be used to answer disputed behavior questions during the rewrite.

Phase 1: Create the Go CLI skeleton

Build the smallest Go binary that preserves the outer command shape.

Tasks:

1. Initialize Go module.
2. Add cmd/audita/main.go.
3. Implement audita process command.
4. Parse core flags:
   • transcript path;
   • --glossary;
   • --output;
   • --report-json;
   • --modules;
   • LLM config flags;
   • normalization flags;
   • work-dir flags.
5. Implement config defaults.
6. Implement environment variable loading.
7. Implement CLI-over-env precedence.
8. Implement redaction for sensitive config values.
9. Load input files but initially write transcript back unchanged.

Definition of done:

• go test ./... passes.
• go run ./cmd/audita process transcript.json --glossary glossary.yaml --output corrected.json succeeds for a minimal fixture.
• The output transcript is valid JSON.
• With --output, stdout is empty except for intentional machine output, preferably empty.
• Logs go to stderr.
• Invalid flags produce a clean error and nonzero exit.

Phase 2: Port schemas and file I/O

Implement typed transcript, glossary, and report data structures.

Tasks:

1. Define transcript segment structs.
2. Support input as bare segment array.
3. Support input as object with segments array.
4. Preserve optional categories.
5. Validate required fields.
6. Validate basic timing shape.
7. Define glossary structs matching the existing YAML format.
8. Implement glossary parsing.
9. Implement report structs at least sufficient for early phases.
10. Implement JSON/YAML read/write helpers.

Definition of done:

• Valid Python-era fixtures parse successfully.
• Invalid fixtures fail with clear errors.
• Transcript round-trips through Go without accidental data loss.
• Glossary fixtures parse successfully.
• Report JSON can be written and parsed by tests.

Phase 3: Port deterministic normalization

Port normalization before any LLM work.

Tasks:

1. Implement same-speaker segment merging.
2. Implement maximum merge gap.
3. Implement ellipsis gap insertion.
4. Implement maximum merged segment duration.
5. Implement maximum merged segment token budget.
6. Reassign strict sequential segment IDs starting at 1.
7. Produce a normalization summary.
8. Preserve source transcript and normalized transcript diagnostics.

Definition of done:

• Normalization tests pass against golden fixtures.
• Segment IDs are deterministic.
• Chronological ordering is deterministic.
• Normalization summary is included in the run report.
• Normalization artifacts are written in the run directory.

Phase 4: Port chunking and token estimation

Implement the section-building logic used by proposal generation and validator batching.

Tasks:

1. Implement approximate token estimator.
2. Implement contiguous transcript sections.
3. Implement minimum and maximum section token settings.
4. Implement exact target section count behavior if supported by the Python implementation.
5. Implement validation prompt batching by token limit.
6. Add tests for edge cases.

Edge cases:

• very small transcript;
• single very large segment;
• many short segments;
• exact target section count impossible;
• min/max token bounds conflict;
• speaker boundaries near section boundaries.

Definition of done:

• Chunking is deterministic.
• Sections preserve transcript order.
• Sections do not drop or duplicate segments.
• Token-boundary behavior is covered by tests.

Phase 5: Port proposal model and application semantics

Implement correction proposals and safe replacement behavior.

Tasks:

1. Define correction proposal struct.
2. Define enriched proposal metadata.
3. Define replacement policies:
   • require_unique;
   • replace_all.
4. Implement proposal preview.
5. Implement proposal application.
6. Implement skipped-change reporting.
7. Handle stale proposals safely.
8. Add tests for every skip reason.

Skip cases should include:

• missing target segment;
• missing original_text;
• multiple matches under require_unique;
• empty corrected text;
• identical original/corrected text after normalization;
• proposal invalidated by earlier edits.

Definition of done:

• Proposal application never panics on malformed proposal input.
• Unsafe proposals are skipped and reported.
• Applied changes and skipped changes are serializable in the report.
• Tests cover both replacement policies.

Phase 6: Implement reports and diagnostics

Build diagnostics and reporting before real LLM calls so failures are inspectable from the beginning.

Tasks:

1. Create per-run directory under configured work dir.
2. Write redacted invocation/config metadata.
3. Write source transcript artifact.
4. Write normalized transcript artifact.
5. Write normalization summary artifact.
6. Write module prompt/response artifacts once module execution exists.
7. Write authoritative report.json into retained run directories.
8. Implement --report-json output path.
9. Implement error.log on failure.
10. Implement retention modes.

Retention behavior:

always  keep all run directories
never   remove successful run directories
auto    keep failed runs and successful runs with skipped corrections

Failed runs are always retained.

Definition of done:

• Clean successful runs obey retention policy.
• Successful runs with skipped corrections are retained under auto.
• Failed runs are always retained.
• Failed runs write report.json and error.log when possible.
• CLI stderr points to retained diagnostics on failure.

Phase 7: Implement the pipeline runner with fake modules

Build the orchestration engine before porting real correction modules.

Tasks:

1. Define module interface.
2. Define validator interface.
3. Define runner context.
4. Define module run report model.
5. Implement module sequence resolution.
6. Implement repeated module instance naming.
7. Implement sequential module execution.
8. Implement fake module for tests.
9. Implement fake validator for tests.
10. Wire proposal application into runner.

Definition of done:

• A fake module can propose a correction and the runner applies it.
• A fake validator can reject a correction and the runner reports it.
• Multiple modules run in configured order.
• Repeated module keys are named deterministically.
• A module failure produces a partial failed report.

Phase 8: Port deterministic validators

Port cheap, deterministic validation before LLM-backed validation.

Tasks:

1. Implement confidence threshold validator.
2. Implement original-text presence validator.
3. Implement non-empty correction validator.
4. Implement identical-text rejection.
5. Implement protected vocabulary logic from glossary.
6. Enforce validator result cardinality.
7. Add tests for malformed validator results.

Definition of done:

• Deterministic validators run before LLM validators.
• Each validator returns exactly one decision per input proposal.
• Missing, duplicate, or unknown proposal indexes produce framework errors.
• Rejected proposals are reported with reasons.

Phase 9: Implement structured LLM client

Add the OpenAI-compatible structured-output client after the deterministic framework is stable.

Tasks:

1. Define StructuredLLMClient interface.
2. Define structured request type.
3. Implement OpenAI-compatible chat completions client.
4. Support configurable base URL, model, API key, timeout, and retries.
5. Support optional API key for self-hosted endpoints.
6. Implement structured JSON response parsing.
7. Implement response validation.
8. Implement retry behavior for malformed structured output.
9. Write raw prompt/response diagnostics.
10. Support separate effective validation LLM settings.

Definition of done:

• Fake LLM tests still pass.
• Real LLM smoke test can run against a configured endpoint.
• Malformed LLM responses are retried or reported cleanly.
• API keys are never logged or written to diagnostics.
• Context timeout/cancellation works.

Phase 10: Add bounded concurrency

Add concurrency only after sequential correctness is established.

Tasks:

1. Add LLM call scheduler/semaphore.
2. Add proposal-generation concurrency across transcript sections.
3. Add validator batching concurrency where useful.
4. Ensure all goroutines are attached to run context.
5. Use errgroup or equivalent controlled error propagation.
6. Add tests for concurrency limits using fake LLM instrumentation.

Rules:

• Module stages remain sequential.
• LLM backend calls must respect configured concurrency.
• No unbounded goroutine creation.
• Cancellation must stop pending work promptly.

Definition of done:

• Concurrency limit is enforced in tests.
• Results remain deterministic where ordering matters.
• Module reports remain stable.
• Cancellation and timeout behavior are covered.

Phase 11: Port real modules one at a time

Port modules after the framework, validators, and LLM client are in place.

Recommended order:

1. grammar
2. glossary
3. homophones
4. spoken_word
5. full default sequence

This order exercises formatting first, then domain-specific correction, then more subtle semantic cleanup.

For each module:

1. Port prompt builder without improving wording.
2. Define structured response type.
3. Define JSON schema if supported by the backend.
4. Convert structured response into framework proposals.
5. Attach intended validators.
6. Add fake-LLM tests.
7. Add fixture-level integration tests.
8. Add real LLM smoke test where practical.
9. Compare qualitative output against Python reference.

Definition of done for each module:

• Fake-LLM tests pass.
• Module-specific fixtures pass.
• Prompt/response diagnostics are written.
• Validator chain is explicit and tested.
• Module report includes applied and skipped changes.

Phase 12: Full-pipeline compatibility testing

Run complete Go pipeline tests after all modules exist.

Tasks:

1. Run full default module sequence with fake LLM responses.
2. Run full default module sequence against small real transcript.
3. Run against at least one real D&D session transcript.
4. Compare output shape against Python reference.
5. Compare report shape against Python reference.
6. Inspect skipped changes manually.
7. Inspect diagnostics manually.
8. Verify downstream parser compatibility.

Definition of done:

• Go output can be consumed by the next pipeline stage.
• Report JSON can be consumed by existing tooling or by documented replacement tooling.
• Real run diagnostics are sufficient for debugging.
• Output quality is at least acceptable compared with Python.

Phase 13: Subprocess integration testing

Test the compiled Go binary exactly as surrounding Go applications will call it.

Tasks:

1. Create integration tests that execute the binary as a subprocess.
2. Test --output path behavior.
3. Test stdout-only transcript output when --output is omitted.
4. Test stderr-only logs/errors.
5. Test --report-json path behavior.
6. Test nonzero exit on invalid input.
7. Test nonzero exit on LLM failure.
8. Test large transcript pipe behavior.
9. Test timeout/cancellation behavior from parent process.

Definition of done:

• No stdout/stderr deadlocks.
• Parent Go process can reliably distinguish success from failure by exit code.
• Parent Go process can parse output transcript and report files.
• Failed runs provide diagnostics paths.

Phase 14: Run Python and Go side by side

For a transition period, keep both implementations available.

Tasks:

1. Give the Python binary a distinct name if needed, such as audita-python.
2. Give the Go binary the canonical audita name only after it is ready.
3. Allow the orchestrator to select implementation during transition.
4. Run the same real sessions through both implementations.
5. Compare reports and final transcripts.
6. Record known intentional differences.
7. Fix unintentional compatibility breaks.

Comparison criteria:

• both complete successfully;
• both preserve transcript JSON shape;
• both run the intended module sequence;
• both report applied/skipped changes;
• Go has cleaner subprocess behavior;
• Go diagnostics are at least as useful as Python diagnostics;
• Go transcript quality is acceptable for downstream use.

Definition of done:

• Go implementation has passed several real-session runs.
• Downstream applications can call Go Audita reliably.
• Any report/schema differences are documented.
• Python can be retired from the main path.

Phase 15: Retire or archive the Python implementation

After the Go version becomes the default, preserve the Python implementation as historical reference unless there is a reason to remove it.

Tasks:

1. Mark Python implementation as archived or prototype.
2. Preserve useful fixtures and tests.
3. Preserve prompts and reference outputs.
4. Remove Python from production orchestration.
5. Update README to describe Go as the active implementation.
6. Update installation instructions.
7. Update operational docs.

Definition of done:

• Repository clearly identifies the Go implementation as active.
• Python code no longer participates in normal operation.
• Historical Python behavior remains available for reference if needed.

Suggested milestone commits

A reasonable commit sequence for agent-assisted implementation:

1. Scaffold Go CLI and config loading.
2. Add transcript/glossary schemas and I/O.
3. Add normalization and normalization tests.
4. Add chunking and token estimation.
5. Add proposal model and replacement policies.
6. Add reports and diagnostics lifecycle.
7. Add pipeline runner with fake modules.
8. Add deterministic validators.
9. Add structured LLM client interface and fake client.
10. Add OpenAI-compatible LLM client.
11. Add bounded LLM scheduler.
12. Port grammar module.
13. Port glossary module.
14. Port homophones module.
15. Port spoken-word module.
16. Enable default full pipeline.
17. Add subprocess integration tests.
18. Update README and operational documentation.

Each milestone should leave the repository in a passing state.

Guidance for LLM implementation agents

When using an LLM agent such as Codex to implement this rewrite, prefer small prompts tied to a single milestone.

Good agent instructions:

• specify the exact phase being implemented;
• name the files or packages to create;
• require tests for the new behavior;
• require go test ./... to pass;
• prohibit prompt wording changes unless the phase is explicitly about prompts;
• require preserving CLI compatibility;
• require preserving stdout/stderr discipline;
• require staging/committing only after tests pass, if the workflow permits commits.

Avoid broad prompts such as “rewrite Audita in Go.” They invite unnecessary redesign and make review difficult.

Review checklist for each phase

Before accepting a phase implementation, check:

• Does it preserve the public contract?
• Does it introduce unnecessary redesign?
• Are errors typed and understandable?
• Are reports still machine-readable?
• Are diagnostics sufficient to debug failures?
• Are API keys redacted?
• Are stdout and stderr clean?
• Are tests deterministic?
• Does go test ./... pass?
• Is the code idiomatic Go rather than Python-shaped Go?

Phase 2 Status (Go Implementation)

The Go rewrite has completed Phase 2 with the following capabilities:

Implemented Features:

1. Typed Schema Validation:

   • Transcript validation supports both bare segment arrays and objects with segments field
   • Glossary validation matches the existing YAML format
   • Comprehensive field validation and error reporting

2. Deterministic Normalization:

   • Chronological sorting of segments
   • Same-speaker adjacent segment merging with configurable gap thresholds
   • Ellipsis insertion for gaps above the ellipsis threshold
   • Configurable maximum segment duration and token limits
   • Sequential ID reassignment starting at 1
   • Category preservation during merging

3. Process Output:

   • Output is normalized transcript JSON (not raw passthrough)
   • Canonical transcript shape as specified in architecture docs
   • Proper stdout/stderr discipline preserved from Phase 1

4. Enhanced Reporting:

   • Success reports include normalization statistics
   • Failure reports include error phase and concise messages
   • No secrets leaked to reports

5. Minimal Diagnostics:

   • Per-run work directories with unique timestamps
   • Source transcript artifacts (raw and parsed)
   • Normalized transcript artifacts
   • Normalization summary artifacts
   • Authoritative report.json in run directory
   • Error logs on failure
   • Basic retention policy implementation

Not Yet Implemented (Future Phases):

• Chunking and token-bounded section building
• LLM proposal generation and structured output
• Validator chains and decision logging
• Module execution pipeline
• Full diagnostics retention policy
• Skipped correction detection for auto retention
• Real module implementations (glossary, homophones, spoken_word, grammar)

Compatibility Notes:

The Go implementation maintains the public CLI contract from the Python implementation:

• audita process transcript.json --glossary glossary.yaml --output corrected.json
• stdout contains transcript JSON when --output is omitted
• stderr contains logs and errors only
• --report-json writes machine-readable reports
• Nonzero exit codes on failure

The normalization behavior is deterministic and matches the architectural specifications, but may differ from Python-era behavior in edge cases until full module implementation is complete.

When to consider an HTTP API

An HTTP API should be deferred until the Go CLI is stable.

Consider HTTP later only if one or more of these become true:

• multiple callers need a shared long-running Audita worker;
• process startup time becomes a meaningful bottleneck;
• a job queue is needed;
• remote execution becomes necessary;
• centralized concurrency control is preferable to per-process control;
• operational monitoring of Audita as a service becomes valuable.

Even then, the HTTP API should wrap the same core Go pipeline used by the CLI. The CLI should remain the simplest and most reliable integration surface.