eric/audita
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 2 Wiki Activity

Update Go rewrite documentation status

Browse Source
This commit is contained in:
Eric Rakestraw
2026-05-11 08:49:26 -05:00
parent c3087aeda6
commit 73249b63d8
2 changed files with 260 additions and 1225 deletions
Download Patch File Download Diff File Expand all files Collapse all files

694
docs/rewrite-notes.md
View File

@@ -1,613 +1,85 @@
# Audita Go Rewrite Notes
## Rewrite strategy
The Go rewrite should be compatibility-first. The existing Python implementation should be treated as the executable specification for public behavior, pipeline semantics, diagnostics, and safety rules.
The rewrite should not begin as a redesign. The first production-capable Go version should be able to stand in for the Python CLI in the surrounding transcript pipeline.
The initial goal is:
```text
same public contract
same pipeline semantics
same safety posture
same diagnostic philosophy
idiomatic Go implementation
reliable subprocess behavior
```
Exact LLM output parity is not required because LLM calls are nondeterministic and may vary by backend, prompt formatting, model, or structured-output implementation. Deterministic framework behavior should be held to a much stricter compatibility standard.
## Core principles
1. Preserve behavior before improving behavior.
2. Port deterministic layers before LLM layers.
3. Test with fake LLMs before testing with real LLMs.
4. Keep module stages sequential until correctness is established.
5. Add bounded concurrency only after the sequential implementation is correct.
6. Keep the CLI contract stable for downstream callers.
7. Keep stdout/stderr behavior clean and predictable.
8. Treat diagnostics and reports as part of the product, not as afterthoughts.
9. Avoid prompt improvements during the port.
10. Prefer explicit Go structs and validation over reflection-heavy abstractions.
## Compatibility targets
The Go implementation should preserve the following public behaviors where practical:
- `audita process` command shape;
- transcript input support for bare segment arrays and `{ "segments": [...] }` objects;
- glossary YAML support;
- default module sequence;
- repeated module instance naming;
- environment variable and CLI flag configuration concepts;
- CLI-over-env precedence;
- output file behavior;
- stdout behavior when `--output` is omitted;
- stderr logging behavior;
- separate `--report-json` machine-readable report behavior;
- work-dir diagnostics behavior;
- retention modes `auto`, `always`, and `never`;
- failed-run diagnostics preservation;
- skipped-change reporting instead of crashing on stale or unsafe proposal application.
## Current Go rewrite status (Phase 3 foundation)
The Go rewrite now includes deterministic foundation pieces needed before real module/LLM behavior:
- typed transcript parsing and validation (bare array or `{ "segments": [...] }`);
- typed glossary YAML parsing and validation;
- deterministic transcript normalization and normalization summary;
- deterministic token estimation;
- contiguous transcript chunking with section metadata;
- correction proposal data models and enriched proposal metadata;
- replacement policy handling (`require_unique`, `replace_all`);
- safe per-segment proposal preview;
- deterministic transcript proposal application in stable proposal-index order;
- applied and skipped change records with stable skip reasons;
- report/diagnostic-oriented chunking summary and chunk metadata hooks;
- minimal framework contracts/interfaces for future modules, validators, and structured LLM clients;
- test-only fake-module composition tests proving chunking and proposal application can work together.
The Go rewrite does **not** yet include end-to-end transcript polishing behavior:
- no real module execution pipeline yet;
- no real `glossary`, `homophones`, `spoken_word`, or `grammar` module implementations;
- no LLM proposal generation;
- no validator-chain execution;
- no concrete structured LLM client implementation;
- no final LLM-driven transcript polishing behavior comparable to Python production runs.
This means the Go binary should not be described as a complete LLM transcript polisher yet.
## Developer quick note
Use these commands while developing deterministic rewrite phases:
```sh
go test ./...
go run ./cmd/audita process internal/cli/testdata/tiny_transcript.json --glossary internal/cli/testdata/tiny_glossary.yaml --output /tmp/audita-go-output.json --report-json /tmp/audita-go-report.json
```
The second command is a representative CLI wiring check only; it is not a real LLM polishing run in the current phase.
## Phase 0: Freeze the Python implementation as the reference
Before writing substantial Go code, preserve the behavior of the current Python implementation.
Tasks:
1. Create or identify a stable branch/tag representing the Python reference implementation.
2. Preserve the existing README and architecture notes.
3. Preserve the Python regression suite.
4. Collect representative transcript/glossary fixtures.
5. Capture reference outputs for deterministic behaviors.
6. Capture several real end-to-end run artifacts for qualitative comparison.
Representative fixtures should include:
- tiny valid transcript;
- transcript with multiple speakers;
- transcript accepted as a bare array;
- transcript accepted as an object with `segments`;
- glossary-supported correction case;
- homophone/mistranscription case;
- spoken-word dysfluency case;
- grammar/punctuation/capitalization case;
- duplicate `original_text` case;
- missing `original_text` case;
- skipped-change case;
- malformed input case;
- failed LLM response case.
Definition of done:
- A developer or LLM agent can run the Python test suite.
- Reference fixtures are committed.
- Reference reports or golden artifacts exist for deterministic comparisons.
- The Python implementation can be used to answer disputed behavior questions during the rewrite.
## Phase 1: Create the Go CLI skeleton
Build the smallest Go binary that preserves the outer command shape.
Tasks:
1. Initialize Go module.
2. Add `cmd/audita/main.go`.
3. Implement `audita process` command.
4. Parse core flags:
- transcript path;
- `--glossary`;
- `--output`;
- `--report-json`;
- `--modules`;
- LLM config flags;
- normalization flags;
- work-dir flags.
5. Implement config defaults.
6. Implement environment variable loading.
7. Implement CLI-over-env precedence.
8. Implement redaction for sensitive config values.
9. Load input files but initially write transcript back unchanged.
Definition of done:
- `go test ./...` passes.
- `go run ./cmd/audita process transcript.json --glossary glossary.yaml --output corrected.json` succeeds for a minimal fixture.
- The output transcript is valid JSON.
- With `--output`, stdout is empty except for intentional machine output, preferably empty.
- Logs go to stderr.
- Invalid flags produce a clean error and nonzero exit.
## Phase 2: Port schemas and file I/O
Implement typed transcript, glossary, and report data structures.
Tasks:
1. Define transcript segment structs.
2. Support input as bare segment array.
3. Support input as object with `segments` array.
4. Preserve optional `categories`.
5. Validate required fields.
6. Validate basic timing shape.
7. Define glossary structs matching the existing YAML format.
8. Implement glossary parsing.
9. Implement report structs at least sufficient for early phases.
10. Implement JSON/YAML read/write helpers.
Definition of done:
- Valid Python-era fixtures parse successfully.
- Invalid fixtures fail with clear errors.
- Transcript round-trips through Go without accidental data loss.
- Glossary fixtures parse successfully.
- Report JSON can be written and parsed by tests.
## Phase 3: Port deterministic normalization
Port normalization before any LLM work.
Tasks:
1. Implement same-speaker segment merging.
2. Implement maximum merge gap.
3. Implement ellipsis gap insertion.
4. Implement maximum merged segment duration.
5. Implement maximum merged segment token budget.
6. Reassign strict sequential segment IDs starting at `1`.
7. Produce a normalization summary.
8. Preserve source transcript and normalized transcript diagnostics.
Definition of done:
- Normalization tests pass against golden fixtures.
- Segment IDs are deterministic.
- Chronological ordering is deterministic.
- Normalization summary is included in the run report.
- Normalization artifacts are written in the run directory.
## Phase 4: Port chunking and token estimation
Implement the section-building logic used by proposal generation and validator batching.
Tasks:
1. Implement approximate token estimator.
2. Implement contiguous transcript sections.
3. Implement minimum and maximum section token settings.
4. Implement exact target section count behavior if supported by the Python implementation.
5. Implement validation prompt batching by token limit.
6. Add tests for edge cases.
Edge cases:
- very small transcript;
- single very large segment;
- many short segments;
- exact target section count impossible;
- min/max token bounds conflict;
- speaker boundaries near section boundaries.
Definition of done:
- Chunking is deterministic.
- Sections preserve transcript order.
- Sections do not drop or duplicate segments.
- Token-boundary behavior is covered by tests.
## Phase 5: Port proposal model and application semantics
Implement correction proposals and safe replacement behavior.
Tasks:
1. Define correction proposal struct.
2. Define enriched proposal metadata.
3. Define replacement policies:
- `require_unique`;
- `replace_all`.
4. Implement proposal preview.
5. Implement proposal application.
6. Implement skipped-change reporting.
7. Handle stale proposals safely.
8. Add tests for every skip reason.
Skip cases should include:
- missing target segment;
- missing `original_text`;
- multiple matches under `require_unique`;
- empty corrected text;
- identical original/corrected text after normalization;
- proposal invalidated by earlier edits.
Definition of done:
- Proposal application never panics on malformed proposal input.
- Unsafe proposals are skipped and reported.
- Applied changes and skipped changes are serializable in the report.
- Tests cover both replacement policies.
## Phase 6: Implement reports and diagnostics
Build diagnostics and reporting before real LLM calls so failures are inspectable from the beginning.
Tasks:
1. Create per-run directory under configured work dir.
2. Write redacted invocation/config metadata.
3. Write source transcript artifact.
4. Write normalized transcript artifact.
5. Write normalization summary artifact.
6. Write module prompt/response artifacts once module execution exists.
7. Write authoritative `report.json` into retained run directories.
8. Implement `--report-json` output path.
9. Implement `error.log` on failure.
10. Implement retention modes.
Retention behavior:
```text
always keep all run directories
never remove successful run directories
auto keep failed runs and successful runs with skipped corrections
```
Failed runs are always retained.
Definition of done:
- Clean successful runs obey retention policy.
- Successful runs with skipped corrections are retained under `auto`.
- Failed runs are always retained.
- Failed runs write `report.json` and `error.log` when possible.
- CLI stderr points to retained diagnostics on failure.
## Phase 7: Implement the pipeline runner with fake modules
Build the orchestration engine before porting real correction modules.
Tasks:
1. Define module interface.
2. Define validator interface.
3. Define runner context.
4. Define module run report model.
5. Implement module sequence resolution.
6. Implement repeated module instance naming.
7. Implement sequential module execution.
8. Implement fake module for tests.
9. Implement fake validator for tests.
10. Wire proposal application into runner.
Definition of done:
- A fake module can propose a correction and the runner applies it.
- A fake validator can reject a correction and the runner reports it.
- Multiple modules run in configured order.
- Repeated module keys are named deterministically.
- A module failure produces a partial failed report.
## Phase 8: Port deterministic validators
Port cheap, deterministic validation before LLM-backed validation.
Tasks:
1. Implement confidence threshold validator.
2. Implement original-text presence validator.
3. Implement non-empty correction validator.
4. Implement identical-text rejection.
5. Implement protected vocabulary logic from glossary.
6. Enforce validator result cardinality.
7. Add tests for malformed validator results.
Definition of done:
- Deterministic validators run before LLM validators.
- Each validator returns exactly one decision per input proposal.
- Missing, duplicate, or unknown proposal indexes produce framework errors.
- Rejected proposals are reported with reasons.
## Phase 9: Implement structured LLM client
Add the OpenAI-compatible structured-output client after the deterministic framework is stable.
Tasks:
1. Define `StructuredLLMClient` interface.
2. Define structured request type.
3. Implement OpenAI-compatible chat completions client.
4. Support configurable base URL, model, API key, timeout, and retries.
5. Support optional API key for self-hosted endpoints.
6. Implement structured JSON response parsing.
7. Implement response validation.
8. Implement retry behavior for malformed structured output.
9. Write raw prompt/response diagnostics.
10. Support separate effective validation LLM settings.
Definition of done:
- Fake LLM tests still pass.
- Real LLM smoke test can run against a configured endpoint.
- Malformed LLM responses are retried or reported cleanly.
- API keys are never logged or written to diagnostics.
- Context timeout/cancellation works.
## Phase 10: Add bounded concurrency
Add concurrency only after sequential correctness is established.
Tasks:
1. Add LLM call scheduler/semaphore.
2. Add proposal-generation concurrency across transcript sections.
3. Add validator batching concurrency where useful.
4. Ensure all goroutines are attached to run context.
5. Use `errgroup` or equivalent controlled error propagation.
6. Add tests for concurrency limits using fake LLM instrumentation.
Rules:
- Module stages remain sequential.
- LLM backend calls must respect configured concurrency.
- No unbounded goroutine creation.
- Cancellation must stop pending work promptly.
Definition of done:
- Concurrency limit is enforced in tests.
- Results remain deterministic where ordering matters.
- Module reports remain stable.
- Cancellation and timeout behavior are covered.
## Phase 11: Port real modules one at a time
Port modules after the framework, validators, and LLM client are in place.
Recommended order:
1. `grammar`
2. `glossary`
3. `homophones`
4. `spoken_word`
5. full default sequence
This order exercises formatting first, then domain-specific correction, then more subtle semantic cleanup.
For each module:
1. Port prompt builder without improving wording.
2. Define structured response type.
3. Define JSON schema if supported by the backend.
4. Convert structured response into framework proposals.
5. Attach intended validators.
6. Add fake-LLM tests.
7. Add fixture-level integration tests.
8. Add real LLM smoke test where practical.
9. Compare qualitative output against Python reference.
Definition of done for each module:
- Fake-LLM tests pass.
- Module-specific fixtures pass.
- Prompt/response diagnostics are written.
- Validator chain is explicit and tested.
- Module report includes applied and skipped changes.
## Phase 12: Full-pipeline compatibility testing
Run complete Go pipeline tests after all modules exist.
Tasks:
1. Run full default module sequence with fake LLM responses.
2. Run full default module sequence against small real transcript.
3. Run against at least one real D&D session transcript.
4. Compare output shape against Python reference.
5. Compare report shape against Python reference.
6. Inspect skipped changes manually.
7. Inspect diagnostics manually.
8. Verify downstream parser compatibility.
Definition of done:
- Go output can be consumed by the next pipeline stage.
- Report JSON can be consumed by existing tooling or by documented replacement tooling.
- Real run diagnostics are sufficient for debugging.
- Output quality is at least acceptable compared with Python.
## Phase 13: Subprocess integration testing
Test the compiled Go binary exactly as surrounding Go applications will call it.
Tasks:
1. Create integration tests that execute the binary as a subprocess.
2. Test `--output` path behavior.
3. Test stdout-only transcript output when `--output` is omitted.
4. Test stderr-only logs/errors.
5. Test `--report-json` path behavior.
6. Test nonzero exit on invalid input.
7. Test nonzero exit on LLM failure.
8. Test large transcript pipe behavior.
9. Test timeout/cancellation behavior from parent process.
Definition of done:
- No stdout/stderr deadlocks.
- Parent Go process can reliably distinguish success from failure by exit code.
- Parent Go process can parse output transcript and report files.
- Failed runs provide diagnostics paths.
## Phase 14: Run Python and Go side by side
For a transition period, keep both implementations available.
Tasks:
1. Give the Python binary a distinct name if needed, such as `audita-python`.
2. Give the Go binary the canonical `audita` name only after it is ready.
3. Allow the orchestrator to select implementation during transition.
4. Run the same real sessions through both implementations.
5. Compare reports and final transcripts.
6. Record known intentional differences.
7. Fix unintentional compatibility breaks.
Comparison criteria:
- both complete successfully;
- both preserve transcript JSON shape;
- both run the intended module sequence;
- both report applied/skipped changes;
- Go has cleaner subprocess behavior;
- Go diagnostics are at least as useful as Python diagnostics;
- Go transcript quality is acceptable for downstream use.
Definition of done:
- Go implementation has passed several real-session runs.
- Downstream applications can call Go Audita reliably.
- Any report/schema differences are documented.
- Python can be retired from the main path.
## Phase 15: Retire or archive the Python implementation
After the Go version becomes the default, preserve the Python implementation as historical reference unless there is a reason to remove it.
Tasks:
1. Mark Python implementation as archived or prototype.
2. Preserve useful fixtures and tests.
3. Preserve prompts and reference outputs.
4. Remove Python from production orchestration.
5. Update README to describe Go as the active implementation.
6. Update installation instructions.
7. Update operational docs.
Definition of done:
- Repository clearly identifies the Go implementation as active.
- Python code no longer participates in normal operation.
- Historical Python behavior remains available for reference if needed.
## Suggested milestone commits
A reasonable commit sequence for agent-assisted implementation:
1. Scaffold Go CLI and config loading.
2. Add transcript/glossary schemas and I/O.
3. Add normalization and normalization tests.
4. Add chunking and token estimation.
5. Add proposal model and replacement policies.
6. Add reports and diagnostics lifecycle.
7. Add pipeline runner with fake modules.
8. Add deterministic validators.
9. Add structured LLM client interface and fake client.
10. Add OpenAI-compatible LLM client.
11. Add bounded LLM scheduler.
12. Port grammar module.
13. Port glossary module.
14. Port homophones module.
15. Port spoken-word module.
16. Enable default full pipeline.
17. Add subprocess integration tests.
18. Update README and operational documentation.
Each milestone should leave the repository in a passing state.
## Guidance for LLM implementation agents
When using an LLM agent such as Codex to implement this rewrite, prefer small prompts tied to a single milestone.
Good agent instructions:
- specify the exact phase being implemented;
- name the files or packages to create;
- require tests for the new behavior;
- require `go test ./...` to pass;
- prohibit prompt wording changes unless the phase is explicitly about prompts;
- require preserving CLI compatibility;
- require preserving stdout/stderr discipline;
- require staging/committing only after tests pass, if the workflow permits commits.
Avoid broad prompts such as “rewrite Audita in Go.” They invite unnecessary redesign and make review difficult.
## Review checklist for each phase
Before accepting a phase implementation, check:
- Does it preserve the public contract?
- Does it introduce unnecessary redesign?
- Are errors typed and understandable?
- Are reports still machine-readable?
- Are diagnostics sufficient to debug failures?
- Are API keys redacted?
- Are stdout and stderr clean?
- Are tests deterministic?
- Does `go test ./...` pass?
- Is the code idiomatic Go rather than Python-shaped Go?
## Historical checkpoint note
Earlier "Phase 2 status" notes are intentionally superseded by
`Current Go rewrite status (Phase 3 foundation)` near the top of this document.
Use that section as the authoritative current-state snapshot.
## When to consider an HTTP API
An HTTP API should be deferred until the Go CLI is stable.
Consider HTTP later only if one or more of these become true:
- multiple callers need a shared long-running Audita worker;
- process startup time becomes a meaningful bottleneck;
- a job queue is needed;
- remote execution becomes necessary;
- centralized concurrency control is preferable to per-process control;
- operational monitoring of Audita as a service becomes valuable.
Even then, the HTTP API should wrap the same core Go pipeline used by the CLI. The CLI should remain the simplest and most reliable integration surface.
## Current implementation status
The Go rewrite is currently in a deterministic foundation stage.
Implemented:
- CLI command surface for `audita process`.
- Config/env/flag loading and validation.
- Transcript/glossary schema parsing and validation.
- Deterministic normalization with summary stats.
- Deterministic chunking with summary stats.
- Per-run diagnostics directory plus normalization/chunking artifacts.
- Minimal process report output (`--report-json` and run-dir `report.json`).
- Framework foundation packages for contracts and proposal preview/apply semantics.
Not yet implemented in runtime pipeline:
- Module runner orchestration.
- Real correction modules.
- Validator-chain execution.
- Structured LLM client integration.
- End-to-end transcript polishing behavior.
## Phase status (completed / partial / pending)
### Completed
- Phase 1: Go CLI skeleton.
- Phase 2: Schemas and file I/O.
- Phase 3: Deterministic normalization.
- Phase 4: Chunking and token estimation (deterministic implementation present).
- Phase 5: Proposal model and application semantics (foundation package implemented and tested).
### Partial
- Phase 6: Reports and diagnostics.
What is complete:
- run directory creation
- source/normalized/chunking artifacts
- report writing
- failure `error.log`
- retention mode parsing/validation
What is still incomplete vs original plan:
- redacted invocation/config metadata artifacts are not written yet
- module prompt/response diagnostics are not present (no module execution yet)
- `auto` retention semantics currently behave like `never` for successful runs
### Pending
- Phase 7+: runner orchestration, validators in runtime path, LLM client, bounded concurrency, real modules, full compatibility and side-by-side rollout phases.
## Deviations from original phase plan
1. Proposal foundation (`internal/framework/proposals`) and framework contracts (`internal/framework/contracts`) were implemented before full runner orchestration.
2. Chunking was integrated directly into the current CLI runtime path before module execution exists.
3. Reporting currently uses a process-level status report with `phase: "phase3-chunking"`, rather than module-level pipeline reports.
4. Work-dir retention `auto` is intentionally deferred and currently simplified.
## Next recommended implementation phase
Implement **Phase 7 (pipeline runner with fake modules)** next.
Concrete target for next phase:
1. Add a production runner package wired from `internal/cli/run.go`.
2. Resolve configured module keys into run specs using existing contracts helper.
3. Execute fake or deterministic in-repo modules sequentially over a working transcript.
4. Run proposal application through `internal/framework/proposals` for each stage.
5. Emit module-level applied/skipped results into report structures.
6. Preserve current stdout/stderr behavior and diagnostics retention behavior.
Rationale:
- This uses already-implemented deterministic pieces.
- It closes the biggest current gap: no actual module pipeline execution.
- It enables validator and LLM phases to be added incrementally afterward.
## Guidance for Codex-style prompts
When requesting the next implementation steps, use narrow prompts such as:
- "Implement runner orchestration using existing contracts and proposal apply package, no LLM calls yet."
- "Add module-level report structs and wire them into process report."
- "Keep CLI contract unchanged and ensure `go test ./...` passes."
Avoid broad prompts like "finish the rewrite"; they blur phase boundaries and make review harder.
## Compatibility reminders
Keep these constraints unchanged while implementing later phases:
- Preserve subprocess-safe stdout/stderr behavior.
- Preserve CLI-over-env precedence.
- Preserve deterministic safety-first proposal application semantics.
- Do not claim module or LLM behavior exists until it is actually in runtime path.