eric/audita
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 2 Wiki Activity

Update Go rewrite documentation status

Browse Source
This commit is contained in:
Eric Rakestraw
2026-05-11 08:49:26 -05:00
parent c3087aeda6
commit 73249b63d8
2 changed files with 260 additions and 1225 deletions
Download Patch File Download Diff File Expand all files Collapse all files

791
docs/architecture.md
View File

@@ -1,656 +1,219 @@
# Audita Go Architecture
## Purpose
## Scope and intent
This document describes:
- the current implemented Go architecture; and
- the intended final architecture for later rewrite phases.
Audita is a framework-first transcript polishing application. It takes a source transcript, a glossary, and runtime configuration; runs an ordered sequence of correction modules; and writes a corrected transcript, a structured run report, and diagnostics artifacts.
Status labels are explicit so future engineers and LLM agents do not assume unimplemented behavior exists.
The Go implementation should preserve the public behavior and safety posture of the existing Python implementation while using idiomatic Go internals. Treat the existing Python implementation, README, architecture notes, tests, and representative outputs as the behavioral specification during the rewrite.
## Current implementation status
Implemented today:
- Go CLI entrypoint and `audita process` wiring.
- Config defaults, env loading, CLI override precedence, and validation.
- Transcript and glossary parsing/validation.
- Deterministic transcript normalization.
- Deterministic token estimation and transcript chunking.
- Per-run diagnostics directory creation plus basic artifacts.
- Minimal process report JSON output.
- Framework foundation packages for contracts and proposal application.
## Design goals
Not implemented in CLI runtime path today:
- Real module execution pipeline (`glossary`, `homophones`, `spoken_word`, `grammar`).
- Structured LLM proposal generation.
- Validator chain execution.
- Pipeline runner orchestration.
- Production LLM scheduler behavior.
- End-to-end transcript polishing beyond normalization.
1. Preserve the existing batch CLI contract.
2. Preserve the existing module pipeline semantics.
3. Preserve deterministic safety checks and skipped-change reporting.
4. Preserve diagnostics and report artifacts as first-class outputs.
5. Make subprocess execution reliable from other Go applications.
6. Keep LLM provider integration small, explicit, and OpenAI-compatible.
7. Keep the design modular enough to add new transcript correction modules and validators.
8. Prefer simple, auditable Go code over agent frameworks or heavy runtime abstractions.
## Non-goals for the initial Go rewrite
The initial Go implementation should not attempt to redesign Audita. In particular, it should not initially:
- replace the CLI-first batch model with an HTTP service;
- redesign the transcript or report schema;
- improve prompt wording during the port;
- change the default module order;
- chase exact nondeterministic LLM output parity with the Python implementation;
- introduce a general-purpose workflow or agent framework;
- require downstream applications to change their integration model.
Later versions may add an HTTP API, new module types, richer report formats, or prompt improvements after the Go CLI is behaviorally stable.
## Public CLI contract
The primary command is:
```sh
audita process transcript.json --glossary glossary.yaml --output corrected.json
```
Optional report output:
```sh
audita process transcript.json --glossary glossary.yaml --output corrected.json --report-json report.json
```
Custom module sequence:
```sh
audita process transcript.json --glossary glossary.yaml --modules grammar --output corrected.json
```
Expected stream behavior:
- If `--output` is provided, corrected transcript JSON is written to that file.
- If `--output` is omitted, corrected transcript JSON is written to stdout.
- Progress logs and human-readable errors are written to stderr.
- `--report-json` writes a machine-readable report to the specified file.
- Report JSON is never mixed into stdout.
- LLM prompt/response diagnostics are written under the per-run work directory, not to stdout.
- Exit code `0` means the run completed successfully.
- Nonzero exit codes mean the run failed; failed runs preserve diagnostics.
This stdout/stderr discipline is a core requirement because Audita is expected to be called as a subprocess by other Go applications.
## Configuration model
Configuration sources should be applied in this order:
1. built-in defaults;
2. environment variables;
3. CLI flags.
CLI flags override environment variables. Environment variables override defaults.
The initial Go implementation should preserve the existing configuration surface where practical, including:
- module sequence;
- primary LLM API key, base URL, model, timeout, retry count, and concurrency;
- validation LLM API key, base URL, model, timeout, retry count, and concurrency;
- proposal-stage section token bounds;
- validation-stage prompt token bounds;
- module-specific confidence thresholds;
- normalization parameters;
- work directory location and retention policy.
Recommended Go shape:
```text
internal/core/config
Config
LLMConfig
NormalizationConfig
ModuleConfig
WorkDirRetention
LoadFromEnv
ApplyCLIOverrides
Validate
EffectiveValidationLLMConfig
```
API credentials must be redacted from reports, logs, diagnostics metadata, and error details.
## High-level runtime flow
`audita process` should execute the following flow:
1. Parse CLI arguments.
2. Load and validate configuration.
3. Load and validate transcript input.
4. Load and validate glossary input.
5. Create a per-run diagnostics directory.
6. Persist redacted invocation/config metadata.
7. Persist the source transcript artifact.
8. Normalize transcript deterministically.
9. Persist normalized transcript and normalization summary.
10. Resolve configured module sequence into module run instances.
11. Execute modules sequentially over a mutable working transcript.
12. For each module instance:
- chunk the current working transcript into contiguous token-bounded sections;
- generate structured correction proposals using the module prompt;
- enrich proposals with module/run metadata;
- run validator chain;
- apply approved proposals according to replacement policy;
- report applied changes and skipped changes.
13. Sort final transcript chronologically.
14. Write corrected transcript to output file or stdout.
15. Write `report.json` if requested and always write authoritative report into retained diagnostics.
16. Apply work-dir retention policy.
17. On failure, preserve diagnostics, write failed report, write `error.log`, print concise stderr summary, and exit nonzero.
## Suggested Go package layout
## Actual Go package layout
```text
cmd/audita/
main.go
internal/cli/
run.go
internal/core/config/
config.go
env.go
flags.go
redaction.go
validation.go
internal/core/schema/
transcript.go
glossary.go
report.go
errors.go
internal/core/io/
transcript.go
glossary.go
report.go
json.go
yaml.go
files.go
internal/core/normalization/
normalize.go
summary.go
tokens.go
internal/core/chunking/
tokens.go
sections.go
batches.go
summary.go
tokens.go
internal/core/diagnostics/
run_dir.go
artifacts.go
redaction.go
retention.go
internal/core/errors/
errors.go
exit_codes.go
internal/core/reporting/
report.go
internal/framework/runner/
runner.go
context.go
result.go
internal/framework/contracts/
contracts.go
internal/framework/proposals/
proposal.go
apply.go
policy.go
internal/framework/llm/
client.go
openai_compatible.go
structured.go
retry.go
scheduler.go
internal/framework/modules/
module.go
registry.go
sequence.go
internal/modules/glossary/
module.go
prompt.go
response.go
internal/modules/homophones/
module.go
prompt.go
response.go
internal/modules/spokenword/
module.go
prompt.go
response.go
internal/modules/grammar/
module.go
prompt.go
response.go
internal/validators/
validator.go
result.go
internal/validators/deterministic/
confidence.go
original_text.go
protected_terms.go
non_empty.go
internal/validators/llm/
validators.go
prompts.go
responses.go
internal/validators/protection/
glossary_terms.go
matching.go
internal/testutil/
fixtures.go
fakellm.go
golden.go
preview.go
apply.go
```
This package layout is a starting point, not a rule. Prefer fewer packages if the implementation becomes fragmented. Keep package boundaries aligned with stable domain concepts: config, schema, normalization, chunking, diagnostics, LLM, proposals, modules, validators, and runner.
## Current CLI behavior
Primary command:
## Core data contracts
### Transcript
Audita should accept either:
1. a bare JSON array of segments; or
2. an object containing a `segments` array.
Segment fields:
```text
id integer
speaker string
start number
end number
text string
categories optional array of strings
```sh
audita process <transcript.json> --glossary <glossary.yaml> [flags]
```
Input may contain source IDs. The normalized internal transcript should use strict sequential IDs starting at `1`.
Final output should use the same normalized segment shape and should be sorted chronologically.
### Glossary
The glossary model should support the existing glossary semantics used by the Python implementation, including domain-specific terms and protected vocabulary used by validators. The Go implementation should preserve the accepted YAML shape rather than inventing a new one during the rewrite.
### Correction proposal
Modules emit structured proposals with the following logical fields:
```text
id target segment ID
original_text span expected to exist in the target segment
corrected_text replacement text
confidence number between 0.0 and 1.0
```
The framework enriches proposals with execution metadata:
```text
proposal_index
module_key
module_instance
section_id or batch_id
validator decisions
application status
skip reason, if any
```
### Validation decision
Each validator must return exactly one decision for each candidate proposal index it was asked to evaluate.
Missing, duplicate, or unknown proposal indexes are framework errors. The framework should not silently ignore malformed validator output.
### Run report
The run report should be machine-readable JSON. It should describe:
- run status;
- start/end timestamps or elapsed duration;
- effective module sequence;
- normalization summary;
- module reports;
- applied changes;
- skipped changes;
- errors, when present;
- diagnostics directory, when retained.
The report should be stable enough for downstream tooling to consume.
## Pipeline model
Audita uses a staged transform model over a mutable working transcript.
```text
source transcript
-> deterministic normalization
-> working transcript
-> module: glossary_1
-> module: homophones
-> module: glossary_2
-> module: spoken_word
-> module: grammar
-> final transcript
```
Module order is architecturally significant. Each module sees the transcript produced by all previous modules.
Default logical module sequence:
```text
glossary, homophones, glossary, spoken_word, grammar
```
Default resolved module instance names:
```text
glossary_1, homophones, glossary_2, spoken_word, grammar
```
Repeated logical module keys should be auto-numbered in reports and diagnostics.
## Module interface
A module should be a small object with stable metadata and proposal behavior.
Conceptual interface:
```go
type TranscriptModule interface {
Key() string
ReplacementPolicy() proposals.ReplacementPolicy
Validators() []validators.Validator
Propose(ctx context.Context, req ProposalRequest) ([]proposals.CorrectionProposal, error)
}
```
A module is responsible for:
- building its proposal prompt;
- requesting structured proposal output from the LLM client;
- mapping structured output into framework proposals;
- attaching its validator chain.
A module should not:
- own global process state;
- write final outputs;
- decide work-dir retention;
- apply proposals directly;
- bypass framework validators;
- perform unbounded concurrency.
## Module responsibilities
### `glossary`
Proposes glossary-supported acoustic corrections and domain-specific term corrections. This module should be conservative and should rely heavily on glossary evidence and protected-vocabulary validation.
### `homophones`
Proposes conservative homophone and mistranscription corrections. This module should avoid stylistic editing and should focus on likely transcription errors.
### `spoken_word`
Proposes conservative dysfluency cleanup that does not affect substantive meaning. This module should be strongly guarded by semantic validators because it is easy to over-edit spoken language.
### `grammar`
Proposes punctuation, capitalization, and spacing cleanup only. This module should not rewrite content for style or clarity beyond formatting and readability corrections.
## Replacement policies
Proposal application should be centralized in the framework.
Supported policies:
```text
require_unique original_text must match exactly once in the target segment
replace_all replace every occurrence of original_text in the target segment
```
If a proposal cannot be applied safely, it should be skipped and reported instead of crashing the run. Examples:
- target segment no longer exists;
- `original_text` is missing;
- `require_unique` found zero matches;
- `require_unique` found multiple matches;
- corrected text is empty after validation;
- proposal became stale after earlier module edits.
## Validator model
Validators filter candidate proposals before application.
Validator categories:
1. deterministic validators;
2. LLM-backed validators.
Deterministic validators should run before LLM validators whenever possible because they are cheaper, faster, and more predictable.
Examples of deterministic validators:
- confidence threshold;
- original text presence;
- non-empty correction;
- identical text rejection;
- protected glossary term checks.
Examples of LLM-backed validators:
- spoken-form plausibility;
- meaning reversal detection;
- editorial review;
- grammar review;
- spoken-word cleanup review.
A validator should return structured results only. Human-readable explanations may be included in reports, but validation logic should consume typed decisions.
## LLM integration
Audita should use a small OpenAI-compatible structured-output client.
The client should support:
- configurable base URL;
- configurable model;
- optional API key;
- per-request timeout;
- retry budget;
- structured JSON response schema;
- raw prompt/response diagnostic capture;
- context cancellation;
- clear error wrapping.
Conceptual interface:
```go
type StructuredLLMClient interface {
CompleteStructured(ctx context.Context, req StructuredRequest, out any) error
}
```
The LLM client should not know about Audita modules, transcript schemas, or validators. It should only know how to submit an OpenAI-compatible request and decode a structured response.
Proposal and validation phases may use different effective LLM settings. Validation settings inherit from proposal settings unless explicitly overridden.
## Structured output policy
For every LLM call, Audita should prefer strict structured JSON output over free-form text parsing.
Each module and LLM validator should define:
- request payload type;
- response payload type;
- JSON schema, where the backend supports schema-constrained output;
- response validation rules;
- retry behavior for malformed or incomplete responses.
Malformed structured responses should become typed errors that include diagnostic references but do not leak API keys or excessive prompt text to stderr.
## Concurrency model
Module stages remain sequential.
Within a module, bounded concurrency is allowed for:
- proposal generation across transcript sections;
- LLM validation batches;
- adjacent independent LLM validators that evaluate the same candidate set;
- prompt/response artifact writes.
All backend LLM calls should pass through a scheduler or semaphore that enforces configured concurrency limits.
Use `context.Context` for cancellation and timeout propagation.
Avoid unbounded goroutine creation. Every concurrent unit should be attached to the current run context and should return errors through a controlled mechanism such as `errgroup`.
## Diagnostics model
A retained run directory should use a predictable structure similar to:
```text
runs/
2026-05-10T210000Z-<short-id>/
invocation.json
source-transcript.json
normalized-transcript.json
normalization-summary.json
modules/
glossary_1/
section-001.prompt.json
section-001.response.json
homophones/
glossary_2/
spoken_word/
grammar/
report.json
error.log
```
Retention modes:
```text
always keep every run directory
never remove successful run directories
auto keep failed runs and successful runs with final skipped corrections
```
Failed runs are always retained.
Diagnostics should be useful for debugging LLM behavior, proposal generation, validation decisions, and replacement failures.
## Error handling model
Use typed errors for expected failure classes:
- configuration errors;
- input validation errors;
- normalization errors;
- module resolution errors;
- LLM request errors;
- structured response errors;
- validator contract errors;
- proposal application errors;
- output write errors.
A pipeline failure should preserve partial progress when possible:
- partial working transcript;
- completed module reports;
- applied changes so far;
- skipped changes so far;
- failed module information;
- error details;
- diagnostics references.
The CLI should print concise stderr output and point to retained diagnostics rather than dumping large prompts or stack traces directly to the terminal.
## Testing strategy
The Go implementation should use three categories of tests.
### Deterministic unit tests
Cover:
- config loading and precedence;
- transcript parsing;
- glossary parsing;
- normalization;
- chunking;
- proposal preview/application;
- replacement policies;
- validator contract enforcement;
- report serialization;
- diagnostics retention decisions.
### Fake-LLM integration tests
Use a fake structured LLM client to test:
- module proposal flow;
- validator flow;
- full pipeline execution;
- repeated module instance names;
- skipped-change reporting;
- malformed LLM responses;
- retry behavior;
- partial failure reporting.
### Subprocess tests
Invoke the compiled `audita` binary from tests and verify:
- stdout contains only corrected transcript JSON when `--output` is omitted;
- stdout is clean when `--output` is provided;
- stderr contains logs/errors only;
- `--report-json` writes a valid report;
- failed runs exit nonzero;
- failed runs retain diagnostics;
- large inputs do not deadlock stdout/stderr pipes;
- cancellation and timeout behavior are reliable.
## Extension points
### Add a module
1. Create a package under `internal/modules/<name>`.
2. Implement the module interface.
3. Define prompt builder and structured response type.
4. Define or reuse validators.
5. Register the logical module key in the module registry.
6. Add fake-LLM tests.
7. Add a fixture-level integration test.
### Add a validator
1. Implement the validator interface.
2. Define typed result/decision structures.
3. For LLM validators, define prompt and structured response schema.
4. Enforce one decision per candidate proposal index.
5. Add tests for approved, rejected, malformed, and missing-decision cases.
6. Attach the validator to module chains intentionally.
### Add an LLM backend
1. Implement `StructuredLLMClient`.
2. Preserve request timeout and context cancellation behavior.
3. Preserve raw prompt/response diagnostic capture.
4. Preserve structured response validation and retry semantics.
5. Add fake or local integration tests.
## Implementation posture
The Go version should be boring infrastructure:
- explicit structs;
- explicit validation;
- small interfaces;
- clear package boundaries;
- no hidden global state;
- deterministic file outputs;
- stable subprocess behavior;
- diagnostics-first failures;
- structured LLM calls rather than text scraping.
The primary measure of success is not that the Go code resembles the Python code. The measure of success is that the Go binary can replace the Python CLI in the surrounding transcript pipeline with fewer operational surprises.
Current runtime flow (`internal/cli/run.go`):
1. Load config from env.
2. Parse flags and apply CLI overrides.
3. Validate transcript positional argument and required `--glossary`.
4. Create per-run diagnostics directory.
5. Read transcript and glossary files.
6. Parse/validate transcript and glossary.
7. Write source transcript artifacts.
8. Normalize transcript.
9. Write normalized transcript and normalization summary artifacts.
10. Chunk normalized transcript and compute chunk summaries.
11. Write chunking summary artifact.
12. Output normalized transcript to `--output` file or stdout.
13. Build process report (`phase` currently set to `phase3-chunking`).
14. Optionally write `--report-json`; always write run-dir `report.json`.
15. Apply work-dir retention.
Important behavior details:
- Glossary is validated but not yet used for correction logic.
- No module execution occurs despite `--modules` config support.
- No LLM calls occur.
- Success path is generally quiet on stderr.
## Implemented data contracts
### Transcript input
Accepted top-level forms:
- bare JSON array of segments
- object with `segments` array
Source segment contract:
- `id` optional integer
- `speaker` non-empty string
- `start` finite non-negative number
- `end` finite non-negative number with `end >= start`
- `text` non-empty string
- `categories` optional array of non-empty strings
Additional checks:
- duplicate explicit source IDs are rejected.
### Transcript output
Current output uses `schema.TranscriptToJSON` and is a bare JSON array of normalized segments:
- `id`, `speaker`, `start`, `end`, `text`, optional `categories`.
### Glossary input
YAML with `glossary` entries. Required fields per entry:
- `name`, `category`, `summary`
Optional:
- `aliases`, `plural`
## Implemented config/env/flag behavior
Precedence:
1. defaults (`config.Default()`)
2. environment (`config.LoadFromEnv()`)
3. CLI flags (`ApplyCLIOverrides`)
Implemented config surfaces include:
- module list
- primary and validation LLM settings
- section token controls and target sections
- confidence thresholds
- normalization controls
- work-dir and retention mode
Current caveat:
- LLM/module-related settings are mostly infrastructure-only today; runtime path does not execute LLM or modules.
## Implemented normalization behavior
Normalization (`internal/core/normalization`) currently:
- sorts by segment start time;
- merges adjacent same-speaker segments when constraints pass;
- uses gap-based joiners:
- gap `< ellipsis_gap` -> single space join
- gap `>= ellipsis_gap` -> `... ` join
- enforces merged duration and token-limit constraints;
- reassigns output IDs sequentially from `1`;
- returns `NormalizationSummary` with merge and skip counters.
Note: merged categories are concatenated (not deduplicated).
## Implemented chunking behavior
Chunking (`internal/core/chunking`) currently provides:
- deterministic heuristic token estimation;
- contiguous sectioning with section metadata;
- max/min section token validation;
- optional `target_sections` handling with target-aware merge/split logic;
- summary and detailed summary generation.
Current behavior details:
- if a single segment exceeds max tokens, it is emitted as its own section (not hard-failed);
- section balancing is deterministic but heuristic.
## Implemented proposal/replacement infrastructure
`internal/framework/proposals` provides deterministic foundation logic:
- `CorrectionProposal` and `EnrichedCorrectionProposal` models;
- replacement policies: `require_unique`, `replace_all`;
- safe preview (`PreviewProposalForSegment`) with stable skip reasons;
- deterministic apply (`ApplyProposals`) in ascending `proposal_index` order;
- applied/skipped change records suitable for reporting.
`internal/framework/contracts` provides interfaces and run-spec metadata scaffolding, including deterministic repeated module instance naming (`ResolveModuleRunSpecs`).
These are framework primitives only; they are not yet wired into CLI runtime module execution.
## Reports and diagnostics (implemented)
Current per-run artifacts include:
- `source-transcript.json`
- `source-transcript-parsed.json`
- `normalized-transcript.json`
- `normalization-summary.json`
- `chunking-summary.json`
- `report.json`
- `error.log` on failure
`--report-json` writes a separate report file when requested.
Retention modes implemented in `ApplyRetention`:
- `always`: keep successful runs
- `never`: remove successful runs
- `auto`: currently same as `never` for successful runs
- failed runs are always retained
## Intended final architecture (not yet implemented)
The intended end-state still matches the rewrite plan:
- sequential module pipeline over a mutable working transcript
- real module implementations (`glossary`, `homophones`, `spoken_word`, `grammar`)
- structured LLM proposal generation
- deterministic and LLM validators
- validator cardinality enforcement in pipeline execution
- proposal application integrated per module stage
- richer run reporting with module-level applied/skipped changes
Until those phases are implemented, documentation and external descriptions should treat the current Go CLI as deterministic preprocessing/reporting infrastructure, not a full LLM transcript polisher.

694
docs/rewrite-notes.md
View File

@@ -1,613 +1,85 @@
# Audita Go Rewrite Notes
## Rewrite strategy
The Go rewrite should be compatibility-first. The existing Python implementation should be treated as the executable specification for public behavior, pipeline semantics, diagnostics, and safety rules.
The rewrite should not begin as a redesign. The first production-capable Go version should be able to stand in for the Python CLI in the surrounding transcript pipeline.
The initial goal is:
```text
same public contract
same pipeline semantics
same safety posture
same diagnostic philosophy
idiomatic Go implementation
reliable subprocess behavior
```
Exact LLM output parity is not required because LLM calls are nondeterministic and may vary by backend, prompt formatting, model, or structured-output implementation. Deterministic framework behavior should be held to a much stricter compatibility standard.
## Core principles
1. Preserve behavior before improving behavior.
2. Port deterministic layers before LLM layers.
3. Test with fake LLMs before testing with real LLMs.
4. Keep module stages sequential until correctness is established.
5. Add bounded concurrency only after the sequential implementation is correct.
6. Keep the CLI contract stable for downstream callers.
7. Keep stdout/stderr behavior clean and predictable.
8. Treat diagnostics and reports as part of the product, not as afterthoughts.
9. Avoid prompt improvements during the port.
10. Prefer explicit Go structs and validation over reflection-heavy abstractions.
## Compatibility targets
The Go implementation should preserve the following public behaviors where practical:
- `audita process` command shape;
- transcript input support for bare segment arrays and `{ "segments": [...] }` objects;
- glossary YAML support;
- default module sequence;
- repeated module instance naming;
- environment variable and CLI flag configuration concepts;
- CLI-over-env precedence;
- output file behavior;
- stdout behavior when `--output` is omitted;
- stderr logging behavior;
- separate `--report-json` machine-readable report behavior;
- work-dir diagnostics behavior;
- retention modes `auto`, `always`, and `never`;
- failed-run diagnostics preservation;
- skipped-change reporting instead of crashing on stale or unsafe proposal application.
## Current Go rewrite status (Phase 3 foundation)
The Go rewrite now includes deterministic foundation pieces needed before real module/LLM behavior:
- typed transcript parsing and validation (bare array or `{ "segments": [...] }`);
- typed glossary YAML parsing and validation;
- deterministic transcript normalization and normalization summary;
- deterministic token estimation;
- contiguous transcript chunking with section metadata;
- correction proposal data models and enriched proposal metadata;
- replacement policy handling (`require_unique`, `replace_all`);
- safe per-segment proposal preview;
- deterministic transcript proposal application in stable proposal-index order;
- applied and skipped change records with stable skip reasons;
- report/diagnostic-oriented chunking summary and chunk metadata hooks;
- minimal framework contracts/interfaces for future modules, validators, and structured LLM clients;
- test-only fake-module composition tests proving chunking and proposal application can work together.
The Go rewrite does **not** yet include end-to-end transcript polishing behavior:
- no real module execution pipeline yet;
- no real `glossary`, `homophones`, `spoken_word`, or `grammar` module implementations;
- no LLM proposal generation;
- no validator-chain execution;
- no concrete structured LLM client implementation;
- no final LLM-driven transcript polishing behavior comparable to Python production runs.
This means the Go binary should not be described as a complete LLM transcript polisher yet.
## Developer quick note
Use these commands while developing deterministic rewrite phases:
```sh
go test ./...
go run ./cmd/audita process internal/cli/testdata/tiny_transcript.json --glossary internal/cli/testdata/tiny_glossary.yaml --output /tmp/audita-go-output.json --report-json /tmp/audita-go-report.json
```
The second command is a representative CLI wiring check only; it is not a real LLM polishing run in the current phase.
## Phase 0: Freeze the Python implementation as the reference
Before writing substantial Go code, preserve the behavior of the current Python implementation.
Tasks:
1. Create or identify a stable branch/tag representing the Python reference implementation.
2. Preserve the existing README and architecture notes.
3. Preserve the Python regression suite.
4. Collect representative transcript/glossary fixtures.
5. Capture reference outputs for deterministic behaviors.
6. Capture several real end-to-end run artifacts for qualitative comparison.
Representative fixtures should include:
- tiny valid transcript;
- transcript with multiple speakers;
- transcript accepted as a bare array;
- transcript accepted as an object with `segments`;
- glossary-supported correction case;
- homophone/mistranscription case;
- spoken-word dysfluency case;
- grammar/punctuation/capitalization case;
- duplicate `original_text` case;
- missing `original_text` case;
- skipped-change case;
- malformed input case;
- failed LLM response case.
Definition of done:
- A developer or LLM agent can run the Python test suite.
- Reference fixtures are committed.
- Reference reports or golden artifacts exist for deterministic comparisons.
- The Python implementation can be used to answer disputed behavior questions during the rewrite.
## Phase 1: Create the Go CLI skeleton
Build the smallest Go binary that preserves the outer command shape.
Tasks:
1. Initialize Go module.
2. Add `cmd/audita/main.go`.
3. Implement `audita process` command.
4. Parse core flags:
- transcript path;
- `--glossary`;
- `--output`;
- `--report-json`;
- `--modules`;
- LLM config flags;
- normalization flags;
- work-dir flags.
5. Implement config defaults.
6. Implement environment variable loading.
7. Implement CLI-over-env precedence.
8. Implement redaction for sensitive config values.
9. Load input files but initially write transcript back unchanged.
Definition of done:
- `go test ./...` passes.
- `go run ./cmd/audita process transcript.json --glossary glossary.yaml --output corrected.json` succeeds for a minimal fixture.
- The output transcript is valid JSON.
- With `--output`, stdout is empty except for intentional machine output, preferably empty.
- Logs go to stderr.
- Invalid flags produce a clean error and nonzero exit.
## Phase 2: Port schemas and file I/O
Implement typed transcript, glossary, and report data structures.
Tasks:
1. Define transcript segment structs.
2. Support input as bare segment array.
3. Support input as object with `segments` array.
4. Preserve optional `categories`.
5. Validate required fields.
6. Validate basic timing shape.
7. Define glossary structs matching the existing YAML format.
8. Implement glossary parsing.
9. Implement report structs at least sufficient for early phases.
10. Implement JSON/YAML read/write helpers.
Definition of done:
- Valid Python-era fixtures parse successfully.
- Invalid fixtures fail with clear errors.
- Transcript round-trips through Go without accidental data loss.
- Glossary fixtures parse successfully.
- Report JSON can be written and parsed by tests.
## Phase 3: Port deterministic normalization
Port normalization before any LLM work.
Tasks:
1. Implement same-speaker segment merging.
2. Implement maximum merge gap.
3. Implement ellipsis gap insertion.
4. Implement maximum merged segment duration.
5. Implement maximum merged segment token budget.
6. Reassign strict sequential segment IDs starting at `1`.
7. Produce a normalization summary.
8. Preserve source transcript and normalized transcript diagnostics.
Definition of done:
- Normalization tests pass against golden fixtures.
- Segment IDs are deterministic.
- Chronological ordering is deterministic.
- Normalization summary is included in the run report.
- Normalization artifacts are written in the run directory.
## Phase 4: Port chunking and token estimation
Implement the section-building logic used by proposal generation and validator batching.
Tasks:
1. Implement approximate token estimator.
2. Implement contiguous transcript sections.
3. Implement minimum and maximum section token settings.
4. Implement exact target section count behavior if supported by the Python implementation.
5. Implement validation prompt batching by token limit.
6. Add tests for edge cases.
Edge cases:
- very small transcript;
- single very large segment;
- many short segments;
- exact target section count impossible;
- min/max token bounds conflict;
- speaker boundaries near section boundaries.
Definition of done:
- Chunking is deterministic.
- Sections preserve transcript order.
- Sections do not drop or duplicate segments.
- Token-boundary behavior is covered by tests.
## Phase 5: Port proposal model and application semantics
Implement correction proposals and safe replacement behavior.
Tasks:
1. Define correction proposal struct.
2. Define enriched proposal metadata.
3. Define replacement policies:
- `require_unique`;
- `replace_all`.
4. Implement proposal preview.
5. Implement proposal application.
6. Implement skipped-change reporting.
7. Handle stale proposals safely.
8. Add tests for every skip reason.
Skip cases should include:
- missing target segment;
- missing `original_text`;
- multiple matches under `require_unique`;
- empty corrected text;
- identical original/corrected text after normalization;
- proposal invalidated by earlier edits.
Definition of done:
- Proposal application never panics on malformed proposal input.
- Unsafe proposals are skipped and reported.
- Applied changes and skipped changes are serializable in the report.
- Tests cover both replacement policies.
## Phase 6: Implement reports and diagnostics
Build diagnostics and reporting before real LLM calls so failures are inspectable from the beginning.
Tasks:
1. Create per-run directory under configured work dir.
2. Write redacted invocation/config metadata.
3. Write source transcript artifact.
4. Write normalized transcript artifact.
5. Write normalization summary artifact.
6. Write module prompt/response artifacts once module execution exists.
7. Write authoritative `report.json` into retained run directories.
8. Implement `--report-json` output path.
9. Implement `error.log` on failure.
10. Implement retention modes.
Retention behavior:
```text
always keep all run directories
never remove successful run directories
auto keep failed runs and successful runs with skipped corrections
```
Failed runs are always retained.
Definition of done:
- Clean successful runs obey retention policy.
- Successful runs with skipped corrections are retained under `auto`.
- Failed runs are always retained.
- Failed runs write `report.json` and `error.log` when possible.
- CLI stderr points to retained diagnostics on failure.
## Phase 7: Implement the pipeline runner with fake modules
Build the orchestration engine before porting real correction modules.
Tasks:
1. Define module interface.
2. Define validator interface.
3. Define runner context.
4. Define module run report model.
5. Implement module sequence resolution.
6. Implement repeated module instance naming.
7. Implement sequential module execution.
8. Implement fake module for tests.
9. Implement fake validator for tests.
10. Wire proposal application into runner.
Definition of done:
- A fake module can propose a correction and the runner applies it.
- A fake validator can reject a correction and the runner reports it.
- Multiple modules run in configured order.
- Repeated module keys are named deterministically.
- A module failure produces a partial failed report.
## Phase 8: Port deterministic validators
Port cheap, deterministic validation before LLM-backed validation.
Tasks:
1. Implement confidence threshold validator.
2. Implement original-text presence validator.
3. Implement non-empty correction validator.
4. Implement identical-text rejection.
5. Implement protected vocabulary logic from glossary.
6. Enforce validator result cardinality.
7. Add tests for malformed validator results.
Definition of done:
- Deterministic validators run before LLM validators.
- Each validator returns exactly one decision per input proposal.
- Missing, duplicate, or unknown proposal indexes produce framework errors.
- Rejected proposals are reported with reasons.
## Phase 9: Implement structured LLM client
Add the OpenAI-compatible structured-output client after the deterministic framework is stable.
Tasks:
1. Define `StructuredLLMClient` interface.
2. Define structured request type.
3. Implement OpenAI-compatible chat completions client.
4. Support configurable base URL, model, API key, timeout, and retries.
5. Support optional API key for self-hosted endpoints.
6. Implement structured JSON response parsing.
7. Implement response validation.
8. Implement retry behavior for malformed structured output.
9. Write raw prompt/response diagnostics.
10. Support separate effective validation LLM settings.
Definition of done:
- Fake LLM tests still pass.
- Real LLM smoke test can run against a configured endpoint.
- Malformed LLM responses are retried or reported cleanly.
- API keys are never logged or written to diagnostics.
- Context timeout/cancellation works.
## Phase 10: Add bounded concurrency
Add concurrency only after sequential correctness is established.
Tasks:
1. Add LLM call scheduler/semaphore.
2. Add proposal-generation concurrency across transcript sections.
3. Add validator batching concurrency where useful.
4. Ensure all goroutines are attached to run context.
5. Use `errgroup` or equivalent controlled error propagation.
6. Add tests for concurrency limits using fake LLM instrumentation.
Rules:
- Module stages remain sequential.
- LLM backend calls must respect configured concurrency.
- No unbounded goroutine creation.
- Cancellation must stop pending work promptly.
Definition of done:
- Concurrency limit is enforced in tests.
- Results remain deterministic where ordering matters.
- Module reports remain stable.
- Cancellation and timeout behavior are covered.
## Phase 11: Port real modules one at a time
Port modules after the framework, validators, and LLM client are in place.
Recommended order:
1. `grammar`
2. `glossary`
3. `homophones`
4. `spoken_word`
5. full default sequence
This order exercises formatting first, then domain-specific correction, then more subtle semantic cleanup.
For each module:
1. Port prompt builder without improving wording.
2. Define structured response type.
3. Define JSON schema if supported by the backend.
4. Convert structured response into framework proposals.
5. Attach intended validators.
6. Add fake-LLM tests.
7. Add fixture-level integration tests.
8. Add real LLM smoke test where practical.
9. Compare qualitative output against Python reference.
Definition of done for each module:
- Fake-LLM tests pass.
- Module-specific fixtures pass.
- Prompt/response diagnostics are written.
- Validator chain is explicit and tested.
- Module report includes applied and skipped changes.
## Phase 12: Full-pipeline compatibility testing
Run complete Go pipeline tests after all modules exist.
Tasks:
1. Run full default module sequence with fake LLM responses.
2. Run full default module sequence against small real transcript.
3. Run against at least one real D&D session transcript.
4. Compare output shape against Python reference.
5. Compare report shape against Python reference.
6. Inspect skipped changes manually.
7. Inspect diagnostics manually.
8. Verify downstream parser compatibility.
Definition of done:
- Go output can be consumed by the next pipeline stage.
- Report JSON can be consumed by existing tooling or by documented replacement tooling.
- Real run diagnostics are sufficient for debugging.
- Output quality is at least acceptable compared with Python.
## Phase 13: Subprocess integration testing
Test the compiled Go binary exactly as surrounding Go applications will call it.
Tasks:
1. Create integration tests that execute the binary as a subprocess.
2. Test `--output` path behavior.
3. Test stdout-only transcript output when `--output` is omitted.
4. Test stderr-only logs/errors.
5. Test `--report-json` path behavior.
6. Test nonzero exit on invalid input.
7. Test nonzero exit on LLM failure.
8. Test large transcript pipe behavior.
9. Test timeout/cancellation behavior from parent process.
Definition of done:
- No stdout/stderr deadlocks.
- Parent Go process can reliably distinguish success from failure by exit code.
- Parent Go process can parse output transcript and report files.
- Failed runs provide diagnostics paths.
## Phase 14: Run Python and Go side by side
For a transition period, keep both implementations available.
Tasks:
1. Give the Python binary a distinct name if needed, such as `audita-python`.
2. Give the Go binary the canonical `audita` name only after it is ready.
3. Allow the orchestrator to select implementation during transition.
4. Run the same real sessions through both implementations.
5. Compare reports and final transcripts.
6. Record known intentional differences.
7. Fix unintentional compatibility breaks.
Comparison criteria:
- both complete successfully;
- both preserve transcript JSON shape;
- both run the intended module sequence;
- both report applied/skipped changes;
- Go has cleaner subprocess behavior;
- Go diagnostics are at least as useful as Python diagnostics;
- Go transcript quality is acceptable for downstream use.
Definition of done:
- Go implementation has passed several real-session runs.
- Downstream applications can call Go Audita reliably.
- Any report/schema differences are documented.
- Python can be retired from the main path.
## Phase 15: Retire or archive the Python implementation
After the Go version becomes the default, preserve the Python implementation as historical reference unless there is a reason to remove it.
Tasks:
1. Mark Python implementation as archived or prototype.
2. Preserve useful fixtures and tests.
3. Preserve prompts and reference outputs.
4. Remove Python from production orchestration.
5. Update README to describe Go as the active implementation.
6. Update installation instructions.
7. Update operational docs.
Definition of done:
- Repository clearly identifies the Go implementation as active.
- Python code no longer participates in normal operation.
- Historical Python behavior remains available for reference if needed.
## Suggested milestone commits
A reasonable commit sequence for agent-assisted implementation:
1. Scaffold Go CLI and config loading.
2. Add transcript/glossary schemas and I/O.
3. Add normalization and normalization tests.
4. Add chunking and token estimation.
5. Add proposal model and replacement policies.
6. Add reports and diagnostics lifecycle.
7. Add pipeline runner with fake modules.
8. Add deterministic validators.
9. Add structured LLM client interface and fake client.
10. Add OpenAI-compatible LLM client.
11. Add bounded LLM scheduler.
12. Port grammar module.
13. Port glossary module.
14. Port homophones module.
15. Port spoken-word module.
16. Enable default full pipeline.
17. Add subprocess integration tests.
18. Update README and operational documentation.
Each milestone should leave the repository in a passing state.
## Guidance for LLM implementation agents
When using an LLM agent such as Codex to implement this rewrite, prefer small prompts tied to a single milestone.
Good agent instructions:
- specify the exact phase being implemented;
- name the files or packages to create;
- require tests for the new behavior;
- require `go test ./...` to pass;
- prohibit prompt wording changes unless the phase is explicitly about prompts;
- require preserving CLI compatibility;
- require preserving stdout/stderr discipline;
- require staging/committing only after tests pass, if the workflow permits commits.
Avoid broad prompts such as “rewrite Audita in Go.” They invite unnecessary redesign and make review difficult.
## Review checklist for each phase
Before accepting a phase implementation, check:
- Does it preserve the public contract?
- Does it introduce unnecessary redesign?
- Are errors typed and understandable?
- Are reports still machine-readable?
- Are diagnostics sufficient to debug failures?
- Are API keys redacted?
- Are stdout and stderr clean?
- Are tests deterministic?
- Does `go test ./...` pass?
- Is the code idiomatic Go rather than Python-shaped Go?
## Historical checkpoint note
Earlier "Phase 2 status" notes are intentionally superseded by
`Current Go rewrite status (Phase 3 foundation)` near the top of this document.
Use that section as the authoritative current-state snapshot.
## When to consider an HTTP API
An HTTP API should be deferred until the Go CLI is stable.
Consider HTTP later only if one or more of these become true:
- multiple callers need a shared long-running Audita worker;
- process startup time becomes a meaningful bottleneck;
- a job queue is needed;
- remote execution becomes necessary;
- centralized concurrency control is preferable to per-process control;
- operational monitoring of Audita as a service becomes valuable.
Even then, the HTTP API should wrap the same core Go pipeline used by the CLI. The CLI should remain the simplest and most reliable integration surface.
## Current implementation status
The Go rewrite is currently in a deterministic foundation stage.
Implemented:
- CLI command surface for `audita process`.
- Config/env/flag loading and validation.
- Transcript/glossary schema parsing and validation.
- Deterministic normalization with summary stats.
- Deterministic chunking with summary stats.
- Per-run diagnostics directory plus normalization/chunking artifacts.
- Minimal process report output (`--report-json` and run-dir `report.json`).
- Framework foundation packages for contracts and proposal preview/apply semantics.
Not yet implemented in runtime pipeline:
- Module runner orchestration.
- Real correction modules.
- Validator-chain execution.
- Structured LLM client integration.
- End-to-end transcript polishing behavior.
## Phase status (completed / partial / pending)
### Completed
- Phase 1: Go CLI skeleton.
- Phase 2: Schemas and file I/O.
- Phase 3: Deterministic normalization.
- Phase 4: Chunking and token estimation (deterministic implementation present).
- Phase 5: Proposal model and application semantics (foundation package implemented and tested).
### Partial
- Phase 6: Reports and diagnostics.
What is complete:
- run directory creation
- source/normalized/chunking artifacts
- report writing
- failure `error.log`
- retention mode parsing/validation
What is still incomplete vs original plan:
- redacted invocation/config metadata artifacts are not written yet
- module prompt/response diagnostics are not present (no module execution yet)
- `auto` retention semantics currently behave like `never` for successful runs
### Pending
- Phase 7+: runner orchestration, validators in runtime path, LLM client, bounded concurrency, real modules, full compatibility and side-by-side rollout phases.
## Deviations from original phase plan
1. Proposal foundation (`internal/framework/proposals`) and framework contracts (`internal/framework/contracts`) were implemented before full runner orchestration.
2. Chunking was integrated directly into the current CLI runtime path before module execution exists.
3. Reporting currently uses a process-level status report with `phase: "phase3-chunking"`, rather than module-level pipeline reports.
4. Work-dir retention `auto` is intentionally deferred and currently simplified.
## Next recommended implementation phase
Implement **Phase 7 (pipeline runner with fake modules)** next.
Concrete target for next phase:
1. Add a production runner package wired from `internal/cli/run.go`.
2. Resolve configured module keys into run specs using existing contracts helper.
3. Execute fake or deterministic in-repo modules sequentially over a working transcript.
4. Run proposal application through `internal/framework/proposals` for each stage.
5. Emit module-level applied/skipped results into report structures.
6. Preserve current stdout/stderr behavior and diagnostics retention behavior.
Rationale:
- This uses already-implemented deterministic pieces.
- It closes the biggest current gap: no actual module pipeline execution.
- It enables validator and LLM phases to be added incrementally afterward.
## Guidance for Codex-style prompts
When requesting the next implementation steps, use narrow prompts such as:
- "Implement runner orchestration using existing contracts and proposal apply package, no LLM calls yet."
- "Add module-level report structs and wire them into process report."
- "Keep CLI contract unchanged and ensure `go test ./...` passes."
Avoid broad prompts like "finish the rewrite"; they blur phase boundaries and make review harder.
## Compatibility reminders
Keep these constraints unchanged while implementing later phases:
- Preserve subprocess-safe stdout/stderr behavior.
- Preserve CLI-over-env precedence.
- Preserve deterministic safety-first proposal application semantics.
- Do not claim module or LLM behavior exists until it is actually in runtime path.