From 87e560dd3de1e9daae7915371f37be4044a7ea36 Mon Sep 17 00:00:00 2001 From: Eric Rakestraw Date: Sun, 10 May 2026 23:04:29 +0000 Subject: [PATCH] Added architecture reference documentation for the upcoming Go rewrite --- docs/architecture.md | 656 ++++++++++++++++++++++++++++++++++ docs/rewrite-notes.md | 582 ++++++++++++++++++++++++++++++ README.md => python/README.md | 8 + 3 files changed, 1246 insertions(+) create mode 100644 docs/architecture.md create mode 100644 docs/rewrite-notes.md rename README.md => python/README.md (96%) diff --git a/docs/architecture.md b/docs/architecture.md new file mode 100644 index 0000000..b397f77 --- /dev/null +++ b/docs/architecture.md @@ -0,0 +1,656 @@ +# Audita Go Architecture + +## Purpose + +Audita is a framework-first transcript polishing application. It takes a source transcript, a glossary, and runtime configuration; runs an ordered sequence of correction modules; and writes a corrected transcript, a structured run report, and diagnostics artifacts. + +The Go implementation should preserve the public behavior and safety posture of the existing Python implementation while using idiomatic Go internals. Treat the existing Python implementation, README, architecture notes, tests, and representative outputs as the behavioral specification during the rewrite. + +## Design goals + +1. Preserve the existing batch CLI contract. +2. Preserve the existing module pipeline semantics. +3. Preserve deterministic safety checks and skipped-change reporting. +4. Preserve diagnostics and report artifacts as first-class outputs. +5. Make subprocess execution reliable from other Go applications. +6. Keep LLM provider integration small, explicit, and OpenAI-compatible. +7. Keep the design modular enough to add new transcript correction modules and validators. +8. Prefer simple, auditable Go code over agent frameworks or heavy runtime abstractions. + +## Non-goals for the initial Go rewrite + +The initial Go implementation should not attempt to redesign Audita. In particular, it should not initially: + +- replace the CLI-first batch model with an HTTP service; +- redesign the transcript or report schema; +- improve prompt wording during the port; +- change the default module order; +- chase exact nondeterministic LLM output parity with the Python implementation; +- introduce a general-purpose workflow or agent framework; +- require downstream applications to change their integration model. + +Later versions may add an HTTP API, new module types, richer report formats, or prompt improvements after the Go CLI is behaviorally stable. + +## Public CLI contract + +The primary command is: + +```sh +audita process transcript.json --glossary glossary.yaml --output corrected.json +``` + +Optional report output: + +```sh +audita process transcript.json --glossary glossary.yaml --output corrected.json --report-json report.json +``` + +Custom module sequence: + +```sh +audita process transcript.json --glossary glossary.yaml --modules grammar --output corrected.json +``` + +Expected stream behavior: + +- If `--output` is provided, corrected transcript JSON is written to that file. +- If `--output` is omitted, corrected transcript JSON is written to stdout. +- Progress logs and human-readable errors are written to stderr. +- `--report-json` writes a machine-readable report to the specified file. +- Report JSON is never mixed into stdout. +- LLM prompt/response diagnostics are written under the per-run work directory, not to stdout. +- Exit code `0` means the run completed successfully. +- Nonzero exit codes mean the run failed; failed runs preserve diagnostics. + +This stdout/stderr discipline is a core requirement because Audita is expected to be called as a subprocess by other Go applications. + +## Configuration model + +Configuration sources should be applied in this order: + +1. built-in defaults; +2. environment variables; +3. CLI flags. + +CLI flags override environment variables. Environment variables override defaults. + +The initial Go implementation should preserve the existing configuration surface where practical, including: + +- module sequence; +- primary LLM API key, base URL, model, timeout, retry count, and concurrency; +- validation LLM API key, base URL, model, timeout, retry count, and concurrency; +- proposal-stage section token bounds; +- validation-stage prompt token bounds; +- module-specific confidence thresholds; +- normalization parameters; +- work directory location and retention policy. + +Recommended Go shape: + +```text +internal/core/config + Config + LLMConfig + NormalizationConfig + ModuleConfig + WorkDirRetention + LoadFromEnv + ApplyCLIOverrides + Validate + EffectiveValidationLLMConfig +``` + +API credentials must be redacted from reports, logs, diagnostics metadata, and error details. + +## High-level runtime flow + +`audita process` should execute the following flow: + +1. Parse CLI arguments. +2. Load and validate configuration. +3. Load and validate transcript input. +4. Load and validate glossary input. +5. Create a per-run diagnostics directory. +6. Persist redacted invocation/config metadata. +7. Persist the source transcript artifact. +8. Normalize transcript deterministically. +9. Persist normalized transcript and normalization summary. +10. Resolve configured module sequence into module run instances. +11. Execute modules sequentially over a mutable working transcript. +12. For each module instance: + - chunk the current working transcript into contiguous token-bounded sections; + - generate structured correction proposals using the module prompt; + - enrich proposals with module/run metadata; + - run validator chain; + - apply approved proposals according to replacement policy; + - report applied changes and skipped changes. +13. Sort final transcript chronologically. +14. Write corrected transcript to output file or stdout. +15. Write `report.json` if requested and always write authoritative report into retained diagnostics. +16. Apply work-dir retention policy. +17. On failure, preserve diagnostics, write failed report, write `error.log`, print concise stderr summary, and exit nonzero. + +## Suggested Go package layout + +```text +cmd/audita/ + main.go + +internal/core/config/ + config.go + env.go + flags.go + validation.go + +internal/core/schema/ + transcript.go + glossary.go + report.go + +internal/core/io/ + transcript.go + glossary.go + report.go + json.go + yaml.go + +internal/core/normalization/ + normalize.go + summary.go + +internal/core/chunking/ + tokens.go + sections.go + batches.go + +internal/core/diagnostics/ + run_dir.go + artifacts.go + redaction.go + retention.go + +internal/core/errors/ + errors.go + exit_codes.go + +internal/framework/runner/ + runner.go + context.go + result.go + +internal/framework/proposals/ + proposal.go + apply.go + policy.go + +internal/framework/llm/ + client.go + openai_compatible.go + structured.go + retry.go + scheduler.go + +internal/framework/modules/ + module.go + registry.go + sequence.go + +internal/modules/glossary/ + module.go + prompt.go + response.go + +internal/modules/homophones/ + module.go + prompt.go + response.go + +internal/modules/spokenword/ + module.go + prompt.go + response.go + +internal/modules/grammar/ + module.go + prompt.go + response.go + +internal/validators/ + validator.go + result.go + +internal/validators/deterministic/ + confidence.go + original_text.go + protected_terms.go + non_empty.go + +internal/validators/llm/ + validators.go + prompts.go + responses.go + +internal/validators/protection/ + glossary_terms.go + matching.go + +internal/testutil/ + fixtures.go + fakellm.go + golden.go +``` + +This package layout is a starting point, not a rule. Prefer fewer packages if the implementation becomes fragmented. Keep package boundaries aligned with stable domain concepts: config, schema, normalization, chunking, diagnostics, LLM, proposals, modules, validators, and runner. + +## Core data contracts + +### Transcript + +Audita should accept either: + +1. a bare JSON array of segments; or +2. an object containing a `segments` array. + +Segment fields: + +```text +id integer +speaker string +start number +end number +text string +categories optional array of strings +``` + +Input may contain source IDs. The normalized internal transcript should use strict sequential IDs starting at `1`. + +Final output should use the same normalized segment shape and should be sorted chronologically. + +### Glossary + +The glossary model should support the existing glossary semantics used by the Python implementation, including domain-specific terms and protected vocabulary used by validators. The Go implementation should preserve the accepted YAML shape rather than inventing a new one during the rewrite. + +### Correction proposal + +Modules emit structured proposals with the following logical fields: + +```text +id target segment ID +original_text span expected to exist in the target segment +corrected_text replacement text +confidence number between 0.0 and 1.0 +``` + +The framework enriches proposals with execution metadata: + +```text +proposal_index +module_key +module_instance +section_id or batch_id +validator decisions +application status +skip reason, if any +``` + +### Validation decision + +Each validator must return exactly one decision for each candidate proposal index it was asked to evaluate. + +Missing, duplicate, or unknown proposal indexes are framework errors. The framework should not silently ignore malformed validator output. + +### Run report + +The run report should be machine-readable JSON. It should describe: + +- run status; +- start/end timestamps or elapsed duration; +- effective module sequence; +- normalization summary; +- module reports; +- applied changes; +- skipped changes; +- errors, when present; +- diagnostics directory, when retained. + +The report should be stable enough for downstream tooling to consume. + +## Pipeline model + +Audita uses a staged transform model over a mutable working transcript. + +```text +source transcript + -> deterministic normalization + -> working transcript + -> module: glossary_1 + -> module: homophones + -> module: glossary_2 + -> module: spoken_word + -> module: grammar + -> final transcript +``` + +Module order is architecturally significant. Each module sees the transcript produced by all previous modules. + +Default logical module sequence: + +```text +glossary, homophones, glossary, spoken_word, grammar +``` + +Default resolved module instance names: + +```text +glossary_1, homophones, glossary_2, spoken_word, grammar +``` + +Repeated logical module keys should be auto-numbered in reports and diagnostics. + +## Module interface + +A module should be a small object with stable metadata and proposal behavior. + +Conceptual interface: + +```go +type TranscriptModule interface { + Key() string + ReplacementPolicy() proposals.ReplacementPolicy + Validators() []validators.Validator + Propose(ctx context.Context, req ProposalRequest) ([]proposals.CorrectionProposal, error) +} +``` + +A module is responsible for: + +- building its proposal prompt; +- requesting structured proposal output from the LLM client; +- mapping structured output into framework proposals; +- attaching its validator chain. + +A module should not: + +- own global process state; +- write final outputs; +- decide work-dir retention; +- apply proposals directly; +- bypass framework validators; +- perform unbounded concurrency. + +## Module responsibilities + +### `glossary` + +Proposes glossary-supported acoustic corrections and domain-specific term corrections. This module should be conservative and should rely heavily on glossary evidence and protected-vocabulary validation. + +### `homophones` + +Proposes conservative homophone and mistranscription corrections. This module should avoid stylistic editing and should focus on likely transcription errors. + +### `spoken_word` + +Proposes conservative dysfluency cleanup that does not affect substantive meaning. This module should be strongly guarded by semantic validators because it is easy to over-edit spoken language. + +### `grammar` + +Proposes punctuation, capitalization, and spacing cleanup only. This module should not rewrite content for style or clarity beyond formatting and readability corrections. + +## Replacement policies + +Proposal application should be centralized in the framework. + +Supported policies: + +```text +require_unique original_text must match exactly once in the target segment +replace_all replace every occurrence of original_text in the target segment +``` + +If a proposal cannot be applied safely, it should be skipped and reported instead of crashing the run. Examples: + +- target segment no longer exists; +- `original_text` is missing; +- `require_unique` found zero matches; +- `require_unique` found multiple matches; +- corrected text is empty after validation; +- proposal became stale after earlier module edits. + +## Validator model + +Validators filter candidate proposals before application. + +Validator categories: + +1. deterministic validators; +2. LLM-backed validators. + +Deterministic validators should run before LLM validators whenever possible because they are cheaper, faster, and more predictable. + +Examples of deterministic validators: + +- confidence threshold; +- original text presence; +- non-empty correction; +- identical text rejection; +- protected glossary term checks. + +Examples of LLM-backed validators: + +- spoken-form plausibility; +- meaning reversal detection; +- editorial review; +- grammar review; +- spoken-word cleanup review. + +A validator should return structured results only. Human-readable explanations may be included in reports, but validation logic should consume typed decisions. + +## LLM integration + +Audita should use a small OpenAI-compatible structured-output client. + +The client should support: + +- configurable base URL; +- configurable model; +- optional API key; +- per-request timeout; +- retry budget; +- structured JSON response schema; +- raw prompt/response diagnostic capture; +- context cancellation; +- clear error wrapping. + +Conceptual interface: + +```go +type StructuredLLMClient interface { + CompleteStructured(ctx context.Context, req StructuredRequest, out any) error +} +``` + +The LLM client should not know about Audita modules, transcript schemas, or validators. It should only know how to submit an OpenAI-compatible request and decode a structured response. + +Proposal and validation phases may use different effective LLM settings. Validation settings inherit from proposal settings unless explicitly overridden. + +## Structured output policy + +For every LLM call, Audita should prefer strict structured JSON output over free-form text parsing. + +Each module and LLM validator should define: + +- request payload type; +- response payload type; +- JSON schema, where the backend supports schema-constrained output; +- response validation rules; +- retry behavior for malformed or incomplete responses. + +Malformed structured responses should become typed errors that include diagnostic references but do not leak API keys or excessive prompt text to stderr. + +## Concurrency model + +Module stages remain sequential. + +Within a module, bounded concurrency is allowed for: + +- proposal generation across transcript sections; +- LLM validation batches; +- adjacent independent LLM validators that evaluate the same candidate set; +- prompt/response artifact writes. + +All backend LLM calls should pass through a scheduler or semaphore that enforces configured concurrency limits. + +Use `context.Context` for cancellation and timeout propagation. + +Avoid unbounded goroutine creation. Every concurrent unit should be attached to the current run context and should return errors through a controlled mechanism such as `errgroup`. + +## Diagnostics model + +A retained run directory should use a predictable structure similar to: + +```text +runs/ + 2026-05-10T210000Z-/ + invocation.json + source-transcript.json + normalized-transcript.json + normalization-summary.json + modules/ + glossary_1/ + section-001.prompt.json + section-001.response.json + homophones/ + glossary_2/ + spoken_word/ + grammar/ + report.json + error.log +``` + +Retention modes: + +```text +always keep every run directory +never remove successful run directories +auto keep failed runs and successful runs with final skipped corrections +``` + +Failed runs are always retained. + +Diagnostics should be useful for debugging LLM behavior, proposal generation, validation decisions, and replacement failures. + +## Error handling model + +Use typed errors for expected failure classes: + +- configuration errors; +- input validation errors; +- normalization errors; +- module resolution errors; +- LLM request errors; +- structured response errors; +- validator contract errors; +- proposal application errors; +- output write errors. + +A pipeline failure should preserve partial progress when possible: + +- partial working transcript; +- completed module reports; +- applied changes so far; +- skipped changes so far; +- failed module information; +- error details; +- diagnostics references. + +The CLI should print concise stderr output and point to retained diagnostics rather than dumping large prompts or stack traces directly to the terminal. + +## Testing strategy + +The Go implementation should use three categories of tests. + +### Deterministic unit tests + +Cover: + +- config loading and precedence; +- transcript parsing; +- glossary parsing; +- normalization; +- chunking; +- proposal preview/application; +- replacement policies; +- validator contract enforcement; +- report serialization; +- diagnostics retention decisions. + +### Fake-LLM integration tests + +Use a fake structured LLM client to test: + +- module proposal flow; +- validator flow; +- full pipeline execution; +- repeated module instance names; +- skipped-change reporting; +- malformed LLM responses; +- retry behavior; +- partial failure reporting. + +### Subprocess tests + +Invoke the compiled `audita` binary from tests and verify: + +- stdout contains only corrected transcript JSON when `--output` is omitted; +- stdout is clean when `--output` is provided; +- stderr contains logs/errors only; +- `--report-json` writes a valid report; +- failed runs exit nonzero; +- failed runs retain diagnostics; +- large inputs do not deadlock stdout/stderr pipes; +- cancellation and timeout behavior are reliable. + +## Extension points + +### Add a module + +1. Create a package under `internal/modules/`. +2. Implement the module interface. +3. Define prompt builder and structured response type. +4. Define or reuse validators. +5. Register the logical module key in the module registry. +6. Add fake-LLM tests. +7. Add a fixture-level integration test. + +### Add a validator + +1. Implement the validator interface. +2. Define typed result/decision structures. +3. For LLM validators, define prompt and structured response schema. +4. Enforce one decision per candidate proposal index. +5. Add tests for approved, rejected, malformed, and missing-decision cases. +6. Attach the validator to module chains intentionally. + +### Add an LLM backend + +1. Implement `StructuredLLMClient`. +2. Preserve request timeout and context cancellation behavior. +3. Preserve raw prompt/response diagnostic capture. +4. Preserve structured response validation and retry semantics. +5. Add fake or local integration tests. + +## Implementation posture + +The Go version should be boring infrastructure: + +- explicit structs; +- explicit validation; +- small interfaces; +- clear package boundaries; +- no hidden global state; +- deterministic file outputs; +- stable subprocess behavior; +- diagnostics-first failures; +- structured LLM calls rather than text scraping. + +The primary measure of success is not that the Go code resembles the Python code. The measure of success is that the Go binary can replace the Python CLI in the surrounding transcript pipeline with fewer operational surprises. diff --git a/docs/rewrite-notes.md b/docs/rewrite-notes.md new file mode 100644 index 0000000..4430f1c --- /dev/null +++ b/docs/rewrite-notes.md @@ -0,0 +1,582 @@ +# Audita Go Rewrite Notes + +## Rewrite strategy + +The Go rewrite should be compatibility-first. The existing Python implementation should be treated as the executable specification for public behavior, pipeline semantics, diagnostics, and safety rules. + +The rewrite should not begin as a redesign. The first production-capable Go version should be able to stand in for the Python CLI in the surrounding transcript pipeline. + +The initial goal is: + +```text +same public contract +same pipeline semantics +same safety posture +same diagnostic philosophy +idiomatic Go implementation +reliable subprocess behavior +``` + +Exact LLM output parity is not required because LLM calls are nondeterministic and may vary by backend, prompt formatting, model, or structured-output implementation. Deterministic framework behavior should be held to a much stricter compatibility standard. + +## Core principles + +1. Preserve behavior before improving behavior. +2. Port deterministic layers before LLM layers. +3. Test with fake LLMs before testing with real LLMs. +4. Keep module stages sequential until correctness is established. +5. Add bounded concurrency only after the sequential implementation is correct. +6. Keep the CLI contract stable for downstream callers. +7. Keep stdout/stderr behavior clean and predictable. +8. Treat diagnostics and reports as part of the product, not as afterthoughts. +9. Avoid prompt improvements during the port. +10. Prefer explicit Go structs and validation over reflection-heavy abstractions. + +## Compatibility targets + +The Go implementation should preserve the following public behaviors where practical: + +- `audita process` command shape; +- transcript input support for bare segment arrays and `{ "segments": [...] }` objects; +- glossary YAML support; +- default module sequence; +- repeated module instance naming; +- environment variable and CLI flag configuration concepts; +- CLI-over-env precedence; +- output file behavior; +- stdout behavior when `--output` is omitted; +- stderr logging behavior; +- separate `--report-json` machine-readable report behavior; +- work-dir diagnostics behavior; +- retention modes `auto`, `always`, and `never`; +- failed-run diagnostics preservation; +- skipped-change reporting instead of crashing on stale or unsafe proposal application. + +## Phase 0: Freeze the Python implementation as the reference + +Before writing substantial Go code, preserve the behavior of the current Python implementation. + +Tasks: + +1. Create or identify a stable branch/tag representing the Python reference implementation. +2. Preserve the existing README and architecture notes. +3. Preserve the Python regression suite. +4. Collect representative transcript/glossary fixtures. +5. Capture reference outputs for deterministic behaviors. +6. Capture several real end-to-end run artifacts for qualitative comparison. + +Representative fixtures should include: + +- tiny valid transcript; +- transcript with multiple speakers; +- transcript accepted as a bare array; +- transcript accepted as an object with `segments`; +- glossary-supported correction case; +- homophone/mistranscription case; +- spoken-word dysfluency case; +- grammar/punctuation/capitalization case; +- duplicate `original_text` case; +- missing `original_text` case; +- skipped-change case; +- malformed input case; +- failed LLM response case. + +Definition of done: + +- A developer or LLM agent can run the Python test suite. +- Reference fixtures are committed. +- Reference reports or golden artifacts exist for deterministic comparisons. +- The Python implementation can be used to answer disputed behavior questions during the rewrite. + +## Phase 1: Create the Go CLI skeleton + +Build the smallest Go binary that preserves the outer command shape. + +Tasks: + +1. Initialize Go module. +2. Add `cmd/audita/main.go`. +3. Implement `audita process` command. +4. Parse core flags: + - transcript path; + - `--glossary`; + - `--output`; + - `--report-json`; + - `--modules`; + - LLM config flags; + - normalization flags; + - work-dir flags. +5. Implement config defaults. +6. Implement environment variable loading. +7. Implement CLI-over-env precedence. +8. Implement redaction for sensitive config values. +9. Load input files but initially write transcript back unchanged. + +Definition of done: + +- `go test ./...` passes. +- `go run ./cmd/audita process transcript.json --glossary glossary.yaml --output corrected.json` succeeds for a minimal fixture. +- The output transcript is valid JSON. +- With `--output`, stdout is empty except for intentional machine output, preferably empty. +- Logs go to stderr. +- Invalid flags produce a clean error and nonzero exit. + +## Phase 2: Port schemas and file I/O + +Implement typed transcript, glossary, and report data structures. + +Tasks: + +1. Define transcript segment structs. +2. Support input as bare segment array. +3. Support input as object with `segments` array. +4. Preserve optional `categories`. +5. Validate required fields. +6. Validate basic timing shape. +7. Define glossary structs matching the existing YAML format. +8. Implement glossary parsing. +9. Implement report structs at least sufficient for early phases. +10. Implement JSON/YAML read/write helpers. + +Definition of done: + +- Valid Python-era fixtures parse successfully. +- Invalid fixtures fail with clear errors. +- Transcript round-trips through Go without accidental data loss. +- Glossary fixtures parse successfully. +- Report JSON can be written and parsed by tests. + +## Phase 3: Port deterministic normalization + +Port normalization before any LLM work. + +Tasks: + +1. Implement same-speaker segment merging. +2. Implement maximum merge gap. +3. Implement ellipsis gap insertion. +4. Implement maximum merged segment duration. +5. Implement maximum merged segment token budget. +6. Reassign strict sequential segment IDs starting at `1`. +7. Produce a normalization summary. +8. Preserve source transcript and normalized transcript diagnostics. + +Definition of done: + +- Normalization tests pass against golden fixtures. +- Segment IDs are deterministic. +- Chronological ordering is deterministic. +- Normalization summary is included in the run report. +- Normalization artifacts are written in the run directory. + +## Phase 4: Port chunking and token estimation + +Implement the section-building logic used by proposal generation and validator batching. + +Tasks: + +1. Implement approximate token estimator. +2. Implement contiguous transcript sections. +3. Implement minimum and maximum section token settings. +4. Implement exact target section count behavior if supported by the Python implementation. +5. Implement validation prompt batching by token limit. +6. Add tests for edge cases. + +Edge cases: + +- very small transcript; +- single very large segment; +- many short segments; +- exact target section count impossible; +- min/max token bounds conflict; +- speaker boundaries near section boundaries. + +Definition of done: + +- Chunking is deterministic. +- Sections preserve transcript order. +- Sections do not drop or duplicate segments. +- Token-boundary behavior is covered by tests. + +## Phase 5: Port proposal model and application semantics + +Implement correction proposals and safe replacement behavior. + +Tasks: + +1. Define correction proposal struct. +2. Define enriched proposal metadata. +3. Define replacement policies: + - `require_unique`; + - `replace_all`. +4. Implement proposal preview. +5. Implement proposal application. +6. Implement skipped-change reporting. +7. Handle stale proposals safely. +8. Add tests for every skip reason. + +Skip cases should include: + +- missing target segment; +- missing `original_text`; +- multiple matches under `require_unique`; +- empty corrected text; +- identical original/corrected text after normalization; +- proposal invalidated by earlier edits. + +Definition of done: + +- Proposal application never panics on malformed proposal input. +- Unsafe proposals are skipped and reported. +- Applied changes and skipped changes are serializable in the report. +- Tests cover both replacement policies. + +## Phase 6: Implement reports and diagnostics + +Build diagnostics and reporting before real LLM calls so failures are inspectable from the beginning. + +Tasks: + +1. Create per-run directory under configured work dir. +2. Write redacted invocation/config metadata. +3. Write source transcript artifact. +4. Write normalized transcript artifact. +5. Write normalization summary artifact. +6. Write module prompt/response artifacts once module execution exists. +7. Write authoritative `report.json` into retained run directories. +8. Implement `--report-json` output path. +9. Implement `error.log` on failure. +10. Implement retention modes. + +Retention behavior: + +```text +always keep all run directories +never remove successful run directories +auto keep failed runs and successful runs with skipped corrections +``` + +Failed runs are always retained. + +Definition of done: + +- Clean successful runs obey retention policy. +- Successful runs with skipped corrections are retained under `auto`. +- Failed runs are always retained. +- Failed runs write `report.json` and `error.log` when possible. +- CLI stderr points to retained diagnostics on failure. + +## Phase 7: Implement the pipeline runner with fake modules + +Build the orchestration engine before porting real correction modules. + +Tasks: + +1. Define module interface. +2. Define validator interface. +3. Define runner context. +4. Define module run report model. +5. Implement module sequence resolution. +6. Implement repeated module instance naming. +7. Implement sequential module execution. +8. Implement fake module for tests. +9. Implement fake validator for tests. +10. Wire proposal application into runner. + +Definition of done: + +- A fake module can propose a correction and the runner applies it. +- A fake validator can reject a correction and the runner reports it. +- Multiple modules run in configured order. +- Repeated module keys are named deterministically. +- A module failure produces a partial failed report. + +## Phase 8: Port deterministic validators + +Port cheap, deterministic validation before LLM-backed validation. + +Tasks: + +1. Implement confidence threshold validator. +2. Implement original-text presence validator. +3. Implement non-empty correction validator. +4. Implement identical-text rejection. +5. Implement protected vocabulary logic from glossary. +6. Enforce validator result cardinality. +7. Add tests for malformed validator results. + +Definition of done: + +- Deterministic validators run before LLM validators. +- Each validator returns exactly one decision per input proposal. +- Missing, duplicate, or unknown proposal indexes produce framework errors. +- Rejected proposals are reported with reasons. + +## Phase 9: Implement structured LLM client + +Add the OpenAI-compatible structured-output client after the deterministic framework is stable. + +Tasks: + +1. Define `StructuredLLMClient` interface. +2. Define structured request type. +3. Implement OpenAI-compatible chat completions client. +4. Support configurable base URL, model, API key, timeout, and retries. +5. Support optional API key for self-hosted endpoints. +6. Implement structured JSON response parsing. +7. Implement response validation. +8. Implement retry behavior for malformed structured output. +9. Write raw prompt/response diagnostics. +10. Support separate effective validation LLM settings. + +Definition of done: + +- Fake LLM tests still pass. +- Real LLM smoke test can run against a configured endpoint. +- Malformed LLM responses are retried or reported cleanly. +- API keys are never logged or written to diagnostics. +- Context timeout/cancellation works. + +## Phase 10: Add bounded concurrency + +Add concurrency only after sequential correctness is established. + +Tasks: + +1. Add LLM call scheduler/semaphore. +2. Add proposal-generation concurrency across transcript sections. +3. Add validator batching concurrency where useful. +4. Ensure all goroutines are attached to run context. +5. Use `errgroup` or equivalent controlled error propagation. +6. Add tests for concurrency limits using fake LLM instrumentation. + +Rules: + +- Module stages remain sequential. +- LLM backend calls must respect configured concurrency. +- No unbounded goroutine creation. +- Cancellation must stop pending work promptly. + +Definition of done: + +- Concurrency limit is enforced in tests. +- Results remain deterministic where ordering matters. +- Module reports remain stable. +- Cancellation and timeout behavior are covered. + +## Phase 11: Port real modules one at a time + +Port modules after the framework, validators, and LLM client are in place. + +Recommended order: + +1. `grammar` +2. `glossary` +3. `homophones` +4. `spoken_word` +5. full default sequence + +This order exercises formatting first, then domain-specific correction, then more subtle semantic cleanup. + +For each module: + +1. Port prompt builder without improving wording. +2. Define structured response type. +3. Define JSON schema if supported by the backend. +4. Convert structured response into framework proposals. +5. Attach intended validators. +6. Add fake-LLM tests. +7. Add fixture-level integration tests. +8. Add real LLM smoke test where practical. +9. Compare qualitative output against Python reference. + +Definition of done for each module: + +- Fake-LLM tests pass. +- Module-specific fixtures pass. +- Prompt/response diagnostics are written. +- Validator chain is explicit and tested. +- Module report includes applied and skipped changes. + +## Phase 12: Full-pipeline compatibility testing + +Run complete Go pipeline tests after all modules exist. + +Tasks: + +1. Run full default module sequence with fake LLM responses. +2. Run full default module sequence against small real transcript. +3. Run against at least one real D&D session transcript. +4. Compare output shape against Python reference. +5. Compare report shape against Python reference. +6. Inspect skipped changes manually. +7. Inspect diagnostics manually. +8. Verify downstream parser compatibility. + +Definition of done: + +- Go output can be consumed by the next pipeline stage. +- Report JSON can be consumed by existing tooling or by documented replacement tooling. +- Real run diagnostics are sufficient for debugging. +- Output quality is at least acceptable compared with Python. + +## Phase 13: Subprocess integration testing + +Test the compiled Go binary exactly as surrounding Go applications will call it. + +Tasks: + +1. Create integration tests that execute the binary as a subprocess. +2. Test `--output` path behavior. +3. Test stdout-only transcript output when `--output` is omitted. +4. Test stderr-only logs/errors. +5. Test `--report-json` path behavior. +6. Test nonzero exit on invalid input. +7. Test nonzero exit on LLM failure. +8. Test large transcript pipe behavior. +9. Test timeout/cancellation behavior from parent process. + +Definition of done: + +- No stdout/stderr deadlocks. +- Parent Go process can reliably distinguish success from failure by exit code. +- Parent Go process can parse output transcript and report files. +- Failed runs provide diagnostics paths. + +## Phase 14: Run Python and Go side by side + +For a transition period, keep both implementations available. + +Tasks: + +1. Give the Python binary a distinct name if needed, such as `audita-python`. +2. Give the Go binary the canonical `audita` name only after it is ready. +3. Allow the orchestrator to select implementation during transition. +4. Run the same real sessions through both implementations. +5. Compare reports and final transcripts. +6. Record known intentional differences. +7. Fix unintentional compatibility breaks. + +Comparison criteria: + +- both complete successfully; +- both preserve transcript JSON shape; +- both run the intended module sequence; +- both report applied/skipped changes; +- Go has cleaner subprocess behavior; +- Go diagnostics are at least as useful as Python diagnostics; +- Go transcript quality is acceptable for downstream use. + +Definition of done: + +- Go implementation has passed several real-session runs. +- Downstream applications can call Go Audita reliably. +- Any report/schema differences are documented. +- Python can be retired from the main path. + +## Phase 15: Retire or archive the Python implementation + +After the Go version becomes the default, preserve the Python implementation as historical reference unless there is a reason to remove it. + +Tasks: + +1. Mark Python implementation as archived or prototype. +2. Preserve useful fixtures and tests. +3. Preserve prompts and reference outputs. +4. Remove Python from production orchestration. +5. Update README to describe Go as the active implementation. +6. Update installation instructions. +7. Update operational docs. + +Definition of done: + +- Repository clearly identifies the Go implementation as active. +- Python code no longer participates in normal operation. +- Historical Python behavior remains available for reference if needed. + +## Suggested milestone commits + +A reasonable commit sequence for agent-assisted implementation: + +1. Scaffold Go CLI and config loading. +2. Add transcript/glossary schemas and I/O. +3. Add normalization and normalization tests. +4. Add chunking and token estimation. +5. Add proposal model and replacement policies. +6. Add reports and diagnostics lifecycle. +7. Add pipeline runner with fake modules. +8. Add deterministic validators. +9. Add structured LLM client interface and fake client. +10. Add OpenAI-compatible LLM client. +11. Add bounded LLM scheduler. +12. Port grammar module. +13. Port glossary module. +14. Port homophones module. +15. Port spoken-word module. +16. Enable default full pipeline. +17. Add subprocess integration tests. +18. Update README and operational documentation. + +Each milestone should leave the repository in a passing state. + +## Guidance for LLM implementation agents + +When using an LLM agent such as Codex to implement this rewrite, prefer small prompts tied to a single milestone. + +Good agent instructions: + +- specify the exact phase being implemented; +- name the files or packages to create; +- require tests for the new behavior; +- require `go test ./...` to pass; +- prohibit prompt wording changes unless the phase is explicitly about prompts; +- require preserving CLI compatibility; +- require preserving stdout/stderr discipline; +- require staging/committing only after tests pass, if the workflow permits commits. + +Avoid broad prompts such as “rewrite Audita in Go.” They invite unnecessary redesign and make review difficult. + +## Review checklist for each phase + +Before accepting a phase implementation, check: + +- Does it preserve the public contract? +- Does it introduce unnecessary redesign? +- Are errors typed and understandable? +- Are reports still machine-readable? +- Are diagnostics sufficient to debug failures? +- Are API keys redacted? +- Are stdout and stderr clean? +- Are tests deterministic? +- Does `go test ./...` pass? +- Is the code idiomatic Go rather than Python-shaped Go? + +## Red flags during the rewrite + +Watch for: + +- module stages running concurrently and changing pipeline semantics; +- prompt wording changes mixed into infrastructure commits; +- validators silently ignoring malformed LLM output; +- proposal application mutating text without report entries; +- report data printed to stdout unexpectedly; +- API keys appearing in diagnostics; +- global mutable LLM clients that make tests order-dependent; +- unbounded goroutine creation; +- filesystem paths hard-coded outside config defaults; +- tests that require a real LLM when a fake LLM would be better. + +## When to consider an HTTP API + +An HTTP API should be deferred until the Go CLI is stable. + +Consider HTTP later only if one or more of these become true: + +- multiple callers need a shared long-running Audita worker; +- process startup time becomes a meaningful bottleneck; +- a job queue is needed; +- remote execution becomes necessary; +- centralized concurrency control is preferable to per-process control; +- operational monitoring of Audita as a service becomes valuable. + +Even then, the HTTP API should wrap the same core Go pipeline used by the CLI. The CLI should remain the simplest and most reliable integration surface. diff --git a/README.md b/python/README.md similarity index 96% rename from README.md rename to python/README.md index 19220a5..4ddd364 100644 --- a/README.md +++ b/python/README.md @@ -1,5 +1,13 @@ # Audita +```text +This directory contains the frozen Python implementation of Audita. +It is retained as the behavioral reference for the Go port. + +The canonical implementation is moving to the Go code at the repository root. +Do not add new features here except to fix reference-test issues needed for port validation. +``` + Audita is a framework-first transcript correction application. The public `audita` package provides: - deterministic transcript normalization