audita/docs/architecture.md

# Audita Go Architecture

## Scope and intent
This document describes:
- the current implemented Go architecture; and
- the intended final architecture for later rewrite phases.

Status labels are explicit so future engineers and LLM agents do not assume unimplemented behavior exists.

## Current implementation status
Implemented today:
- Go CLI entrypoint and `audita process` wiring.
- Config defaults, env loading, CLI override precedence, and validation.
- Transcript and glossary parsing/validation.
- Deterministic transcript normalization.
- Deterministic token estimation and transcript chunking.
- Per-run diagnostics directory creation plus Phase 6 process-level artifacts.
- Process report JSON output with diagnostics artifact references.
- Framework foundation packages for contracts and proposal application.
- Production runner orchestration package with deterministic sequential module execution.
- Module-level report structures with applied/skipped change records.

Not implemented in CLI runtime path today:
- Real module execution pipeline (`glossary`, `homophones`, `spoken_word`, `grammar`).
- Structured LLM proposal generation.
- Validator chain execution.
- Production LLM scheduler behavior.
- End-to-end transcript polishing with real module behavior.

## Actual Go package layout

```text
cmd/audita/
  main.go

internal/cli/
  run.go

internal/core/config/
  config.go
  env.go
  flags.go
  redaction.go
  validation.go

internal/core/schema/
  transcript.go
  glossary.go
  errors.go

internal/core/io/
  files.go

internal/core/normalization/
  normalize.go
  tokens.go

internal/core/chunking/
  sections.go
  summary.go
  tokens.go

internal/core/diagnostics/
  run_dir.go

internal/core/reporting/
  report.go

internal/framework/contracts/
  contracts.go

internal/framework/proposals/
  proposal.go
  policy.go
  preview.go
  apply.go

internal/framework/runner/
  runner.go
```

## Current CLI behavior
Primary command:

```sh
audita process <transcript.json> --glossary <glossary.yaml> [flags]
```

Current runtime flow (`internal/cli/run.go`):
1. Load config from env.
2. Parse flags and apply CLI overrides.
3. Validate transcript positional argument and required `--glossary`.
4. Create per-run diagnostics directory.
5. Read transcript and glossary files.
6. Parse/validate transcript and glossary.
7. Write source transcript artifacts.
8. Normalize transcript.
9. Write normalized transcript and normalization summary artifacts.
10. Chunk normalized transcript and compute chunk summaries.
11. Write chunking summary artifact.
12. Optionally execute runner modules sequentially when an injected module registry/factory is available (used by deterministic tests today).
13. Output working transcript to `--output` file or stdout.
14. Build process report (`phase` currently set to `phase7-runner`).
15. Optionally write `--report-json`; always write run-dir `report.json`.
16. Apply work-dir retention.

Important behavior details:
- Glossary is validated but not yet used for real correction module logic.
- Default production CLI behavior remains deterministic normalization/chunking output because no real module implementations are registered yet.
- No LLM calls occur.
- Success path is generally quiet on stderr.
- Source IDs are preserved into a canonical transcript before normalization; normalization then reassigns output IDs sequentially from `1`.

## Implemented data contracts

### Transcript input
Accepted top-level forms:
- bare JSON array of segments
- object with `segments` array

Source segment contract:
- `id` optional integer
- `speaker` non-empty string
- `start` finite non-negative number
- `end` finite non-negative number with `end >= start`
- `text` non-empty string
- `categories` optional array of non-empty strings

Additional checks:
- duplicate explicit source IDs are rejected.

### Transcript output
Current output uses `schema.TranscriptToJSON` and is a bare JSON array of normalized segments:
- `id`, `speaker`, `start`, `end`, `text`, optional `categories`.

### Glossary input
YAML with `glossary` entries. Required fields per entry:
- `name`, `category`, `summary`

Optional:
- `aliases`, `plural`

## Implemented config/env/flag behavior
Precedence:
1. defaults (`config.Default()`)
2. environment (`config.LoadFromEnv()`)
3. CLI flags (`ApplyCLIOverrides`)

Implemented config surfaces include:
- module list
- primary and validation LLM settings
- section token controls and target sections
- confidence thresholds
- normalization controls
- work-dir and retention mode

Current caveat:
- LLM/module-related settings are mostly infrastructure-only today; runtime path does not execute LLM or modules.

## Implemented normalization behavior
Normalization (`internal/core/normalization`) currently:
- sorts by segment start time;
- merges adjacent same-speaker segments when constraints pass;
- uses gap-based joiners:
  - gap `< ellipsis_gap` -> single space join
  - gap `>= ellipsis_gap` -> `... ` join
- enforces merged duration and token-limit constraints;
- reassigns output IDs sequentially from `1`;
- returns `NormalizationSummary` with merge and skip counters.

Note: merged categories are concatenated (not deduplicated).

## Implemented chunking behavior
Chunking (`internal/core/chunking`) currently provides:
- deterministic heuristic token estimation;
- contiguous sectioning with section metadata;
- max/min section token validation;
- optional `target_sections` handling with target-aware merge/split logic;
- summary and detailed summary generation.

Current behavior details:
- if a single segment exceeds max tokens, it is emitted as its own section (not hard-failed);
- section balancing is deterministic but heuristic.

## Implemented proposal/replacement infrastructure
`internal/framework/proposals` provides deterministic foundation logic:
- `CorrectionProposal` and `EnrichedCorrectionProposal` models;
- replacement policies: `require_unique`, `replace_all`;
- safe preview (`PreviewProposalForSegment`) with stable skip reasons;
- deterministic apply (`ApplyProposals`) in ascending `proposal_index` order;
- applied/skipped change records suitable for reporting.

`internal/framework/contracts` provides interfaces and run-spec metadata scaffolding, including deterministic repeated module instance naming (`ResolveModuleRunSpecs`).

These primitives are wired into the production runner and report model. Real module implementations are still pending.

## Reports and diagnostics (implemented)
Current per-run artifacts include:
- `source-transcript.json`
- `source-transcript-parsed.json`
- `normalized-transcript.json`
- `normalization-summary.json`
- `chunking-summary.json`
- `invocation.json`
- `effective-config.json` (redacted credentials)
- `report.json`
- `error.log` on failure

`--report-json` writes a separate report file when requested.

Current process reports include diagnostics metadata references for:
- diagnostics directory path;
- source transcript artifact path;
- parsed source transcript artifact path;
- normalized transcript artifact path;
- normalization summary artifact path;
- chunking summary artifact path;
- invocation metadata artifact path;
- redacted effective-config artifact path;
- error-log artifact path on failure.

Current process reports also include:
- module-level results (when runner modules execute), including applied/skipped proposal changes;
- run-level module summary totals and failed module instance metadata.

Retention modes implemented in `ApplyRetention`:
- `always`: keep all run directories.
- `never`: keep successful run directories.
- `auto`: keep failed runs and successful runs with skipped corrections.
- failed runs are always retained.

Current runtime note:
- real module execution is not implemented yet, so normal successful runs generally have no skipped corrections and `auto` typically removes clean successful run directories.

Intentionally deferred to module/LLM phases:
- module prompt/response diagnostics artifacts are not produced yet because module execution and LLM calls are not in the runtime path.

## Current tests and quality posture
Implemented tests currently cover:
- CLI argument handling and behavior (`internal/cli/run_test.go`)
- subprocess stdout/stderr and exit-code behavior (`cmd/audita/main_integration_test.go`)
- config/env/override validation (`internal/core/config/*_test.go`)
- transcript and glossary schema validation (`internal/core/schema/*_test.go`)
- deterministic normalization (`internal/core/normalization/*_test.go`)
- deterministic chunking and summaries (`internal/core/chunking/*_test.go`)
- proposal preview/apply semantics (`internal/framework/proposals/*_test.go`)
- contracts/foundation composition tests (`internal/framework/contracts/*_test.go`)
- runner sequencing and failure behavior with deterministic fake modules (`internal/framework/runner/*_test.go`)
- CLI runner integration through injected fake module factories (`internal/cli/run_test.go`)

Not covered yet (because not implemented): validator runtime flow with approvals/rejections and real LLM integration.

## Intended final architecture (not yet implemented)
The intended end-state still matches the rewrite plan:
- sequential module pipeline over a mutable working transcript
- real module implementations (`glossary`, `homophones`, `spoken_word`, `grammar`)
- structured LLM proposal generation
- deterministic and LLM validators
- validator cardinality enforcement in pipeline execution
- proposal application integrated per module stage
- prompt/response diagnostics for LLM/module stages

Until those phases are implemented, documentation and external descriptions should treat the current Go CLI as deterministic preprocessing/reporting infrastructure, not a full LLM transcript polisher.