eric/audita

Fork 0

Files

Eric Rakestraw 68e2d9b549 Complete Phase 18 operational hardening

2026-05-12 13:32:35 +00:00

27 KiB

Raw Blame History

Audita Go Architecture

Scope and intent

This document describes:

the current implemented Go architecture; and
the intended final architecture for later rewrite phases.

Status labels are explicit so future engineers and LLM agents do not assume unimplemented behavior exists.

Current implementation status

Implemented today:

Go CLI entrypoint and audita process wiring.
Config defaults, env loading, CLI override precedence, and validation.
Transcript and glossary parsing/validation.
Deterministic transcript normalization.
Deterministic token estimation and transcript chunking.
Per-run diagnostics directory creation plus Phase 6 process-level artifacts.
Process report JSON output with diagnostics artifact references.
Framework foundation packages for contracts and proposal application.
Production runner orchestration package with deterministic sequential module execution.
Module-level report structures with applied/skipped change records.
Runtime validator models and deterministic validators.
Deterministic validator-chain execution in the runner with cardinality enforcement.
Module-level validator decision/rejection reporting.
Internal structured LLM client contract plus an instructor-go-backed adapter package.
Bounded LLM scheduler/semaphore infrastructure with context-aware permit handling.
Runtime primary/validation LLM effective-config resolution helpers with validation inheritance.
Generic JSON prompt/response diagnostics writer primitives with secret redaction.
LLM-backed validator models, prompt builders, batching, and runtime execution.
Runner wiring for LLM validators via the internal structured LLM abstraction and scheduler hooks.
LLM validator diagnostics artifacts and report-level decision metadata paths.
Shared LLM proposal-generation helper with structured correction-set parsing.
Deterministic proposal-index assignment and enriched proposal mapping for shared generation.
Proposal-generation diagnostics artifacts with secret redaction.
Production module registry scaffolding with known-key recognition and explicit unsupported/unimplemented errors.
Production grammar module implementation in internal/modules/grammar.
Production glossary module implementation in internal/modules/glossary.
Production homophones module implementation in internal/modules/homophones.
Production spoken_word module implementation in internal/modules/spoken_word.
Explicit runtime support for --modules grammar through the production runner path.
Explicit runtime support for --modules glossary, including repeated stages such as --modules glossary,glossary.
Explicit runtime support for --modules homophones through the production runner path.
Explicit runtime support for --modules spoken_word through the production runner path.

Not implemented in CLI runtime path today:

Rollout and Python retirement work (Phase 19).

Current reality:

all production modules exist and are wired into the default runtime path.
a normal audita process run without --modules now executes the full sequence:
- glossary
- homophones
- glossary
- spoken_word
- grammar
repeated glossary stages are deterministic and reported distinctly as glossary_1 and glossary_2.

Phase sequencing note:

Phase 9 LLM infrastructure is complete (structured client, scheduler, effective config resolution, diagnostics primitives);
Phase 10 LLM-backed validator runtime integration is complete;
Phase 11 shared proposal-generation framework and module-registry scaffolding are complete;
Phase 12 grammar module implementation and explicit runtime wiring are complete;
Phase 13 glossary module and protected-term behavior are complete;
Phase 14 homophones module implementation and explicit runtime wiring are complete;
Phase 15 spoken-word module implementation and explicit runtime wiring are complete;
Phase 16 default full pipeline integration is complete;
Phase 17 parity fixture suite is complete;
Phase 18 operational hardening and subprocess integration are complete;
next recommended phase is Phase 19 (documentation, rollout, and Python retirement).

Actual Go package layout

cmd/audita/
  main.go

internal/cli/
  run.go

internal/core/config/
  config.go
  env.go
  flags.go
  redaction.go
  validation.go

internal/core/schema/
  transcript.go
  glossary.go
  errors.go

internal/core/io/
  files.go

internal/core/normalization/
  normalize.go
  tokens.go

internal/core/chunking/
  sections.go
  summary.go
  tokens.go

internal/core/diagnostics/
  run_dir.go

internal/core/reporting/
  report.go

internal/framework/contracts/
  contracts.go

internal/framework/proposals/
  proposal.go
  policy.go
  preview.go
  apply.go

internal/framework/runner/
  runner.go

internal/framework/proposal_generation/
  generate.go

internal/framework/modules/
  registry.go

internal/modules/grammar/
  module.go
  prompt.go

internal/modules/glossary/
  module.go
  prompt.go

internal/modules/homophones/
  module.go
  prompt.go

internal/modules/spoken_word/
  module.go
  prompt.go

internal/framework/validators/
  models.go
  deterministic.go
  llm_models.go
  llm_prompt_builders.go
  llm_batching.go
  llm_validators.go

internal/framework/llm/
  instructor_client.go
  scheduler.go
  effective_config.go
  diagnostics.go

Current CLI behavior

Primary command:

audita process <transcript.json> --glossary <glossary.yaml> [flags]

Current runtime flow (internal/cli/run.go):

Load config from env.
Parse flags and apply CLI overrides.
Validate transcript positional argument and required --glossary.
Create per-run diagnostics directory.
Read transcript and glossary files.
Parse/validate transcript and glossary.
Write source transcript artifacts.
Normalize transcript.
Write normalized transcript and normalization summary artifacts.
Chunk normalized transcript and compute chunk summaries.
Write chunking summary artifact.
Execute runner modules sequentially:
- default run path uses configured default sequence (glossary,homophones,glossary,spoken_word,grammar);
- explicit --modules overrides the default sequence;
- test/injected module factory path remains available for deterministic runtime tests.
Output working transcript to --output file or stdout.
Build process report (phase currently set to phase16-default-pipeline-integration).
Optionally write --report-json; always write run-dir report.json.
Apply work-dir retention.

Parity fixture status:

representative Python-parity fixture coverage exists under internal/cli/testdata/parity;
parity tests use fake structured LLM responses for deterministic behavior, including default full-pipeline shape assertions;
parity comparisons intentionally ignore nondeterministic metadata (timestamps, run IDs, temp paths, token usage) and remain strict for deterministic contract fields (transcript content, module order/instance naming, applied/skipped/rejected counts, and status).
intentional Python-vs-Go differences and open parity gaps are documented in docs/python-parity.md.

Important behavior details:

Glossary is validated and is used for explicit glossary/grammar/homophones/spoken_word module correction paths.
Default production CLI behavior now executes the full production module sequence unless --modules override is supplied.
Explicit --modules grammar, --modules glossary, --modules homophones, and --modules spoken_word continue to run production module paths with LLM-backed proposal generation and validator-chain execution.
Default runs (without explicit module selection) perform LLM calls through production module and validator paths.
Success path is generally quiet on stderr.
Source IDs are preserved into a canonical transcript before normalization; normalization then reassigns output IDs sequentially from 1.

Implemented data contracts

Transcript input

Accepted top-level forms:

bare JSON array of segments
object with segments array

Source segment contract:

id optional integer
speaker non-empty string
start finite non-negative number
end finite non-negative number with end >= start
text non-empty string
categories optional array of non-empty strings

Additional checks:

duplicate explicit source IDs are rejected.

Transcript output

Current output uses schema.TranscriptToJSON and is a bare JSON array of normalized segments:

id, speaker, start, end, text, optional categories.

Glossary input

YAML with glossary entries. Required fields per entry:

name, category, summary

Optional:

aliases, plural

Implemented config/env/flag behavior

Precedence:

defaults (config.Default())
environment (config.LoadFromEnv())
CLI flags (ApplyCLIOverrides)

Implemented config surfaces include:

module list
primary and validation LLM settings
section token controls and target sections
confidence thresholds
normalization controls
work-dir and retention mode

Current caveat:

LLM/module-related settings are active for default and explicit module-run paths.

Implemented structured LLM infrastructure

internal/framework/contracts now defines a typed structured-completion contract:

StructuredLLMClient.CompleteStructured(ctx, req, out)
caller-owned typed decode target via out pointer.

internal/framework/llm provides InstructorClient, an internal adapter over github.com/jxnl/instructor-go:

configurable base_url, model, optional API key, retries, mode, HTTP client, and request timeout;
OpenAI-compatible endpoint behavior (for example OpenAI/OpenRouter/local-compatible base URLs);
default mode is JSON mode (ModeJSON), with optional tool-call mode (ModeToolCall);
request message translation from contracts.LLMMessage to chat-completions messages;
response metadata mapping (provider/model/token usage) into Audita-owned response types;
API-key redaction in adapter-returned errors.

Current runtime boundary:

the default CLI runtime path (without explicit module selection) instantiates the full production module sequence.
LLM calls are exercised in production in both default full-pipeline runs and explicit --modules runs, and in tests when fake/injected clients are used.
normal go test ./... does not require real LLM credentials or Python dependencies.

internal/framework/llm also provides:

a bounded Scheduler for controlled concurrent LLM calls with reliable permit release;
primary/validation effective-config resolution helpers, including validation inheritance fallback to primary settings;
generic interaction diagnostics primitives that write machine-readable JSON artifacts for request metadata, request payload, response payload, and optional error payload with secret redaction.

Implemented normalization behavior

Normalization (internal/core/normalization) currently:

sorts by segment start time;
merges adjacent same-speaker segments when constraints pass;
uses gap-based joiners:
- gap < ellipsis_gap -> single space join
- gap >= ellipsis_gap -> ... join
enforces merged duration and token-limit constraints;
reassigns output IDs sequentially from 1;
returns NormalizationSummary with merge and skip counters.

Note: merged categories are concatenated (not deduplicated).

Implemented chunking behavior

Chunking (internal/core/chunking) currently provides:

deterministic heuristic token estimation;
contiguous sectioning with section metadata;
max/min section token validation;
optional target_sections handling with target-aware merge/split logic;
summary and detailed summary generation.

Current behavior details:

if a single segment exceeds max tokens, it is emitted as its own section (not hard-failed);
section balancing is deterministic but heuristic.

Implemented proposal/replacement infrastructure

internal/framework/proposals provides deterministic foundation logic:

CorrectionProposal and EnrichedCorrectionProposal models;
replacement policies: require_unique, replace_all;
safe preview (PreviewProposalForSegment) with stable skip reasons;
deterministic apply (ApplyProposals) in ascending proposal_index order;
applied/skipped change records suitable for reporting.

internal/framework/contracts provides interfaces and run-spec metadata scaffolding, including deterministic repeated module instance naming (ResolveModuleRunSpecs).

These primitives are wired into the production runner and report model. The grammar, glossary, homophones, and spoken_word modules are implemented.

Implemented validator runtime infrastructure

internal/framework/validators provides deterministic validator infrastructure:

runtime validation request/result models;
stable validator reason codes;
cardinality enforcement for validator decisions:
- missing proposal indexes fail
- duplicate proposal indexes fail
- unknown proposal indexes fail
deterministic validators:
- confidence threshold by module key/config threshold
- original-text presence against current working transcript
- non-empty corrected text
- identical/no-effect rejection
- conservative protected glossary-term guard for non-glossary modules

internal/framework/runner executes validator chains in order for each module and applies only validator-approved proposals. Validator rejections are reported distinctly from proposal-application skips.

Implemented LLM-backed validator infrastructure

internal/framework/validators now includes LLM-backed validator support:

typed request/response models for structured LLM validation;
prompt builders for:
- spoken-form plausibility
- meaning reversal detection
- editorial review
- grammar review
- spoken-word review
deterministic batching by validation_max_prompt_tokens;
strict cardinality validation of structured LLM decisions (missing/duplicate/unknown indexes fail);
safe failure behavior for malformed/invalid structured responses.

internal/framework/runner wires LLM validators into existing validator chains using:

the internal structured LLM client abstraction (contracts.StructuredLLMClient);
bounded scheduler hooks for validator call execution;
diagnostics writer hooks for machine-readable prompt/response artifacts with secret redaction.

Implemented shared proposal-generation infrastructure

internal/framework/proposal_generation provides a reusable, prompt-agnostic helper for future real modules:

structured request model including module key/instance, replacement policy, working transcript context, optional section metadata, glossary, config, and diagnostics context;
structured correction-set response model (corrections) mapped into existing proposals.CorrectionProposal and proposals.EnrichedCorrectionProposal models;
deterministic proposal-index assignment through a caller-provided start_index;
structured LLM calls through contracts.StructuredLLMClient only (no direct provider calls);
optional bounded execution through scheduler hooks (contracts.LLMScheduler);
prompt/response diagnostics artifact writing via the generic internal/framework/llm diagnostics primitives with redaction of API keys/secrets.

This helper only produces candidate proposals; validator-chain execution and proposal application remain runner responsibilities.

Implemented production module-registry scaffolding

internal/framework/modules now provides a production registry scaffold:

recognizes intended module keys:
- glossary
- homophones
- spoken_word
- grammar
supports explicit constructor registration with dependency injection for:
- run spec
- config
- glossary
- proposal/validation structured LLM clients
- proposal/validation schedulers
- diagnostics directory context
returns explicit errors for unknown keys (unsupported_module) and recognized-but-unimplemented keys (unimplemented_module).

The grammar, glossary, homophones, and spoken_word module keys are now registered and constructible.

Implemented grammar production module

internal/modules/grammar now provides the first production module:

prompt builder faithfully constrained to punctuation/capitalization/spacing/article cleanup;
explicit guardrails against meaning-changing rewrites, style rewrites, summarization, and invention;
proposal generation through internal/framework/proposal_generation and contracts.StructuredLLMClient;
scheduler-aware proposal calls through existing contracts.LLMScheduler hooks;
replacement policy require_unique (matching Python implementation);
validator chain integration using existing deterministic + LLM-backed validators;
grammar confidence threshold enforcement through existing validator/config infrastructure;
module-level reporting and diagnostics capture through existing runner/reporting paths.

Implemented glossary production module

internal/modules/glossary now provides the second production module:

prompt builder aligned to Python glossary-module intent, constrained to glossary-backed domain/acoustic corrections;
prompt context includes glossary names, aliases, categories, summaries, and plural forms where available;
guardrails against broad style rewriting and against replacing unrelated terms simply because they appear in glossary entries;
proposal generation through internal/framework/proposal_generation and contracts.StructuredLLMClient;
scheduler-aware proposal calls through existing contracts.LLMScheduler hooks;
replacement policy replace_all (matching Python glossary behavior);
validator chain integration using existing deterministic + LLM-backed validators;
glossary confidence threshold enforcement through existing validator/config infrastructure;
module-level reporting and diagnostics capture through existing runner/reporting paths;
explicit support for repeated glossary stages with deterministic instance names (glossary_1, glossary_2, ...), where later stages see prior-stage working transcript changes.

Implemented protected-term behavior

internal/framework/validators/protected_terms.go provides deterministic glossary-derived protected vocabulary:

extracts protected terms from glossary names and aliases;
includes explicit plural fields and synthetic plural forms where safe;
deduplicates and returns stable ordering for repeatable behavior/tests.

This vocabulary is used by deterministic validators for both glossary-stage and non-glossary-stage protection checks, keeping protected-term guardrails active across modules.

Implemented homophones production module

internal/modules/homophones now provides the third production module:

prompt builder aligned to Python homophones-module intent, constrained to conservative homophone/near-homophone/mistranscription corrections;
prompt context includes protected glossary names/aliases/plurals to avoid damaging known terms;
explicit guardrails against punctuation cleanup, grammar cleanup, style rewriting, summarization, and content invention;
proposal generation through internal/framework/proposal_generation and contracts.StructuredLLMClient;
scheduler-aware proposal calls through existing contracts.LLMScheduler hooks;
replacement policy require_unique (matching Python homophones behavior);
validator chain integration using existing deterministic + LLM-backed validators;
homophones confidence threshold enforcement through existing validator/config infrastructure;
protected-term guardrails for non-glossary modules remain active and are exercised through the homophones path;
module-level reporting and diagnostics capture through existing runner/reporting paths.

Implemented spoken_word production module

internal/modules/spoken_word now provides the fourth production module:

prompt builder aligned to Python spoken_word-module intent, constrained to conservative dysfluency cleanup;
strong prompt guardrails preserving meaning/intent/voice/named entities/domain terms and substantive content;
explicit guardrails against summarization, style rewriting, grammar-only cleanup, punctuation-only cleanup, invention, and meaning-changing rewrites;
proposal generation through internal/framework/proposal_generation and contracts.StructuredLLMClient;
scheduler-aware proposal calls through existing contracts.LLMScheduler hooks;
replacement policy require_unique (matching Python spoken_word behavior);
validator chain integration using existing deterministic + LLM-backed validators, including strong semantic guardrails (spoken_word_review, meaning_reversal_review);
spoken_word confidence threshold enforcement through existing validator/config infrastructure;
protected-term guardrails for non-glossary modules remain active and are exercised through the spoken_word path;
module-level reporting and diagnostics capture through existing runner/reporting paths.

Reports and diagnostics (implemented)

Current per-run artifacts include:

source-transcript.json
source-transcript-parsed.json
normalized-transcript.json
normalization-summary.json
chunking-summary.json
invocation.json
effective-config.json (redacted credentials)
report.json
error.log on failure

--report-json writes a separate report file when requested.

Current process reports include diagnostics metadata references for:

diagnostics directory path;
source transcript artifact path;
parsed source transcript artifact path;
normalized transcript artifact path;
normalization summary artifact path;
chunking summary artifact path;
invocation metadata artifact path;
redacted effective-config artifact path;
error-log artifact path on failure.

Current process reports also include:

module-level results (when runner modules execute), including applied/skipped proposal changes;
run-level module summary totals and failed module instance metadata.
module-level validator decisions and validator rejections.
optional decision-level diagnostic artifact paths for validator LLM interactions when available.

Retention modes implemented in ApplyRetention:

always: keep all run directories.
never: keep successful run directories.
auto: keep failed runs and successful runs with skipped corrections.
failed runs are always retained.

Current runtime note:

default non-explicit runs usually have no module-level skipped corrections, so auto commonly removes clean successful run directories.
explicit grammar/glossary/homophones/spoken_word runs can produce validator rejections and application skips, which are reflected in reports and retention input.

Intentionally deferred to module/LLM phases:

real domain proposal prompts and production module implementations remain tied to later module phases.

Current tests and quality posture

Implemented tests currently cover:

CLI argument handling and behavior (internal/cli/run_test.go)
subprocess stdout/stderr and exit-code behavior (cmd/audita/main_integration_test.go)
config/env/override validation (internal/core/config/*_test.go)
transcript and glossary schema validation (internal/core/schema/*_test.go)
deterministic normalization (internal/core/normalization/*_test.go)
deterministic chunking and summaries (internal/core/chunking/*_test.go)
proposal preview/apply semantics (internal/framework/proposals/*_test.go)
contracts/foundation composition tests (internal/framework/contracts/*_test.go)
runner sequencing and failure behavior with deterministic fake modules (internal/framework/runner/*_test.go)
CLI runner integration through injected fake module factories (internal/cli/run_test.go)
validator models, cardinality enforcement, and deterministic validators (internal/framework/validators/*_test.go)
LLM-backed validator batching, prompt builders, structured-response safety, scheduler hooks, and diagnostics redaction (internal/framework/validators/*_test.go, internal/framework/runner/*_test.go)
shared proposal-generation request/response parsing, deterministic indexing, scheduler hooks, and diagnostics redaction (internal/framework/proposal_generation/*_test.go, internal/framework/runner/*_test.go)
production module-registry known-key recognition and unsupported/unimplemented error behavior (internal/framework/modules/*_test.go, internal/cli/run_test.go)
production grammar module prompt constraints, proposal mapping, validator-chain behavior, confidence-threshold enforcement, diagnostics redaction, and explicit CLI/runtime integration (internal/modules/grammar/*_test.go, internal/cli/run_test.go, internal/framework/runner/*_test.go)
production glossary module prompt constraints, proposal mapping, validator-chain behavior, confidence-threshold enforcement, diagnostics redaction, repeated-stage behavior, and explicit CLI/runtime integration (internal/modules/glossary/*_test.go, internal/cli/run_test.go, internal/framework/runner/*_test.go)
production homophones module prompt constraints, proposal mapping, validator-chain behavior, confidence-threshold enforcement, diagnostics redaction, protected-term behavior, and explicit CLI/runtime integration (internal/modules/homophones/*_test.go, internal/cli/run_test.go, internal/framework/runner/*_test.go)
production spoken_word module prompt constraints, proposal mapping, validator-chain behavior, semantic guardrail behavior, confidence-threshold enforcement, diagnostics redaction, protected-term behavior, and explicit CLI/runtime integration (internal/modules/spoken_word/*_test.go, internal/cli/run_test.go, internal/framework/runner/*_test.go)
glossary-derived protected-term extraction and stable behavior (internal/framework/validators/protected_terms_test.go)
default full-pipeline runtime shape and ordering (internal/cli/run_test.go, cmd/audita/main_integration_test.go, internal/cli/parity_test.go)
subprocess operational hardening behavior including large-input, failure-mode, timeout/cancellation, backend-failure, and partial-progress paths (cmd/audita/main_integration_test.go)
report/diagnostics redaction and artifact-shape behavior across success and failure paths (internal/cli/run_test.go, cmd/audita/main_integration_test.go)

Operational hardening status (Phase 18)

The runtime now includes hardened subprocess behavior for parent-process callers:

deterministic success/failure exit codes;
strict stdout/stderr separation suitable for machine orchestration;
failure stderr summaries that include diagnostics location when available;
retained failure diagnostics (report.json, error.log, and artifacts written before failure);
deterministic timeout/cancellation behavior in tests;
redaction coverage for API keys/secrets across reports, diagnostics artifacts, and surfaced errors.

Operational caller guidance is documented in docs/subprocess-operations.md.

Remaining work (Phase 19)

Remaining rewrite work is documentation/rollout/retirement-oriented:

repository-level rollout documentation and migration guidance;
Python retirement/archival decisions and related documentation cleanup;
final release-facing communication of Go-primary operational guidance.

27 KiB Raw Blame History