Migrate internal architecture docs to docs/internal

This commit is contained in:
2026-05-24 01:02:47 +00:00
parent 6344fc91ba
commit e5944f9875
15 changed files with 581 additions and 633 deletions

View File

@@ -1,14 +1,16 @@
# Audita Architecture Index
This file is the entrypoint for architecture documentation.
This file is the architecture documentation entrypoint.
Core architecture overview:
- [Architecture Overview](./architecture/architecture.md)
Implemented internals:
- [Internal Overview](./internal/overview.md)
- [Pipeline](./internal/pipeline.md)
- [Modules](./internal/modules.md)
- [Validators](./internal/validators.md)
- [LLM Runtime](./internal/llm-runtime.md)
- [Diagnostics and Reporting](./internal/diagnostics-reporting.md)
- [Prompts](./internal/prompts.md)
- [Output Schemas](./internal/output-schemas.md)
Focused architecture contracts:
External/runtime contract summary:
- [Public Contract](./architecture/public-contract.md)
- [Diagnostics](./architecture/diagnostics.md)
- [Structured LLM](./architecture/structured-llm.md)
- [Validators](./architecture/validators.md)
- [Prompts](./architecture/prompts.md)
- [Output Schemas](./architecture/output-schemas.md)

View File

@@ -1,153 +1,5 @@
# Audita Architecture
# Moved: Internal Overview
## Scope
This document describes the production architecture implemented in this repository today.
Implemented internal architecture docs now live in [`docs/internal/`](../internal/).
Audita is a single-process Go CLI that:
- loads effective runtime configuration;
- reads transcript and glossary inputs;
- normalizes and sections transcripts;
- runs a built-in module pipeline with validator chains;
- writes transcript output and run diagnostics.
## Runtime entrypoints
Primary CLI commands:
- `audita process <transcript.json> --glossary <glossary.yaml> [flags]`
- `audita config validate --config <config.yml>`
- `audita config print-effective [--config <config.yml>]`
Command ownership lives in `internal/cli/run.go`.
## Configuration model
`internal/core/config` owns defaults, file parsing, environment overrides, CLI overrides, and validation.
Effective-config loading for `process` and `config print-effective` is centralized in:
- `ResolveConfigPath`
- `LoadEffectiveConfig`
Effective precedence for `audita process`:
1. defaults
2. config file
3. environment overrides
4. CLI overrides
`audita config validate` is intentionally file-only validation:
- load versioned file;
- apply onto defaults;
- validate;
- do not apply environment overrides.
Supported module and output-schema keys are validated through shared catalogs:
- module keys: `internal/core/modulecatalog`
- output schemas: `internal/core/outputschema`
## Pipeline and module orchestration
The built-in module sequence is configured in runtime config and executed by `internal/framework/runner` through resolved module specs.
Current default sequence:
- `glossary`
- `homophones`
- `glossary`
- `spoken_word`
- `grammar`
Execution behavior:
- modules execute serially over the working transcript;
- section proposal work can run concurrently within a module;
- validator execution happens on generated proposals before application;
- approved proposals are applied once per module in deterministic proposal-index order.
Production modules remain separate packages:
- `internal/modules/glossary`
- `internal/modules/homophones`
- `internal/modules/spoken_word`
- `internal/modules/grammar`
## Proposal generation and prompt context
Shared proposal plumbing is centralized in `internal/framework/proposal_generation`.
Module packages provide:
- module identity and replacement policy;
- module-specific prompt message building;
- built-in validator chain selection.
Shared prompt payload helpers are in `internal/framework/promptcontext`.
## Validator architecture
Built-in validator construction and chain composition live in `internal/validators`.
Shared validator runtime mechanics live in `internal/framework/validators`.
Execution class metadata (deterministic vs LLM-backed) is centralized in `internal/validators/metadata` and used for ordering and reporting classification.
## Structured LLM boundary
All production LLM calls go through the internal contract:
- `contracts.StructuredLLMClient`
- `CompleteStructured(ctx, req, out)`
The OpenAI-compatible HTTP adapter is implemented in `internal/framework/llm`.
Structured response schemas are registered in `internal/framework/responseschema` and attached to requests via `response_format` metadata.
Malformed structured-output detection is centralized in `internal/framework/structuredoutput` and reused by proposal generation and validator execution so downgrade behavior stays consistent.
## Stage naming and diagnostics metadata
Diagnostics stage naming is centralized in `internal/framework/stagename`:
- module proposal stage names;
- proposal-generation stage names;
- validator batch stage names.
Prompt metadata and response-schema metadata each expose canonical diagnostics maps via:
- `prompts.Metadata.DiagnosticsMap()`
- `responseschema.Schema.DiagnosticsMap()`
## Diagnostics and reporting
Run-directory artifacts are owned by `internal/core/diagnostics`.
Stable artifact names are centralized constants (for example transcript artifacts, `invocation.json`, `effective-config.json`, `utilization-diagnostics.json`, `correction-ledger.json`, `report.json`, `error.log`).
Report diagnostics path metadata is constructed through `BuildDiagnosticsMetadata`, which keeps run-directory artifact references consistent between success and failure reports.
## Secret redaction
Redaction responsibilities are split by concern:
- structural config redaction: `config.Config.Redacted()`
- byte/string payload redaction for diagnostics and surfaced errors: framework redaction utilities.
Configured LLM secret extraction is centralized in `llm.ConfiguredSecrets(cfg)` and reused across proposal and validator diagnostics paths.
## Output contracts
Transcript output schema selection is owned by `internal/core/outputschema`.
Supported schemas:
- `bare-segments`
- `audita-v1`
Unknown schema keys fail validation and runtime resolution.
## Key package map
Core packages:
- `internal/core/config`
- `internal/core/schema`
- `internal/core/normalization`
- `internal/core/chunking`
- `internal/core/diagnostics`
- `internal/core/reporting`
- `internal/core/modulecatalog`
- `internal/core/outputschema`
Framework packages:
- `internal/framework/contracts`
- `internal/framework/proposals`
- `internal/framework/proposal_generation`
- `internal/framework/promptcontext`
- `internal/framework/runner`
- `internal/framework/validators`
- `internal/framework/llm`
- `internal/framework/responseschema`
- `internal/framework/stagename`
- `internal/framework/structuredoutput`
Domain packages:
- `internal/modules/*`
- `internal/validators/*`
- `internal/prompts`
Start with [`docs/internal/overview.md`](../internal/overview.md).

View File

@@ -1,104 +1,5 @@
# Audita Diagnostics
# Moved: Diagnostics and Reporting
This document describes the run-directory diagnostics artifacts produced by `audita process`.
## Purpose
Diagnostics provide machine-readable run context and execution artifacts for:
- failure debugging;
- validator/correction review;
- post-run performance analysis.
Diagnostics are written under the configured work directory (`--work-dir`) when run-directory initialization succeeds.
## Core artifacts
Typical artifacts in each run directory:
- `source-transcript.json`
- `source-transcript-parsed.json`
- `normalized-transcript.json`
- `normalization-summary.json`
- `chunking-summary.json`
- `invocation.json`
- `effective-config.json` (redacted)
- module/validator LLM interaction artifacts
- `report.json`
- `error.log` on failure
## Utilization diagnostics artifact
Artifact:
- `utilization-diagnostics.json`
High-level fields:
- `effective_concurrency`:
- total/proposal/validation LLM concurrency limits in effect.
- `run_timing`:
- run wall time;
- scheduler queue wait time;
- LLM execution time;
- deterministic validator time;
- max/average in-flight LLM calls.
- `llm_calls`:
- total proposal and validation LLM call counts.
- `modules`:
- module-level timing summaries.
- `validators`:
- per-validator timing summaries keyed by stable validator key.
## Correction ledger artifact
Artifact:
- `correction-ledger.json`
Ledger records are flattened review entries derived from module results and include:
- module/proposal identity (`module_key`, `module_instance`, `proposal_index`, `segment_id`);
- correction text fields and replacement policy when available;
- disposition:
- `applied`
- `rejected`
- `skipped`
- `failed`
- stable reason codes/messages;
- deterministic and LLM validator decision snapshots using stable validator keys.
Validator rejection and proposal-application skip are distinct dispositions.
Module warnings are reported in module results and diagnostics metadata, but do not create standalone correction-ledger rows.
## Report references
`report.json` and optional `--report-json` output include diagnostics metadata paths for:
- utilization diagnostics artifact;
- correction ledger artifact;
- existing transcript/normalization/chunking/invocation/effective-config artifacts.
Module report entries also include warning records for malformed proposal-generation payloads and malformed validator batches.
## Retention behavior
Run-directory retention follows configured policy:
- `always`: keep all run directories;
- `never`: keep successful run directories;
- `auto`: keep failed runs and successful runs with skipped/rejected corrections.
## Redaction guarantees
API keys and other configured secrets are redacted from:
- `effective-config.json`;
- LLM interaction diagnostics artifacts;
- reports and surfaced errors.
## Debugging guide
When debugging:
- slow runs:
- inspect `utilization-diagnostics.json` (`run_timing`, `modules`, `validators`, in-flight metrics).
- validator rejections:
- inspect `correction-ledger.json` rejected entries and matching validator decisions;
- inspect validator response diagnostics payloads.
- module warnings:
- inspect module `warnings` entries in `report.json` or `--report-json`;
- follow any diagnostic artifact path on the warning to the recorded error/response payload.
- application skips:
- inspect `correction-ledger.json` skipped entries and skip reason codes;
- compare with validator decisions to distinguish validation rejection vs apply-time skip.
Implemented diagnostics and reporting internals now live at:
- [`docs/internal/diagnostics-reporting.md`](../internal/diagnostics-reporting.md)
- [`docs/operations.md`](../operations.md)

View File

@@ -1,88 +1,3 @@
# Audita Output Schemas
# Moved: Output Schemas
This document describes the built-in transcript output schema registry used by `audita process`.
## Supported schema names
### `bare-segments`
Status:
- implemented
- default output schema
Shape:
- top-level JSON array of transcript segments
Segment fields:
- `id`
- `speaker`
- `start`
- `end`
- `text`
- optional `categories`
Compatibility:
- this preserves the long-standing output shape used by existing consumers.
### `audita-v1`
Status:
- implemented
Shape:
- top-level JSON object:
- `schema`: `"audita-v1"`
- `version`: `"v1"`
- `segments`: transcript segment array
Segment fields inside `segments` match `bare-segments` segment fields.
Compatibility:
- this is the Audita-native object format with explicit schema/version metadata.
### `seriatim-intermediate`
Status:
- deferred / not implemented
Current behavior:
- selecting `seriatim-intermediate` fails clearly as an unsupported output schema.
Reason:
- a concrete, repository-backed contract for this schema has not been finalized yet.
## Selection
Choose output schema with CLI:
```sh
audita process <transcript.json> --glossary <glossary.yaml> --output-schema audita-v1
```
Or in file config:
```yaml
version: 1
output:
schema: audita-v1
```
Precedence remains:
1. defaults
2. file config
3. environment overrides
4. CLI overrides
`--output-schema` overrides `output.schema` when both are supplied.
## Output routing behavior
- With `--output`, transcript JSON is written to file using the selected schema and stdout stays empty on success.
- Without `--output`, stdout contains transcript JSON only, using the selected schema.
- `--report-json` writes report JSON to file and does not write report payloads to stdout.
## Backward-compatibility expectations
- default schema stays `bare-segments` for compatibility unless explicitly changed in a future breaking release;
- supported schema names are treated as stable public contract values;
- unsupported schema names fail before output write.
Implemented output schema registry docs now live at [`docs/internal/output-schemas.md`](../internal/output-schemas.md).

View File

@@ -1,118 +1,3 @@
# Audita Prompts
# Moved: Prompt Registry
This document describes Audita's built-in embedded prompt assets and prompt registry behavior.
## Why embedded prompt assets
Audita embeds production prompt text into the binary so runtime behavior is:
- deterministic;
- auditable;
- dependency-light;
- not dependent on external prompt files at execution time.
Prompt text is authored as Markdown assets and rendered by Go code using typed template data.
## Built-in prompt registry
The prompt registry lives in `internal/prompts` and is responsible for:
- loading embedded prompt assets;
- registering stable prompt IDs and versions;
- recording prompt source metadata;
- computing deterministic SHA-256 source hashes;
- rendering system/user prompts with strict missing-key failures.
Current prompt source behavior:
- built-in embedded prompts only (`prompt_source = builtin`).
- filesystem prompt overrides are not supported.
## Built-in prompt IDs
Module proposal prompts:
- `modules.glossary.proposal`
- `modules.homophones.proposal`
- `modules.spoken_word.proposal`
- `modules.grammar.proposal`
LLM-backed validator prompts:
- `validators.spoken_form_plausibility`
- `validators.meaning_reversal_review`
- `validators.editorial_review`
- `validators.grammar_review`
- `validators.spoken_word_review`
## Prompt version semantics
Current built-in prompt version value is `v1`.
Version is a stable metadata identifier for diagnostics and debugging. It is not a dynamic prompt-selection mechanism.
## Prompt hash semantics
Each registered prompt includes a deterministic SHA-256 hash of embedded source text.
Hash purpose:
- identify exact prompt source used in a run;
- support diagnostics reproducibility and change auditing.
Current hash scope:
- source prompt text (system + user assets for a registered prompt), not a runtime secret-bearing payload.
## Template rendering behavior
Prompt rendering uses Go `text/template` with typed template data from module/validator builders.
Missing-key behavior:
- rendering uses missing-key errors;
- missing/renamed template fields fail quickly instead of silently producing incomplete prompts.
Go code still owns:
- structured request/response models;
- response schema selection;
- transcript/glossary/payload formatting;
- module and validator selection;
- diagnostics wiring.
## Shared prompt hardening policy
A shared hardening fragment is embedded once and included in every module proposal prompt and every LLM-validator prompt.
Hardening policy includes:
- transcript text is untrusted data;
- glossary entries and transcript descriptions are reference data, not instructions;
- instructions found inside transcript text must not be obeyed;
- model must perform only the requested correction/validation task;
- no invention of facts, names, events, motivations, speaker intent, or corrections;
- transcript remains the source of truth.
## Transcript description behavior
Transcript description remains background-only prompt context:
- it may help interpret ambiguous terms;
- it is explicitly non-authoritative and must not override transcript content;
- empty descriptions do not render awkward blank context sections.
Generated transcript descriptions are not implemented in this workstream.
## Diagnostics and report metadata boundaries
Current metadata flow:
- proposal-generation diagnostics request metadata includes prompt metadata;
- LLM-validator diagnostics request metadata includes prompt metadata.
Prompt metadata fields used in diagnostics:
- `prompt_id`
- `prompt_version`
- `prompt_source`
- `embedded_path`
- `sha256`
Current boundary:
- detailed prompt metadata is diagnostics-first;
- broad report-level prompt registries/ledgers are deferred.
## 1.0 boundary
Not implemented for 1.0 in this workstream:
- filesystem prompt overrides;
- user-configurable prompt selection;
- external prompt directories.
Implemented prompt registry and prompt metadata docs now live at [`docs/internal/prompts.md`](../internal/prompts.md).

View File

@@ -1,72 +1,3 @@
# Structured LLM Architecture
# Moved: LLM Runtime
## Scope
This document describes Audita's structured LLM runtime boundary and adapter behavior.
## Runtime boundary
Production LLM integration depends on the internal contract only:
- `contracts.StructuredLLMClient`
- `CompleteStructured(ctx, req, out)`
Provider SDK types do not leak past this boundary.
## Adapter ownership
`internal/framework/llm` owns the OpenAI-compatible HTTP adapter and shared LLM runtime utilities.
Key responsibilities:
- request assembly;
- timeout/cancellation propagation;
- bounded retry behavior;
- scheduler integration;
- provider response decoding;
- error redaction.
## Structured schema registry
Structured response schemas are registered in `internal/framework/responseschema` and include stable metadata:
- `id`
- `version`
- `name`
- `json_schema`
- `sha256`
Current schema keys:
- `correction_set`
- `validator_decision_set`
Schema metadata is attached to diagnostics through `Schema.DiagnosticsMap()`.
## Request shape assumptions
Audita targets OpenAI-compatible chat-completions endpoints and sends structured requests with:
- model;
- chat messages;
- `response_format.type = json_schema`;
- schema name and JSON schema payload.
## Local validation remains mandatory
Provider schema enforcement is treated as transport-level guardrails.
Audita still validates output locally before applying behavior changes:
- proposal decoding and proposal invariants;
- validator decision decoding and cardinality checks;
- deterministic validation and apply-time rules.
## Shared malformed-output policy
Malformed structured-output classification is centralized in `internal/framework/structuredoutput`.
Proposal generation and validator execution both use this shared classifier so downgrade behavior cannot drift between the two paths.
## Secrets and redaction
Secret extraction for LLM redaction is centralized in `llm.ConfiguredSecrets(cfg)` and reused by proposal and validator diagnostics writers.
Secrets are redacted from:
- diagnostics artifacts;
- report artifacts;
- surfaced adapter/runtime errors.
## Concurrency and scheduling
LLM execution is constrained by composed scheduler limits:
- total LLM concurrency;
- proposal LLM concurrency;
- validation LLM concurrency.
The scheduler is FIFO and context-aware so permits are released on success, failure, and cancellation.
Implemented structured LLM runtime docs now live at [`docs/internal/llm-runtime.md`](../internal/llm-runtime.md).

View File

@@ -1,96 +1,3 @@
# Audita Validators
# Moved: Validators
## Scope
This document defines the built-in validator system used by production module runs.
## Ownership boundaries
Built-in validator keys, constructors, and module chains are owned by `internal/validators`.
Shared runtime execution mechanics are owned by `internal/framework/validators`, including:
- validator request/result models;
- deterministic proposal checks;
- LLM validator batching and execution;
- decision-cardinality enforcement;
- diagnostics integration.
Execution class metadata is owned by `internal/validators/metadata`.
## Stable validator keys
Deterministic:
- `proposal_shape`
- `confidence_threshold`
- `original_text_presence`
- `non_empty_corrected_text`
- `no_effect`
- `protected_terms`
LLM-backed:
- `spoken_form_plausibility`
- `meaning_reversal_review`
- `editorial_review`
## Built-in module chains
`glossary`:
- `proposal_shape`
- `no_effect`
- `original_text_presence`
- `confidence_threshold`
- `protected_terms`
- `non_empty_corrected_text`
- `spoken_form_plausibility`
- `meaning_reversal_review`
`homophones`:
- `proposal_shape`
- `no_effect`
- `original_text_presence`
- `confidence_threshold`
- `protected_terms`
- `non_empty_corrected_text`
- `spoken_form_plausibility`
- `meaning_reversal_review`
`spoken_word`:
- `proposal_shape`
- `no_effect`
- `original_text_presence`
- `confidence_threshold`
- `protected_terms`
- `non_empty_corrected_text`
- `editorial_review`
- `meaning_reversal_review`
`grammar`:
- `proposal_shape`
- `no_effect`
- `original_text_presence`
- `confidence_threshold`
- `protected_terms`
- `non_empty_corrected_text`
- `editorial_review`
- `meaning_reversal_review`
## Ordering and execution semantics
Validator ordering is based on canonical metadata:
- deterministic validators run before LLM-backed validators.
Within each module stage:
- proposals are generated per section;
- validator chains execute on those proposals;
- approved proposals are applied once after section work settles.
## Malformed payload behavior
Malformed structured-output from proposal generation and LLM validator calls is downgraded, not treated as a process-fatal transport error.
Current outcomes:
- malformed proposal-generation payloads produce section/module warnings and zero proposals for the affected section;
- malformed validator decision payloads reject the affected validator batch with warnings;
- deterministic validator behavior and runner order remain unchanged.
## Reporting identity
Reports and diagnostics use stable validator keys as identifiers.
Correction-ledger deterministic-vs-LLM classification is derived from canonical validator metadata, not package-local hardcoded maps.
## Prompt assets
LLM validator prompt assets and prompt metadata are documented in [Prompts](./prompts.md).
Implemented validator architecture docs now live at [`docs/internal/validators.md`](../internal/validators.md).

View File

@@ -0,0 +1,79 @@
# Audita Diagnostics and Reporting
## Scope
This document describes diagnostics artifacts, process report mapping, and correction ledger generation.
## Run Directory Ownership
`internal/core/diagnostics` owns run-directory creation, artifact writes, and retention decisions.
Stable artifact names include:
- `source-transcript.json`
- `source-transcript-parsed.json`
- `normalized-transcript.json`
- `normalization-summary.json`
- `chunking-summary.json`
- `utilization-diagnostics.json`
- `correction-ledger.json`
- `invocation.json`
- `effective-config.json`
- `report.json`
- `error.log` (failure)
## Process Report Mapping
`internal/framework/processreport` maps runner/CLI execution facts into `reporting.ProcessReport`.
Report metadata fields include:
- `report_schema_name` (`audita-process-report`)
- `report_schema_version` (`v1`)
- `output_schema`
- `config_version` (when file config exists)
The report includes:
- top-level status/error phase/error message;
- normalization/chunking summaries;
- diagnostics metadata paths;
- per-module results and module summary.
## Correction Ledger
`internal/framework/processreport/BuildCorrectionLedger` flattens run results into `correction-ledger.json` entries.
Dispositions:
- `applied`
- `skipped`
- `rejected`
- `failed`
Validator decisions are split into deterministic and LLM-backed groups using validator metadata classification.
## Report Write Paths
- run directory always attempts to write `report.json` when possible;
- optional `--report-json` writes an external report file;
- on failure paths, report writing is best-effort and does not mask primary run errors.
## Retention Interaction
Current retention behavior:
- failed runs are retained;
- `always` keeps successful runs;
- `auto` removes only clean successful runs;
- `never` currently retains successful runs in current implementation.
## Redaction
Redacted data expectations:
- effective config artifact uses config redaction;
- diagnostics payloads and surfaced errors use LLM secret redaction;
- reports should not include raw API key values.
## Key Tests
- `internal/core/diagnostics/*_test.go`
- `internal/framework/processreport/*_test.go`
- `internal/core/reporting/report_test.go`
- `internal/cli/run_test.go`
- `cmd/audita/main_integration_test.go`

View File

@@ -0,0 +1,67 @@
# Audita LLM Runtime
## Scope
This document describes the structured LLM runtime and scheduler behavior.
## Client Boundary
All runtime LLM calls go through `contracts.StructuredLLMClient`.
Primary adapter:
- `internal/framework/llm/OpenAICompatibleClient`
## Request/Response Behavior
The OpenAI-compatible adapter sends chat completions requests with:
- model;
- messages;
- `response_format.type = json_schema`;
- strict schema envelope (`name`, `schema`, `strict=true`).
The response is decoded into the requested structured output target.
## Response Schema Registry
Structured response schemas are registered in `internal/framework/responseschema`:
- `correction_set`
- `validator_decision_set`
Each schema includes stable diagnostics metadata (`id`, `version`, `name`, `sha256`).
## Retries and Error Handling
Adapter retries apply to retryable conditions (for example transport/decoding/retryable status classes) up to configured `max_retries`.
Errors are sanitized to redact configured API-key values before surfacing.
Malformed structured output detection is shared through `internal/framework/structuredoutput` and is used by:
- proposal generation;
- LLM-backed validators.
## Scheduling and Concurrency
`internal/framework/llm/Scheduler` provides FIFO, context-aware permit gating.
Runner composes scheduler limits across:
- total LLM concurrency;
- proposal LLM concurrency;
- validation LLM concurrency.
Scheduler release is guarded to avoid permit leaks on cancellation/error.
## Diagnostics and Redaction
`internal/framework/llm/DiagnosticsWriter` writes request/response/error artifacts.
Configured secrets are derived from `llm.ConfiguredSecrets(cfg)` and redacted from:
- diagnostics payloads;
- surfaced runtime/adapter errors.
## Key Tests
- `internal/framework/llm/openai_compatible_client_test.go`
- `internal/framework/llm/scheduler_test.go`
- `internal/framework/llm/diagnostics_test.go`
- `internal/framework/responseschema/registry_test.go`
- `internal/framework/structuredoutput/malformed_test.go`

58
docs/internal/modules.md Normal file
View File

@@ -0,0 +1,58 @@
# Audita Modules
## Scope
This document covers module contracts and built-in module packages.
## Module Contract
Modules implement `contracts.TranscriptModule`:
- `Key()`
- `ReplacementPolicy()`
- `Validators()`
- `Propose(ctx, req)`
Runner resolves configured module specs to module instances through `internal/framework/modules`.
## Built-In Modules
Current module packages:
- `internal/modules/glossary`
- `internal/modules/homophones`
- `internal/modules/spoken_word`
- `internal/modules/grammar`
Current replacement policies:
- `glossary`: `replace_all`
- `homophones`: `require_unique`
- `spoken_word`: `require_unique`
- `grammar`: `require_unique`
## Proposal Generation Ownership
Shared proposal-generation plumbing is centralized in:
- `internal/framework/proposal_generation`
Module packages own:
- prompt selection (`internal/prompts` prompt IDs);
- module-specific prompt payload construction.
Shared prompt helpers live in `internal/framework/promptcontext`.
## Validator Chain Ownership
Built-in chains are resolved in `internal/validators` per module key.
Module packages call the built-in chain resolver at construction.
## Failure and Warning Behavior
- module setup failures surface as `runner_setup` or module setup errors;
- module runtime failures surface as `runner_execution` with partial module results preserved;
- malformed structured proposal payloads are downgraded to warnings and section-level proposal rejection.
## Key Tests
- `internal/modules/*/module_test.go`
- `internal/framework/modules/registry_test.go`
- `internal/framework/proposal_generation/*_test.go`
- `internal/cli/run_test.go` (pipeline/report integration)

View File

@@ -0,0 +1,46 @@
# Audita Output Schemas
## Scope
This document describes the implemented transcript output schema registry.
## Registry Ownership
Output schema registry is owned by `internal/core/outputschema`.
Supported schema keys:
- `bare-segments`
- `audita-v1`
## Schemas
`bare-segments`:
- top-level JSON array of transcript segments.
`audita-v1`:
- top-level JSON object with:
- `schema: "audita-v1"`
- `version: "v1"`
- `segments: [...]`
Segment fields include `id`, `speaker`, `start`, `end`, `text`, and optional `categories`.
## Validation and Resolution
Config validation and runtime resolution both reject unsupported schema keys.
Unknown schema keys fail with `unsupported output schema` before output emission.
## Output Emission
The selected schema is used by `audita process` when writing:
- output file (`--output`) or
- stdout (when no `--output`).
Report metadata records selected `output_schema`.
## Key Tests
- `internal/core/outputschema/registry_test.go`
- `internal/core/config/config_test.go`
- `internal/cli/run_test.go`

92
docs/internal/overview.md Normal file
View File

@@ -0,0 +1,92 @@
# Audita Internal Overview
## Scope
This document is the internal architecture entry point for developers and coding agents.
It summarizes:
- package boundaries;
- the main `process` execution path;
- where to add new code safely.
## Package Map
CLI and command orchestration:
- `internal/cli`
Core deterministic components:
- `internal/core/config`
- `internal/core/schema`
- `internal/core/normalization`
- `internal/core/chunking`
- `internal/core/outputschema`
- `internal/core/diagnostics`
- `internal/core/reporting`
- `internal/core/modulecatalog`
Framework orchestration and contracts:
- `internal/framework/contracts`
- `internal/framework/modules`
- `internal/framework/proposal_generation`
- `internal/framework/proposals`
- `internal/framework/runner`
- `internal/framework/validators`
- `internal/framework/llm`
- `internal/framework/responseschema`
- `internal/framework/structuredoutput`
- `internal/framework/processreport`
- `internal/framework/promptcontext`
- `internal/framework/stagename`
Domain implementations:
- `internal/modules/*`
- `internal/validators/*`
- `internal/prompts`
## Main Execution Path (`audita process`)
High-level flow:
1. CLI loads effective config and validates CLI requirements.
2. Run directory is created and invocation/effective config artifacts are written.
3. Transcript/glossary files are loaded and parsed.
4. Transcript is normalized and chunked.
5. `runner.Run` executes configured module instances.
6. Proposals are validated, applied deterministically, and serialized in selected output schema.
7. Process report, utilization diagnostics, correction ledger, and retention decisions are finalized.
## Boundary Summary
- `internal/core/*` owns deterministic, reusable logic and persistence-independent rules.
- `internal/framework/*` owns orchestration contracts and reusable runtime plumbing.
- `internal/modules/*` owns module-specific proposal behavior and prompt usage.
- `internal/validators/*` owns validator composition and built-in chain assembly.
- `internal/prompts` owns embedded prompt assets and metadata registry.
## Where To Add New Code
Add config fields:
- `internal/core/config`
Add module behavior:
- one package under `internal/modules/<module_key>`
- registration/wiring through `internal/framework/modules` and config module list
Add validators:
- implementation under `internal/validators/<validator_key>`
- registry/chain wiring in `internal/validators`
Add runtime orchestration behavior:
- `internal/framework/*` (runner/proposal/validator/LLM plumbing)
Add CLI surface:
- `internal/cli`
## Related Internal Docs
- [`docs/internal/pipeline.md`](pipeline.md)
- [`docs/internal/modules.md`](modules.md)
- [`docs/internal/validators.md`](validators.md)
- [`docs/internal/llm-runtime.md`](llm-runtime.md)
- [`docs/internal/diagnostics-reporting.md`](diagnostics-reporting.md)
- [`docs/internal/prompts.md`](prompts.md)
- [`docs/internal/output-schemas.md`](output-schemas.md)

79
docs/internal/pipeline.md Normal file
View File

@@ -0,0 +1,79 @@
# Audita Internal Pipeline
## Scope
This document describes the implemented `audita process` pipeline.
## Inputs
Pipeline inputs are:
- effective config (`internal/core/config`);
- transcript JSON (`internal/core/schema`);
- glossary YAML (`internal/core/schema`).
## Pipeline Phases
1. Input loading and schema validation
- transcript and glossary files are read and parsed.
- schema failures stop the run with `transcript_schema` or `glossary_schema`.
2. Normalization
- canonical transcript segments are normalized by configured gap/duration/token settings.
- normalization summary artifacts are written.
3. Chunking
- normalized transcript is chunked with configured max/min tokens and target sections.
4. Module proposal generation
- runner executes configured module instances in sequence.
- each module proposes corrections per section.
- per-section proposal generation can run concurrently.
5. Validator filtering
- validators run on candidate proposals before apply.
- deterministic validators run before LLM-backed validators.
- LLM validator inputs are batched by max prompt token limit.
6. Deterministic apply
- approved proposals are applied via replacement policy.
- applied/skipped/rejected outcomes are recorded.
7. Output and reporting
- final transcript is serialized with selected output schema.
- report, utilization diagnostics, and correction ledger are written.
- retention policy is applied to run directory.
## Runner Outputs
`runner.Run` returns:
- final transcript;
- per-module results;
- utilization diagnostics.
CLI/reporting then map this into process report and diagnostics artifacts.
## Failure Behavior
Representative failure phases include:
- `run_dir_creation`
- `transcript_read`, `glossary_read`
- `transcript_schema`, `glossary_schema`
- `chunking`
- `runner_setup`, `runner_execution`
- `output_schema`, `serialization`, `output_write`, `stdout_write`
When diagnostics are available, failure stderr includes diagnostics path.
## Invariants
- module execution order follows configured module sequence;
- proposal/validator nondeterminism is isolated before deterministic apply;
- proposal indices are assigned deterministically by section order;
- output/report artifacts are generated from run results, not speculative state.
## Key Tests
- `internal/framework/runner/runner_test.go`
- `internal/framework/proposal_generation/*_test.go`
- `internal/cli/run_test.go`
- `cmd/audita/main_integration_test.go`

62
docs/internal/prompts.md Normal file
View File

@@ -0,0 +1,62 @@
# Audita Prompt Registry
## Scope
This document describes embedded prompt assets, prompt metadata, and rendering behavior.
## Registry Ownership
Prompt registry lives in `internal/prompts` and embeds assets under `internal/prompts/assets/**`.
Registered prompt IDs:
- `modules.glossary.proposal`
- `modules.homophones.proposal`
- `modules.spoken_word.proposal`
- `modules.grammar.proposal`
- `validators.spoken_form_plausibility`
- `validators.meaning_reversal_review`
- `validators.editorial_review`
- `validators.grammar_review`
- `validators.spoken_word_review`
## Metadata Model
Each prompt has metadata:
- `prompt_id`
- `prompt_version`
- `prompt_source`
- `embedded_path`
- `sha256`
Current source/version values:
- `prompt_source = builtin`
- `prompt_version = v1`
## Rendering
`prompts.RenderUserSystem(promptID, data)` renders system/user templates.
Template behavior:
- uses Go `text/template`;
- `missingkey=error` is enabled;
- output is trimmed.
A shared hardening fragment is embedded once and referenced by prompt templates.
## Prompt Context Inputs
Shared prompt payload helpers:
- transcript section JSON (`internal/framework/promptcontext/MarshalTranscriptSectionJSON`)
- transcript description block (`TranscriptDescriptionBlock`)
Modules and LLM validators provide typed data maps to render prompt assets.
## Diagnostics Integration
Prompt metadata is attached to proposal/validator diagnostics request metadata using `Metadata.DiagnosticsMap()`.
## Key Tests
- `internal/prompts/registry_test.go`
- `internal/framework/promptcontext/*_test.go`
- module and validator prompt builder tests

View File

@@ -0,0 +1,72 @@
# Audita Validators
## Scope
This document describes validator composition, execution order, and decision handling.
## Ownership
Built-in validator keys and chains:
- `internal/validators`
Shared validator runtime mechanics:
- `internal/framework/validators`
Execution-class metadata:
- `internal/validators/metadata`
## Built-In Validator Keys
Deterministic:
- `proposal_shape`
- `confidence_threshold`
- `original_text_presence`
- `non_empty_corrected_text`
- `no_effect`
- `protected_terms`
LLM-backed:
- `spoken_form_plausibility`
- `meaning_reversal_review`
- `editorial_review`
## Built-In Chains
Module chains are defined in `internal/validators/chains.go`.
Glossary, homophones, spoken_word, and grammar each resolve a fixed ordered chain.
## Runtime Execution
For each module section:
1. run deterministic validators;
2. run LLM-backed validators;
3. record decisions and warnings;
4. carry only approved proposals forward.
Decision cardinality is enforced: each candidate proposal must receive exactly one decision per validator.
## LLM Validator Batching
LLM validators:
- build canonical validation request payloads;
- batch by `validation_max_prompt_tokens`;
- call structured LLM client using response schema registry.
Oversized single proposals are rejected with `validator_input_too_large`.
Malformed LLM validator responses are downgraded to warnings and rejected batch decisions.
## Decision and Rejection Reporting
Runner records:
- `validator_decisions`
- `validator_rejected`
- warning records (including malformed response warnings)
Correction ledger classifies deterministic vs LLM validator decisions using canonical metadata classes.
## Key Tests
- `internal/validators/*_test.go`
- `internal/framework/validators/*_test.go`
- `internal/framework/processreport/correction_ledger_test.go`
- `internal/cli/run_test.go`