Migrate internal architecture docs to docs/internal

This commit is contained in:
2026-05-24 01:02:47 +00:00
parent 6344fc91ba
commit e5944f9875
15 changed files with 581 additions and 633 deletions

View File

@@ -1,153 +1,5 @@
# Audita Architecture
# Moved: Internal Overview
## Scope
This document describes the production architecture implemented in this repository today.
Implemented internal architecture docs now live in [`docs/internal/`](../internal/).
Audita is a single-process Go CLI that:
- loads effective runtime configuration;
- reads transcript and glossary inputs;
- normalizes and sections transcripts;
- runs a built-in module pipeline with validator chains;
- writes transcript output and run diagnostics.
## Runtime entrypoints
Primary CLI commands:
- `audita process <transcript.json> --glossary <glossary.yaml> [flags]`
- `audita config validate --config <config.yml>`
- `audita config print-effective [--config <config.yml>]`
Command ownership lives in `internal/cli/run.go`.
## Configuration model
`internal/core/config` owns defaults, file parsing, environment overrides, CLI overrides, and validation.
Effective-config loading for `process` and `config print-effective` is centralized in:
- `ResolveConfigPath`
- `LoadEffectiveConfig`
Effective precedence for `audita process`:
1. defaults
2. config file
3. environment overrides
4. CLI overrides
`audita config validate` is intentionally file-only validation:
- load versioned file;
- apply onto defaults;
- validate;
- do not apply environment overrides.
Supported module and output-schema keys are validated through shared catalogs:
- module keys: `internal/core/modulecatalog`
- output schemas: `internal/core/outputschema`
## Pipeline and module orchestration
The built-in module sequence is configured in runtime config and executed by `internal/framework/runner` through resolved module specs.
Current default sequence:
- `glossary`
- `homophones`
- `glossary`
- `spoken_word`
- `grammar`
Execution behavior:
- modules execute serially over the working transcript;
- section proposal work can run concurrently within a module;
- validator execution happens on generated proposals before application;
- approved proposals are applied once per module in deterministic proposal-index order.
Production modules remain separate packages:
- `internal/modules/glossary`
- `internal/modules/homophones`
- `internal/modules/spoken_word`
- `internal/modules/grammar`
## Proposal generation and prompt context
Shared proposal plumbing is centralized in `internal/framework/proposal_generation`.
Module packages provide:
- module identity and replacement policy;
- module-specific prompt message building;
- built-in validator chain selection.
Shared prompt payload helpers are in `internal/framework/promptcontext`.
## Validator architecture
Built-in validator construction and chain composition live in `internal/validators`.
Shared validator runtime mechanics live in `internal/framework/validators`.
Execution class metadata (deterministic vs LLM-backed) is centralized in `internal/validators/metadata` and used for ordering and reporting classification.
## Structured LLM boundary
All production LLM calls go through the internal contract:
- `contracts.StructuredLLMClient`
- `CompleteStructured(ctx, req, out)`
The OpenAI-compatible HTTP adapter is implemented in `internal/framework/llm`.
Structured response schemas are registered in `internal/framework/responseschema` and attached to requests via `response_format` metadata.
Malformed structured-output detection is centralized in `internal/framework/structuredoutput` and reused by proposal generation and validator execution so downgrade behavior stays consistent.
## Stage naming and diagnostics metadata
Diagnostics stage naming is centralized in `internal/framework/stagename`:
- module proposal stage names;
- proposal-generation stage names;
- validator batch stage names.
Prompt metadata and response-schema metadata each expose canonical diagnostics maps via:
- `prompts.Metadata.DiagnosticsMap()`
- `responseschema.Schema.DiagnosticsMap()`
## Diagnostics and reporting
Run-directory artifacts are owned by `internal/core/diagnostics`.
Stable artifact names are centralized constants (for example transcript artifacts, `invocation.json`, `effective-config.json`, `utilization-diagnostics.json`, `correction-ledger.json`, `report.json`, `error.log`).
Report diagnostics path metadata is constructed through `BuildDiagnosticsMetadata`, which keeps run-directory artifact references consistent between success and failure reports.
## Secret redaction
Redaction responsibilities are split by concern:
- structural config redaction: `config.Config.Redacted()`
- byte/string payload redaction for diagnostics and surfaced errors: framework redaction utilities.
Configured LLM secret extraction is centralized in `llm.ConfiguredSecrets(cfg)` and reused across proposal and validator diagnostics paths.
## Output contracts
Transcript output schema selection is owned by `internal/core/outputschema`.
Supported schemas:
- `bare-segments`
- `audita-v1`
Unknown schema keys fail validation and runtime resolution.
## Key package map
Core packages:
- `internal/core/config`
- `internal/core/schema`
- `internal/core/normalization`
- `internal/core/chunking`
- `internal/core/diagnostics`
- `internal/core/reporting`
- `internal/core/modulecatalog`
- `internal/core/outputschema`
Framework packages:
- `internal/framework/contracts`
- `internal/framework/proposals`
- `internal/framework/proposal_generation`
- `internal/framework/promptcontext`
- `internal/framework/runner`
- `internal/framework/validators`
- `internal/framework/llm`
- `internal/framework/responseschema`
- `internal/framework/stagename`
- `internal/framework/structuredoutput`
Domain packages:
- `internal/modules/*`
- `internal/validators/*`
- `internal/prompts`
Start with [`docs/internal/overview.md`](../internal/overview.md).

View File

@@ -1,104 +1,5 @@
# Audita Diagnostics
# Moved: Diagnostics and Reporting
This document describes the run-directory diagnostics artifacts produced by `audita process`.
## Purpose
Diagnostics provide machine-readable run context and execution artifacts for:
- failure debugging;
- validator/correction review;
- post-run performance analysis.
Diagnostics are written under the configured work directory (`--work-dir`) when run-directory initialization succeeds.
## Core artifacts
Typical artifacts in each run directory:
- `source-transcript.json`
- `source-transcript-parsed.json`
- `normalized-transcript.json`
- `normalization-summary.json`
- `chunking-summary.json`
- `invocation.json`
- `effective-config.json` (redacted)
- module/validator LLM interaction artifacts
- `report.json`
- `error.log` on failure
## Utilization diagnostics artifact
Artifact:
- `utilization-diagnostics.json`
High-level fields:
- `effective_concurrency`:
- total/proposal/validation LLM concurrency limits in effect.
- `run_timing`:
- run wall time;
- scheduler queue wait time;
- LLM execution time;
- deterministic validator time;
- max/average in-flight LLM calls.
- `llm_calls`:
- total proposal and validation LLM call counts.
- `modules`:
- module-level timing summaries.
- `validators`:
- per-validator timing summaries keyed by stable validator key.
## Correction ledger artifact
Artifact:
- `correction-ledger.json`
Ledger records are flattened review entries derived from module results and include:
- module/proposal identity (`module_key`, `module_instance`, `proposal_index`, `segment_id`);
- correction text fields and replacement policy when available;
- disposition:
- `applied`
- `rejected`
- `skipped`
- `failed`
- stable reason codes/messages;
- deterministic and LLM validator decision snapshots using stable validator keys.
Validator rejection and proposal-application skip are distinct dispositions.
Module warnings are reported in module results and diagnostics metadata, but do not create standalone correction-ledger rows.
## Report references
`report.json` and optional `--report-json` output include diagnostics metadata paths for:
- utilization diagnostics artifact;
- correction ledger artifact;
- existing transcript/normalization/chunking/invocation/effective-config artifacts.
Module report entries also include warning records for malformed proposal-generation payloads and malformed validator batches.
## Retention behavior
Run-directory retention follows configured policy:
- `always`: keep all run directories;
- `never`: keep successful run directories;
- `auto`: keep failed runs and successful runs with skipped/rejected corrections.
## Redaction guarantees
API keys and other configured secrets are redacted from:
- `effective-config.json`;
- LLM interaction diagnostics artifacts;
- reports and surfaced errors.
## Debugging guide
When debugging:
- slow runs:
- inspect `utilization-diagnostics.json` (`run_timing`, `modules`, `validators`, in-flight metrics).
- validator rejections:
- inspect `correction-ledger.json` rejected entries and matching validator decisions;
- inspect validator response diagnostics payloads.
- module warnings:
- inspect module `warnings` entries in `report.json` or `--report-json`;
- follow any diagnostic artifact path on the warning to the recorded error/response payload.
- application skips:
- inspect `correction-ledger.json` skipped entries and skip reason codes;
- compare with validator decisions to distinguish validation rejection vs apply-time skip.
Implemented diagnostics and reporting internals now live at:
- [`docs/internal/diagnostics-reporting.md`](../internal/diagnostics-reporting.md)
- [`docs/operations.md`](../operations.md)

View File

@@ -1,88 +1,3 @@
# Audita Output Schemas
# Moved: Output Schemas
This document describes the built-in transcript output schema registry used by `audita process`.
## Supported schema names
### `bare-segments`
Status:
- implemented
- default output schema
Shape:
- top-level JSON array of transcript segments
Segment fields:
- `id`
- `speaker`
- `start`
- `end`
- `text`
- optional `categories`
Compatibility:
- this preserves the long-standing output shape used by existing consumers.
### `audita-v1`
Status:
- implemented
Shape:
- top-level JSON object:
- `schema`: `"audita-v1"`
- `version`: `"v1"`
- `segments`: transcript segment array
Segment fields inside `segments` match `bare-segments` segment fields.
Compatibility:
- this is the Audita-native object format with explicit schema/version metadata.
### `seriatim-intermediate`
Status:
- deferred / not implemented
Current behavior:
- selecting `seriatim-intermediate` fails clearly as an unsupported output schema.
Reason:
- a concrete, repository-backed contract for this schema has not been finalized yet.
## Selection
Choose output schema with CLI:
```sh
audita process <transcript.json> --glossary <glossary.yaml> --output-schema audita-v1
```
Or in file config:
```yaml
version: 1
output:
schema: audita-v1
```
Precedence remains:
1. defaults
2. file config
3. environment overrides
4. CLI overrides
`--output-schema` overrides `output.schema` when both are supplied.
## Output routing behavior
- With `--output`, transcript JSON is written to file using the selected schema and stdout stays empty on success.
- Without `--output`, stdout contains transcript JSON only, using the selected schema.
- `--report-json` writes report JSON to file and does not write report payloads to stdout.
## Backward-compatibility expectations
- default schema stays `bare-segments` for compatibility unless explicitly changed in a future breaking release;
- supported schema names are treated as stable public contract values;
- unsupported schema names fail before output write.
Implemented output schema registry docs now live at [`docs/internal/output-schemas.md`](../internal/output-schemas.md).

View File

@@ -1,118 +1,3 @@
# Audita Prompts
# Moved: Prompt Registry
This document describes Audita's built-in embedded prompt assets and prompt registry behavior.
## Why embedded prompt assets
Audita embeds production prompt text into the binary so runtime behavior is:
- deterministic;
- auditable;
- dependency-light;
- not dependent on external prompt files at execution time.
Prompt text is authored as Markdown assets and rendered by Go code using typed template data.
## Built-in prompt registry
The prompt registry lives in `internal/prompts` and is responsible for:
- loading embedded prompt assets;
- registering stable prompt IDs and versions;
- recording prompt source metadata;
- computing deterministic SHA-256 source hashes;
- rendering system/user prompts with strict missing-key failures.
Current prompt source behavior:
- built-in embedded prompts only (`prompt_source = builtin`).
- filesystem prompt overrides are not supported.
## Built-in prompt IDs
Module proposal prompts:
- `modules.glossary.proposal`
- `modules.homophones.proposal`
- `modules.spoken_word.proposal`
- `modules.grammar.proposal`
LLM-backed validator prompts:
- `validators.spoken_form_plausibility`
- `validators.meaning_reversal_review`
- `validators.editorial_review`
- `validators.grammar_review`
- `validators.spoken_word_review`
## Prompt version semantics
Current built-in prompt version value is `v1`.
Version is a stable metadata identifier for diagnostics and debugging. It is not a dynamic prompt-selection mechanism.
## Prompt hash semantics
Each registered prompt includes a deterministic SHA-256 hash of embedded source text.
Hash purpose:
- identify exact prompt source used in a run;
- support diagnostics reproducibility and change auditing.
Current hash scope:
- source prompt text (system + user assets for a registered prompt), not a runtime secret-bearing payload.
## Template rendering behavior
Prompt rendering uses Go `text/template` with typed template data from module/validator builders.
Missing-key behavior:
- rendering uses missing-key errors;
- missing/renamed template fields fail quickly instead of silently producing incomplete prompts.
Go code still owns:
- structured request/response models;
- response schema selection;
- transcript/glossary/payload formatting;
- module and validator selection;
- diagnostics wiring.
## Shared prompt hardening policy
A shared hardening fragment is embedded once and included in every module proposal prompt and every LLM-validator prompt.
Hardening policy includes:
- transcript text is untrusted data;
- glossary entries and transcript descriptions are reference data, not instructions;
- instructions found inside transcript text must not be obeyed;
- model must perform only the requested correction/validation task;
- no invention of facts, names, events, motivations, speaker intent, or corrections;
- transcript remains the source of truth.
## Transcript description behavior
Transcript description remains background-only prompt context:
- it may help interpret ambiguous terms;
- it is explicitly non-authoritative and must not override transcript content;
- empty descriptions do not render awkward blank context sections.
Generated transcript descriptions are not implemented in this workstream.
## Diagnostics and report metadata boundaries
Current metadata flow:
- proposal-generation diagnostics request metadata includes prompt metadata;
- LLM-validator diagnostics request metadata includes prompt metadata.
Prompt metadata fields used in diagnostics:
- `prompt_id`
- `prompt_version`
- `prompt_source`
- `embedded_path`
- `sha256`
Current boundary:
- detailed prompt metadata is diagnostics-first;
- broad report-level prompt registries/ledgers are deferred.
## 1.0 boundary
Not implemented for 1.0 in this workstream:
- filesystem prompt overrides;
- user-configurable prompt selection;
- external prompt directories.
Implemented prompt registry and prompt metadata docs now live at [`docs/internal/prompts.md`](../internal/prompts.md).

View File

@@ -1,72 +1,3 @@
# Structured LLM Architecture
# Moved: LLM Runtime
## Scope
This document describes Audita's structured LLM runtime boundary and adapter behavior.
## Runtime boundary
Production LLM integration depends on the internal contract only:
- `contracts.StructuredLLMClient`
- `CompleteStructured(ctx, req, out)`
Provider SDK types do not leak past this boundary.
## Adapter ownership
`internal/framework/llm` owns the OpenAI-compatible HTTP adapter and shared LLM runtime utilities.
Key responsibilities:
- request assembly;
- timeout/cancellation propagation;
- bounded retry behavior;
- scheduler integration;
- provider response decoding;
- error redaction.
## Structured schema registry
Structured response schemas are registered in `internal/framework/responseschema` and include stable metadata:
- `id`
- `version`
- `name`
- `json_schema`
- `sha256`
Current schema keys:
- `correction_set`
- `validator_decision_set`
Schema metadata is attached to diagnostics through `Schema.DiagnosticsMap()`.
## Request shape assumptions
Audita targets OpenAI-compatible chat-completions endpoints and sends structured requests with:
- model;
- chat messages;
- `response_format.type = json_schema`;
- schema name and JSON schema payload.
## Local validation remains mandatory
Provider schema enforcement is treated as transport-level guardrails.
Audita still validates output locally before applying behavior changes:
- proposal decoding and proposal invariants;
- validator decision decoding and cardinality checks;
- deterministic validation and apply-time rules.
## Shared malformed-output policy
Malformed structured-output classification is centralized in `internal/framework/structuredoutput`.
Proposal generation and validator execution both use this shared classifier so downgrade behavior cannot drift between the two paths.
## Secrets and redaction
Secret extraction for LLM redaction is centralized in `llm.ConfiguredSecrets(cfg)` and reused by proposal and validator diagnostics writers.
Secrets are redacted from:
- diagnostics artifacts;
- report artifacts;
- surfaced adapter/runtime errors.
## Concurrency and scheduling
LLM execution is constrained by composed scheduler limits:
- total LLM concurrency;
- proposal LLM concurrency;
- validation LLM concurrency.
The scheduler is FIFO and context-aware so permits are released on success, failure, and cancellation.
Implemented structured LLM runtime docs now live at [`docs/internal/llm-runtime.md`](../internal/llm-runtime.md).

View File

@@ -1,96 +1,3 @@
# Audita Validators
# Moved: Validators
## Scope
This document defines the built-in validator system used by production module runs.
## Ownership boundaries
Built-in validator keys, constructors, and module chains are owned by `internal/validators`.
Shared runtime execution mechanics are owned by `internal/framework/validators`, including:
- validator request/result models;
- deterministic proposal checks;
- LLM validator batching and execution;
- decision-cardinality enforcement;
- diagnostics integration.
Execution class metadata is owned by `internal/validators/metadata`.
## Stable validator keys
Deterministic:
- `proposal_shape`
- `confidence_threshold`
- `original_text_presence`
- `non_empty_corrected_text`
- `no_effect`
- `protected_terms`
LLM-backed:
- `spoken_form_plausibility`
- `meaning_reversal_review`
- `editorial_review`
## Built-in module chains
`glossary`:
- `proposal_shape`
- `no_effect`
- `original_text_presence`
- `confidence_threshold`
- `protected_terms`
- `non_empty_corrected_text`
- `spoken_form_plausibility`
- `meaning_reversal_review`
`homophones`:
- `proposal_shape`
- `no_effect`
- `original_text_presence`
- `confidence_threshold`
- `protected_terms`
- `non_empty_corrected_text`
- `spoken_form_plausibility`
- `meaning_reversal_review`
`spoken_word`:
- `proposal_shape`
- `no_effect`
- `original_text_presence`
- `confidence_threshold`
- `protected_terms`
- `non_empty_corrected_text`
- `editorial_review`
- `meaning_reversal_review`
`grammar`:
- `proposal_shape`
- `no_effect`
- `original_text_presence`
- `confidence_threshold`
- `protected_terms`
- `non_empty_corrected_text`
- `editorial_review`
- `meaning_reversal_review`
## Ordering and execution semantics
Validator ordering is based on canonical metadata:
- deterministic validators run before LLM-backed validators.
Within each module stage:
- proposals are generated per section;
- validator chains execute on those proposals;
- approved proposals are applied once after section work settles.
## Malformed payload behavior
Malformed structured-output from proposal generation and LLM validator calls is downgraded, not treated as a process-fatal transport error.
Current outcomes:
- malformed proposal-generation payloads produce section/module warnings and zero proposals for the affected section;
- malformed validator decision payloads reject the affected validator batch with warnings;
- deterministic validator behavior and runner order remain unchanged.
## Reporting identity
Reports and diagnostics use stable validator keys as identifiers.
Correction-ledger deterministic-vs-LLM classification is derived from canonical validator metadata, not package-local hardcoded maps.
## Prompt assets
LLM validator prompt assets and prompt metadata are documented in [Prompts](./prompts.md).
Implemented validator architecture docs now live at [`docs/internal/validators.md`](../internal/validators.md).