From e5944f98754b9fbb8136a5858a649def056b587b Mon Sep 17 00:00:00 2001 From: Eric Rakestraw Date: Sun, 24 May 2026 01:02:47 +0000 Subject: [PATCH] Migrate internal architecture docs to docs/internal --- docs/architecture.md | 20 ++-- docs/architecture/architecture.md | 154 +------------------------ docs/architecture/diagnostics.md | 107 +---------------- docs/architecture/output-schemas.md | 89 +------------- docs/architecture/prompts.md | 119 +------------------ docs/architecture/structured-llm.md | 73 +----------- docs/architecture/validators.md | 97 +--------------- docs/internal/diagnostics-reporting.md | 79 +++++++++++++ docs/internal/llm-runtime.md | 67 +++++++++++ docs/internal/modules.md | 58 ++++++++++ docs/internal/output-schemas.md | 46 ++++++++ docs/internal/overview.md | 92 +++++++++++++++ docs/internal/pipeline.md | 79 +++++++++++++ docs/internal/prompts.md | 62 ++++++++++ docs/internal/validators.md | 72 ++++++++++++ 15 files changed, 581 insertions(+), 633 deletions(-) create mode 100644 docs/internal/diagnostics-reporting.md create mode 100644 docs/internal/llm-runtime.md create mode 100644 docs/internal/modules.md create mode 100644 docs/internal/output-schemas.md create mode 100644 docs/internal/overview.md create mode 100644 docs/internal/pipeline.md create mode 100644 docs/internal/prompts.md create mode 100644 docs/internal/validators.md diff --git a/docs/architecture.md b/docs/architecture.md index 56d3466..c538f2c 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -1,14 +1,16 @@ # Audita Architecture Index -This file is the entrypoint for architecture documentation. +This file is the architecture documentation entrypoint. -Core architecture overview: -- [Architecture Overview](./architecture/architecture.md) +Implemented internals: +- [Internal Overview](./internal/overview.md) +- [Pipeline](./internal/pipeline.md) +- [Modules](./internal/modules.md) +- [Validators](./internal/validators.md) +- [LLM Runtime](./internal/llm-runtime.md) +- [Diagnostics and Reporting](./internal/diagnostics-reporting.md) +- [Prompts](./internal/prompts.md) +- [Output Schemas](./internal/output-schemas.md) -Focused architecture contracts: +External/runtime contract summary: - [Public Contract](./architecture/public-contract.md) -- [Diagnostics](./architecture/diagnostics.md) -- [Structured LLM](./architecture/structured-llm.md) -- [Validators](./architecture/validators.md) -- [Prompts](./architecture/prompts.md) -- [Output Schemas](./architecture/output-schemas.md) diff --git a/docs/architecture/architecture.md b/docs/architecture/architecture.md index 70193af..466473a 100644 --- a/docs/architecture/architecture.md +++ b/docs/architecture/architecture.md @@ -1,153 +1,5 @@ -# Audita Architecture +# Moved: Internal Overview -## Scope -This document describes the production architecture implemented in this repository today. +Implemented internal architecture docs now live in [`docs/internal/`](../internal/). -Audita is a single-process Go CLI that: -- loads effective runtime configuration; -- reads transcript and glossary inputs; -- normalizes and sections transcripts; -- runs a built-in module pipeline with validator chains; -- writes transcript output and run diagnostics. - -## Runtime entrypoints -Primary CLI commands: -- `audita process --glossary [flags]` -- `audita config validate --config ` -- `audita config print-effective [--config ]` - -Command ownership lives in `internal/cli/run.go`. - -## Configuration model -`internal/core/config` owns defaults, file parsing, environment overrides, CLI overrides, and validation. - -Effective-config loading for `process` and `config print-effective` is centralized in: -- `ResolveConfigPath` -- `LoadEffectiveConfig` - -Effective precedence for `audita process`: -1. defaults -2. config file -3. environment overrides -4. CLI overrides - -`audita config validate` is intentionally file-only validation: -- load versioned file; -- apply onto defaults; -- validate; -- do not apply environment overrides. - -Supported module and output-schema keys are validated through shared catalogs: -- module keys: `internal/core/modulecatalog` -- output schemas: `internal/core/outputschema` - -## Pipeline and module orchestration -The built-in module sequence is configured in runtime config and executed by `internal/framework/runner` through resolved module specs. - -Current default sequence: -- `glossary` -- `homophones` -- `glossary` -- `spoken_word` -- `grammar` - -Execution behavior: -- modules execute serially over the working transcript; -- section proposal work can run concurrently within a module; -- validator execution happens on generated proposals before application; -- approved proposals are applied once per module in deterministic proposal-index order. - -Production modules remain separate packages: -- `internal/modules/glossary` -- `internal/modules/homophones` -- `internal/modules/spoken_word` -- `internal/modules/grammar` - -## Proposal generation and prompt context -Shared proposal plumbing is centralized in `internal/framework/proposal_generation`. - -Module packages provide: -- module identity and replacement policy; -- module-specific prompt message building; -- built-in validator chain selection. - -Shared prompt payload helpers are in `internal/framework/promptcontext`. - -## Validator architecture -Built-in validator construction and chain composition live in `internal/validators`. - -Shared validator runtime mechanics live in `internal/framework/validators`. - -Execution class metadata (deterministic vs LLM-backed) is centralized in `internal/validators/metadata` and used for ordering and reporting classification. - -## Structured LLM boundary -All production LLM calls go through the internal contract: -- `contracts.StructuredLLMClient` -- `CompleteStructured(ctx, req, out)` - -The OpenAI-compatible HTTP adapter is implemented in `internal/framework/llm`. - -Structured response schemas are registered in `internal/framework/responseschema` and attached to requests via `response_format` metadata. - -Malformed structured-output detection is centralized in `internal/framework/structuredoutput` and reused by proposal generation and validator execution so downgrade behavior stays consistent. - -## Stage naming and diagnostics metadata -Diagnostics stage naming is centralized in `internal/framework/stagename`: -- module proposal stage names; -- proposal-generation stage names; -- validator batch stage names. - -Prompt metadata and response-schema metadata each expose canonical diagnostics maps via: -- `prompts.Metadata.DiagnosticsMap()` -- `responseschema.Schema.DiagnosticsMap()` - -## Diagnostics and reporting -Run-directory artifacts are owned by `internal/core/diagnostics`. - -Stable artifact names are centralized constants (for example transcript artifacts, `invocation.json`, `effective-config.json`, `utilization-diagnostics.json`, `correction-ledger.json`, `report.json`, `error.log`). - -Report diagnostics path metadata is constructed through `BuildDiagnosticsMetadata`, which keeps run-directory artifact references consistent between success and failure reports. - -## Secret redaction -Redaction responsibilities are split by concern: -- structural config redaction: `config.Config.Redacted()` -- byte/string payload redaction for diagnostics and surfaced errors: framework redaction utilities. - -Configured LLM secret extraction is centralized in `llm.ConfiguredSecrets(cfg)` and reused across proposal and validator diagnostics paths. - -## Output contracts -Transcript output schema selection is owned by `internal/core/outputschema`. - -Supported schemas: -- `bare-segments` -- `audita-v1` - -Unknown schema keys fail validation and runtime resolution. - -## Key package map -Core packages: -- `internal/core/config` -- `internal/core/schema` -- `internal/core/normalization` -- `internal/core/chunking` -- `internal/core/diagnostics` -- `internal/core/reporting` -- `internal/core/modulecatalog` -- `internal/core/outputschema` - -Framework packages: -- `internal/framework/contracts` -- `internal/framework/proposals` -- `internal/framework/proposal_generation` -- `internal/framework/promptcontext` -- `internal/framework/runner` -- `internal/framework/validators` -- `internal/framework/llm` -- `internal/framework/responseschema` -- `internal/framework/stagename` -- `internal/framework/structuredoutput` - -Domain packages: -- `internal/modules/*` -- `internal/validators/*` -- `internal/prompts` +Start with [`docs/internal/overview.md`](../internal/overview.md). diff --git a/docs/architecture/diagnostics.md b/docs/architecture/diagnostics.md index a96bf3a..307019f 100644 --- a/docs/architecture/diagnostics.md +++ b/docs/architecture/diagnostics.md @@ -1,104 +1,5 @@ -# Audita Diagnostics +# Moved: Diagnostics and Reporting -This document describes the run-directory diagnostics artifacts produced by `audita process`. - -## Purpose - -Diagnostics provide machine-readable run context and execution artifacts for: -- failure debugging; -- validator/correction review; -- post-run performance analysis. - -Diagnostics are written under the configured work directory (`--work-dir`) when run-directory initialization succeeds. - -## Core artifacts - -Typical artifacts in each run directory: -- `source-transcript.json` -- `source-transcript-parsed.json` -- `normalized-transcript.json` -- `normalization-summary.json` -- `chunking-summary.json` -- `invocation.json` -- `effective-config.json` (redacted) -- module/validator LLM interaction artifacts -- `report.json` -- `error.log` on failure - -## Utilization diagnostics artifact - -Artifact: -- `utilization-diagnostics.json` - -High-level fields: -- `effective_concurrency`: - - total/proposal/validation LLM concurrency limits in effect. -- `run_timing`: - - run wall time; - - scheduler queue wait time; - - LLM execution time; - - deterministic validator time; - - max/average in-flight LLM calls. -- `llm_calls`: - - total proposal and validation LLM call counts. -- `modules`: - - module-level timing summaries. -- `validators`: - - per-validator timing summaries keyed by stable validator key. - -## Correction ledger artifact - -Artifact: -- `correction-ledger.json` - -Ledger records are flattened review entries derived from module results and include: -- module/proposal identity (`module_key`, `module_instance`, `proposal_index`, `segment_id`); -- correction text fields and replacement policy when available; -- disposition: - - `applied` - - `rejected` - - `skipped` - - `failed` -- stable reason codes/messages; -- deterministic and LLM validator decision snapshots using stable validator keys. - -Validator rejection and proposal-application skip are distinct dispositions. -Module warnings are reported in module results and diagnostics metadata, but do not create standalone correction-ledger rows. - -## Report references - -`report.json` and optional `--report-json` output include diagnostics metadata paths for: -- utilization diagnostics artifact; -- correction ledger artifact; -- existing transcript/normalization/chunking/invocation/effective-config artifacts. - -Module report entries also include warning records for malformed proposal-generation payloads and malformed validator batches. - -## Retention behavior - -Run-directory retention follows configured policy: -- `always`: keep all run directories; -- `never`: keep successful run directories; -- `auto`: keep failed runs and successful runs with skipped/rejected corrections. - -## Redaction guarantees - -API keys and other configured secrets are redacted from: -- `effective-config.json`; -- LLM interaction diagnostics artifacts; -- reports and surfaced errors. - -## Debugging guide - -When debugging: -- slow runs: - - inspect `utilization-diagnostics.json` (`run_timing`, `modules`, `validators`, in-flight metrics). -- validator rejections: - - inspect `correction-ledger.json` rejected entries and matching validator decisions; - - inspect validator response diagnostics payloads. -- module warnings: - - inspect module `warnings` entries in `report.json` or `--report-json`; - - follow any diagnostic artifact path on the warning to the recorded error/response payload. -- application skips: - - inspect `correction-ledger.json` skipped entries and skip reason codes; - - compare with validator decisions to distinguish validation rejection vs apply-time skip. +Implemented diagnostics and reporting internals now live at: +- [`docs/internal/diagnostics-reporting.md`](../internal/diagnostics-reporting.md) +- [`docs/operations.md`](../operations.md) diff --git a/docs/architecture/output-schemas.md b/docs/architecture/output-schemas.md index b170438..dbd5f58 100644 --- a/docs/architecture/output-schemas.md +++ b/docs/architecture/output-schemas.md @@ -1,88 +1,3 @@ -# Audita Output Schemas +# Moved: Output Schemas -This document describes the built-in transcript output schema registry used by `audita process`. - -## Supported schema names - -### `bare-segments` - -Status: -- implemented -- default output schema - -Shape: -- top-level JSON array of transcript segments - -Segment fields: -- `id` -- `speaker` -- `start` -- `end` -- `text` -- optional `categories` - -Compatibility: -- this preserves the long-standing output shape used by existing consumers. - -### `audita-v1` - -Status: -- implemented - -Shape: -- top-level JSON object: - - `schema`: `"audita-v1"` - - `version`: `"v1"` - - `segments`: transcript segment array - -Segment fields inside `segments` match `bare-segments` segment fields. - -Compatibility: -- this is the Audita-native object format with explicit schema/version metadata. - -### `seriatim-intermediate` - -Status: -- deferred / not implemented - -Current behavior: -- selecting `seriatim-intermediate` fails clearly as an unsupported output schema. - -Reason: -- a concrete, repository-backed contract for this schema has not been finalized yet. - -## Selection - -Choose output schema with CLI: - -```sh -audita process --glossary --output-schema audita-v1 -``` - -Or in file config: - -```yaml -version: 1 -output: - schema: audita-v1 -``` - -Precedence remains: -1. defaults -2. file config -3. environment overrides -4. CLI overrides - -`--output-schema` overrides `output.schema` when both are supplied. - -## Output routing behavior - -- With `--output`, transcript JSON is written to file using the selected schema and stdout stays empty on success. -- Without `--output`, stdout contains transcript JSON only, using the selected schema. -- `--report-json` writes report JSON to file and does not write report payloads to stdout. - -## Backward-compatibility expectations - -- default schema stays `bare-segments` for compatibility unless explicitly changed in a future breaking release; -- supported schema names are treated as stable public contract values; -- unsupported schema names fail before output write. +Implemented output schema registry docs now live at [`docs/internal/output-schemas.md`](../internal/output-schemas.md). diff --git a/docs/architecture/prompts.md b/docs/architecture/prompts.md index 2e684c4..1e2b909 100644 --- a/docs/architecture/prompts.md +++ b/docs/architecture/prompts.md @@ -1,118 +1,3 @@ -# Audita Prompts +# Moved: Prompt Registry -This document describes Audita's built-in embedded prompt assets and prompt registry behavior. - -## Why embedded prompt assets - -Audita embeds production prompt text into the binary so runtime behavior is: -- deterministic; -- auditable; -- dependency-light; -- not dependent on external prompt files at execution time. - -Prompt text is authored as Markdown assets and rendered by Go code using typed template data. - -## Built-in prompt registry - -The prompt registry lives in `internal/prompts` and is responsible for: -- loading embedded prompt assets; -- registering stable prompt IDs and versions; -- recording prompt source metadata; -- computing deterministic SHA-256 source hashes; -- rendering system/user prompts with strict missing-key failures. - -Current prompt source behavior: -- built-in embedded prompts only (`prompt_source = builtin`). -- filesystem prompt overrides are not supported. - -## Built-in prompt IDs - -Module proposal prompts: -- `modules.glossary.proposal` -- `modules.homophones.proposal` -- `modules.spoken_word.proposal` -- `modules.grammar.proposal` - -LLM-backed validator prompts: -- `validators.spoken_form_plausibility` -- `validators.meaning_reversal_review` -- `validators.editorial_review` -- `validators.grammar_review` -- `validators.spoken_word_review` - -## Prompt version semantics - -Current built-in prompt version value is `v1`. - -Version is a stable metadata identifier for diagnostics and debugging. It is not a dynamic prompt-selection mechanism. - -## Prompt hash semantics - -Each registered prompt includes a deterministic SHA-256 hash of embedded source text. - -Hash purpose: -- identify exact prompt source used in a run; -- support diagnostics reproducibility and change auditing. - -Current hash scope: -- source prompt text (system + user assets for a registered prompt), not a runtime secret-bearing payload. - -## Template rendering behavior - -Prompt rendering uses Go `text/template` with typed template data from module/validator builders. - -Missing-key behavior: -- rendering uses missing-key errors; -- missing/renamed template fields fail quickly instead of silently producing incomplete prompts. - -Go code still owns: -- structured request/response models; -- response schema selection; -- transcript/glossary/payload formatting; -- module and validator selection; -- diagnostics wiring. - -## Shared prompt hardening policy - -A shared hardening fragment is embedded once and included in every module proposal prompt and every LLM-validator prompt. - -Hardening policy includes: -- transcript text is untrusted data; -- glossary entries and transcript descriptions are reference data, not instructions; -- instructions found inside transcript text must not be obeyed; -- model must perform only the requested correction/validation task; -- no invention of facts, names, events, motivations, speaker intent, or corrections; -- transcript remains the source of truth. - -## Transcript description behavior - -Transcript description remains background-only prompt context: -- it may help interpret ambiguous terms; -- it is explicitly non-authoritative and must not override transcript content; -- empty descriptions do not render awkward blank context sections. - -Generated transcript descriptions are not implemented in this workstream. - -## Diagnostics and report metadata boundaries - -Current metadata flow: -- proposal-generation diagnostics request metadata includes prompt metadata; -- LLM-validator diagnostics request metadata includes prompt metadata. - -Prompt metadata fields used in diagnostics: -- `prompt_id` -- `prompt_version` -- `prompt_source` -- `embedded_path` -- `sha256` - -Current boundary: -- detailed prompt metadata is diagnostics-first; -- broad report-level prompt registries/ledgers are deferred. - -## 1.0 boundary - -Not implemented for 1.0 in this workstream: -- filesystem prompt overrides; -- user-configurable prompt selection; -- external prompt directories. +Implemented prompt registry and prompt metadata docs now live at [`docs/internal/prompts.md`](../internal/prompts.md). diff --git a/docs/architecture/structured-llm.md b/docs/architecture/structured-llm.md index 3d31332..91d1092 100644 --- a/docs/architecture/structured-llm.md +++ b/docs/architecture/structured-llm.md @@ -1,72 +1,3 @@ -# Structured LLM Architecture +# Moved: LLM Runtime -## Scope -This document describes Audita's structured LLM runtime boundary and adapter behavior. - -## Runtime boundary -Production LLM integration depends on the internal contract only: -- `contracts.StructuredLLMClient` -- `CompleteStructured(ctx, req, out)` - -Provider SDK types do not leak past this boundary. - -## Adapter ownership -`internal/framework/llm` owns the OpenAI-compatible HTTP adapter and shared LLM runtime utilities. - -Key responsibilities: -- request assembly; -- timeout/cancellation propagation; -- bounded retry behavior; -- scheduler integration; -- provider response decoding; -- error redaction. - -## Structured schema registry -Structured response schemas are registered in `internal/framework/responseschema` and include stable metadata: -- `id` -- `version` -- `name` -- `json_schema` -- `sha256` - -Current schema keys: -- `correction_set` -- `validator_decision_set` - -Schema metadata is attached to diagnostics through `Schema.DiagnosticsMap()`. - -## Request shape assumptions -Audita targets OpenAI-compatible chat-completions endpoints and sends structured requests with: -- model; -- chat messages; -- `response_format.type = json_schema`; -- schema name and JSON schema payload. - -## Local validation remains mandatory -Provider schema enforcement is treated as transport-level guardrails. - -Audita still validates output locally before applying behavior changes: -- proposal decoding and proposal invariants; -- validator decision decoding and cardinality checks; -- deterministic validation and apply-time rules. - -## Shared malformed-output policy -Malformed structured-output classification is centralized in `internal/framework/structuredoutput`. - -Proposal generation and validator execution both use this shared classifier so downgrade behavior cannot drift between the two paths. - -## Secrets and redaction -Secret extraction for LLM redaction is centralized in `llm.ConfiguredSecrets(cfg)` and reused by proposal and validator diagnostics writers. - -Secrets are redacted from: -- diagnostics artifacts; -- report artifacts; -- surfaced adapter/runtime errors. - -## Concurrency and scheduling -LLM execution is constrained by composed scheduler limits: -- total LLM concurrency; -- proposal LLM concurrency; -- validation LLM concurrency. - -The scheduler is FIFO and context-aware so permits are released on success, failure, and cancellation. +Implemented structured LLM runtime docs now live at [`docs/internal/llm-runtime.md`](../internal/llm-runtime.md). diff --git a/docs/architecture/validators.md b/docs/architecture/validators.md index 08c84b2..b55f6d1 100644 --- a/docs/architecture/validators.md +++ b/docs/architecture/validators.md @@ -1,96 +1,3 @@ -# Audita Validators +# Moved: Validators -## Scope -This document defines the built-in validator system used by production module runs. - -## Ownership boundaries -Built-in validator keys, constructors, and module chains are owned by `internal/validators`. - -Shared runtime execution mechanics are owned by `internal/framework/validators`, including: -- validator request/result models; -- deterministic proposal checks; -- LLM validator batching and execution; -- decision-cardinality enforcement; -- diagnostics integration. - -Execution class metadata is owned by `internal/validators/metadata`. - -## Stable validator keys -Deterministic: -- `proposal_shape` -- `confidence_threshold` -- `original_text_presence` -- `non_empty_corrected_text` -- `no_effect` -- `protected_terms` - -LLM-backed: -- `spoken_form_plausibility` -- `meaning_reversal_review` -- `editorial_review` - -## Built-in module chains -`glossary`: -- `proposal_shape` -- `no_effect` -- `original_text_presence` -- `confidence_threshold` -- `protected_terms` -- `non_empty_corrected_text` -- `spoken_form_plausibility` -- `meaning_reversal_review` - -`homophones`: -- `proposal_shape` -- `no_effect` -- `original_text_presence` -- `confidence_threshold` -- `protected_terms` -- `non_empty_corrected_text` -- `spoken_form_plausibility` -- `meaning_reversal_review` - -`spoken_word`: -- `proposal_shape` -- `no_effect` -- `original_text_presence` -- `confidence_threshold` -- `protected_terms` -- `non_empty_corrected_text` -- `editorial_review` -- `meaning_reversal_review` - -`grammar`: -- `proposal_shape` -- `no_effect` -- `original_text_presence` -- `confidence_threshold` -- `protected_terms` -- `non_empty_corrected_text` -- `editorial_review` -- `meaning_reversal_review` - -## Ordering and execution semantics -Validator ordering is based on canonical metadata: -- deterministic validators run before LLM-backed validators. - -Within each module stage: -- proposals are generated per section; -- validator chains execute on those proposals; -- approved proposals are applied once after section work settles. - -## Malformed payload behavior -Malformed structured-output from proposal generation and LLM validator calls is downgraded, not treated as a process-fatal transport error. - -Current outcomes: -- malformed proposal-generation payloads produce section/module warnings and zero proposals for the affected section; -- malformed validator decision payloads reject the affected validator batch with warnings; -- deterministic validator behavior and runner order remain unchanged. - -## Reporting identity -Reports and diagnostics use stable validator keys as identifiers. - -Correction-ledger deterministic-vs-LLM classification is derived from canonical validator metadata, not package-local hardcoded maps. - -## Prompt assets -LLM validator prompt assets and prompt metadata are documented in [Prompts](./prompts.md). +Implemented validator architecture docs now live at [`docs/internal/validators.md`](../internal/validators.md). diff --git a/docs/internal/diagnostics-reporting.md b/docs/internal/diagnostics-reporting.md new file mode 100644 index 0000000..ebf4a7b --- /dev/null +++ b/docs/internal/diagnostics-reporting.md @@ -0,0 +1,79 @@ +# Audita Diagnostics and Reporting + +## Scope + +This document describes diagnostics artifacts, process report mapping, and correction ledger generation. + +## Run Directory Ownership + +`internal/core/diagnostics` owns run-directory creation, artifact writes, and retention decisions. + +Stable artifact names include: +- `source-transcript.json` +- `source-transcript-parsed.json` +- `normalized-transcript.json` +- `normalization-summary.json` +- `chunking-summary.json` +- `utilization-diagnostics.json` +- `correction-ledger.json` +- `invocation.json` +- `effective-config.json` +- `report.json` +- `error.log` (failure) + +## Process Report Mapping + +`internal/framework/processreport` maps runner/CLI execution facts into `reporting.ProcessReport`. + +Report metadata fields include: +- `report_schema_name` (`audita-process-report`) +- `report_schema_version` (`v1`) +- `output_schema` +- `config_version` (when file config exists) + +The report includes: +- top-level status/error phase/error message; +- normalization/chunking summaries; +- diagnostics metadata paths; +- per-module results and module summary. + +## Correction Ledger + +`internal/framework/processreport/BuildCorrectionLedger` flattens run results into `correction-ledger.json` entries. + +Dispositions: +- `applied` +- `skipped` +- `rejected` +- `failed` + +Validator decisions are split into deterministic and LLM-backed groups using validator metadata classification. + +## Report Write Paths + +- run directory always attempts to write `report.json` when possible; +- optional `--report-json` writes an external report file; +- on failure paths, report writing is best-effort and does not mask primary run errors. + +## Retention Interaction + +Current retention behavior: +- failed runs are retained; +- `always` keeps successful runs; +- `auto` removes only clean successful runs; +- `never` currently retains successful runs in current implementation. + +## Redaction + +Redacted data expectations: +- effective config artifact uses config redaction; +- diagnostics payloads and surfaced errors use LLM secret redaction; +- reports should not include raw API key values. + +## Key Tests + +- `internal/core/diagnostics/*_test.go` +- `internal/framework/processreport/*_test.go` +- `internal/core/reporting/report_test.go` +- `internal/cli/run_test.go` +- `cmd/audita/main_integration_test.go` diff --git a/docs/internal/llm-runtime.md b/docs/internal/llm-runtime.md new file mode 100644 index 0000000..7f1e5f5 --- /dev/null +++ b/docs/internal/llm-runtime.md @@ -0,0 +1,67 @@ +# Audita LLM Runtime + +## Scope + +This document describes the structured LLM runtime and scheduler behavior. + +## Client Boundary + +All runtime LLM calls go through `contracts.StructuredLLMClient`. + +Primary adapter: +- `internal/framework/llm/OpenAICompatibleClient` + +## Request/Response Behavior + +The OpenAI-compatible adapter sends chat completions requests with: +- model; +- messages; +- `response_format.type = json_schema`; +- strict schema envelope (`name`, `schema`, `strict=true`). + +The response is decoded into the requested structured output target. + +## Response Schema Registry + +Structured response schemas are registered in `internal/framework/responseschema`: +- `correction_set` +- `validator_decision_set` + +Each schema includes stable diagnostics metadata (`id`, `version`, `name`, `sha256`). + +## Retries and Error Handling + +Adapter retries apply to retryable conditions (for example transport/decoding/retryable status classes) up to configured `max_retries`. + +Errors are sanitized to redact configured API-key values before surfacing. + +Malformed structured output detection is shared through `internal/framework/structuredoutput` and is used by: +- proposal generation; +- LLM-backed validators. + +## Scheduling and Concurrency + +`internal/framework/llm/Scheduler` provides FIFO, context-aware permit gating. + +Runner composes scheduler limits across: +- total LLM concurrency; +- proposal LLM concurrency; +- validation LLM concurrency. + +Scheduler release is guarded to avoid permit leaks on cancellation/error. + +## Diagnostics and Redaction + +`internal/framework/llm/DiagnosticsWriter` writes request/response/error artifacts. + +Configured secrets are derived from `llm.ConfiguredSecrets(cfg)` and redacted from: +- diagnostics payloads; +- surfaced runtime/adapter errors. + +## Key Tests + +- `internal/framework/llm/openai_compatible_client_test.go` +- `internal/framework/llm/scheduler_test.go` +- `internal/framework/llm/diagnostics_test.go` +- `internal/framework/responseschema/registry_test.go` +- `internal/framework/structuredoutput/malformed_test.go` diff --git a/docs/internal/modules.md b/docs/internal/modules.md new file mode 100644 index 0000000..a7ef601 --- /dev/null +++ b/docs/internal/modules.md @@ -0,0 +1,58 @@ +# Audita Modules + +## Scope + +This document covers module contracts and built-in module packages. + +## Module Contract + +Modules implement `contracts.TranscriptModule`: +- `Key()` +- `ReplacementPolicy()` +- `Validators()` +- `Propose(ctx, req)` + +Runner resolves configured module specs to module instances through `internal/framework/modules`. + +## Built-In Modules + +Current module packages: +- `internal/modules/glossary` +- `internal/modules/homophones` +- `internal/modules/spoken_word` +- `internal/modules/grammar` + +Current replacement policies: +- `glossary`: `replace_all` +- `homophones`: `require_unique` +- `spoken_word`: `require_unique` +- `grammar`: `require_unique` + +## Proposal Generation Ownership + +Shared proposal-generation plumbing is centralized in: +- `internal/framework/proposal_generation` + +Module packages own: +- prompt selection (`internal/prompts` prompt IDs); +- module-specific prompt payload construction. + +Shared prompt helpers live in `internal/framework/promptcontext`. + +## Validator Chain Ownership + +Built-in chains are resolved in `internal/validators` per module key. +Module packages call the built-in chain resolver at construction. + +## Failure and Warning Behavior + +- module setup failures surface as `runner_setup` or module setup errors; +- module runtime failures surface as `runner_execution` with partial module results preserved; +- malformed structured proposal payloads are downgraded to warnings and section-level proposal rejection. + +## Key Tests + +- `internal/modules/*/module_test.go` +- `internal/framework/modules/registry_test.go` +- `internal/framework/proposal_generation/*_test.go` +- `internal/cli/run_test.go` (pipeline/report integration) diff --git a/docs/internal/output-schemas.md b/docs/internal/output-schemas.md new file mode 100644 index 0000000..74346c8 --- /dev/null +++ b/docs/internal/output-schemas.md @@ -0,0 +1,46 @@ +# Audita Output Schemas + +## Scope + +This document describes the implemented transcript output schema registry. + +## Registry Ownership + +Output schema registry is owned by `internal/core/outputschema`. + +Supported schema keys: +- `bare-segments` +- `audita-v1` + +## Schemas + +`bare-segments`: +- top-level JSON array of transcript segments. + +`audita-v1`: +- top-level JSON object with: + - `schema: "audita-v1"` + - `version: "v1"` + - `segments: [...]` + +Segment fields include `id`, `speaker`, `start`, `end`, `text`, and optional `categories`. + +## Validation and Resolution + +Config validation and runtime resolution both reject unsupported schema keys. + +Unknown schema keys fail with `unsupported output schema` before output emission. + +## Output Emission + +The selected schema is used by `audita process` when writing: +- output file (`--output`) or +- stdout (when no `--output`). + +Report metadata records selected `output_schema`. + +## Key Tests + +- `internal/core/outputschema/registry_test.go` +- `internal/core/config/config_test.go` +- `internal/cli/run_test.go` diff --git a/docs/internal/overview.md b/docs/internal/overview.md new file mode 100644 index 0000000..8aaf68d --- /dev/null +++ b/docs/internal/overview.md @@ -0,0 +1,92 @@ +# Audita Internal Overview + +## Scope + +This document is the internal architecture entry point for developers and coding agents. + +It summarizes: +- package boundaries; +- the main `process` execution path; +- where to add new code safely. + +## Package Map + +CLI and command orchestration: +- `internal/cli` + +Core deterministic components: +- `internal/core/config` +- `internal/core/schema` +- `internal/core/normalization` +- `internal/core/chunking` +- `internal/core/outputschema` +- `internal/core/diagnostics` +- `internal/core/reporting` +- `internal/core/modulecatalog` + +Framework orchestration and contracts: +- `internal/framework/contracts` +- `internal/framework/modules` +- `internal/framework/proposal_generation` +- `internal/framework/proposals` +- `internal/framework/runner` +- `internal/framework/validators` +- `internal/framework/llm` +- `internal/framework/responseschema` +- `internal/framework/structuredoutput` +- `internal/framework/processreport` +- `internal/framework/promptcontext` +- `internal/framework/stagename` + +Domain implementations: +- `internal/modules/*` +- `internal/validators/*` +- `internal/prompts` + +## Main Execution Path (`audita process`) + +High-level flow: +1. CLI loads effective config and validates CLI requirements. +2. Run directory is created and invocation/effective config artifacts are written. +3. Transcript/glossary files are loaded and parsed. +4. Transcript is normalized and chunked. +5. `runner.Run` executes configured module instances. +6. Proposals are validated, applied deterministically, and serialized in selected output schema. +7. Process report, utilization diagnostics, correction ledger, and retention decisions are finalized. + +## Boundary Summary + +- `internal/core/*` owns deterministic, reusable logic and persistence-independent rules. +- `internal/framework/*` owns orchestration contracts and reusable runtime plumbing. +- `internal/modules/*` owns module-specific proposal behavior and prompt usage. +- `internal/validators/*` owns validator composition and built-in chain assembly. +- `internal/prompts` owns embedded prompt assets and metadata registry. + +## Where To Add New Code + +Add config fields: +- `internal/core/config` + +Add module behavior: +- one package under `internal/modules/` +- registration/wiring through `internal/framework/modules` and config module list + +Add validators: +- implementation under `internal/validators/` +- registry/chain wiring in `internal/validators` + +Add runtime orchestration behavior: +- `internal/framework/*` (runner/proposal/validator/LLM plumbing) + +Add CLI surface: +- `internal/cli` + +## Related Internal Docs + +- [`docs/internal/pipeline.md`](pipeline.md) +- [`docs/internal/modules.md`](modules.md) +- [`docs/internal/validators.md`](validators.md) +- [`docs/internal/llm-runtime.md`](llm-runtime.md) +- [`docs/internal/diagnostics-reporting.md`](diagnostics-reporting.md) +- [`docs/internal/prompts.md`](prompts.md) +- [`docs/internal/output-schemas.md`](output-schemas.md) diff --git a/docs/internal/pipeline.md b/docs/internal/pipeline.md new file mode 100644 index 0000000..5011449 --- /dev/null +++ b/docs/internal/pipeline.md @@ -0,0 +1,79 @@ +# Audita Internal Pipeline + +## Scope + +This document describes the implemented `audita process` pipeline. + +## Inputs + +Pipeline inputs are: +- effective config (`internal/core/config`); +- transcript JSON (`internal/core/schema`); +- glossary YAML (`internal/core/schema`). + +## Pipeline Phases + +1. Input loading and schema validation +- transcript and glossary files are read and parsed. +- schema failures stop the run with `transcript_schema` or `glossary_schema`. + +2. Normalization +- canonical transcript segments are normalized by configured gap/duration/token settings. +- normalization summary artifacts are written. + +3. Chunking +- normalized transcript is chunked with configured max/min tokens and target sections. + +4. Module proposal generation +- runner executes configured module instances in sequence. +- each module proposes corrections per section. +- per-section proposal generation can run concurrently. + +5. Validator filtering +- validators run on candidate proposals before apply. +- deterministic validators run before LLM-backed validators. +- LLM validator inputs are batched by max prompt token limit. + +6. Deterministic apply +- approved proposals are applied via replacement policy. +- applied/skipped/rejected outcomes are recorded. + +7. Output and reporting +- final transcript is serialized with selected output schema. +- report, utilization diagnostics, and correction ledger are written. +- retention policy is applied to run directory. + +## Runner Outputs + +`runner.Run` returns: +- final transcript; +- per-module results; +- utilization diagnostics. + +CLI/reporting then map this into process report and diagnostics artifacts. + +## Failure Behavior + +Representative failure phases include: +- `run_dir_creation` +- `transcript_read`, `glossary_read` +- `transcript_schema`, `glossary_schema` +- `chunking` +- `runner_setup`, `runner_execution` +- `output_schema`, `serialization`, `output_write`, `stdout_write` + +When diagnostics are available, failure stderr includes diagnostics path. + +## Invariants + +- module execution order follows configured module sequence; +- proposal/validator nondeterminism is isolated before deterministic apply; +- proposal indices are assigned deterministically by section order; +- output/report artifacts are generated from run results, not speculative state. + +## Key Tests + +- `internal/framework/runner/runner_test.go` +- `internal/framework/proposal_generation/*_test.go` +- `internal/cli/run_test.go` +- `cmd/audita/main_integration_test.go` diff --git a/docs/internal/prompts.md b/docs/internal/prompts.md new file mode 100644 index 0000000..b6543e4 --- /dev/null +++ b/docs/internal/prompts.md @@ -0,0 +1,62 @@ +# Audita Prompt Registry + +## Scope + +This document describes embedded prompt assets, prompt metadata, and rendering behavior. + +## Registry Ownership + +Prompt registry lives in `internal/prompts` and embeds assets under `internal/prompts/assets/**`. + +Registered prompt IDs: +- `modules.glossary.proposal` +- `modules.homophones.proposal` +- `modules.spoken_word.proposal` +- `modules.grammar.proposal` +- `validators.spoken_form_plausibility` +- `validators.meaning_reversal_review` +- `validators.editorial_review` +- `validators.grammar_review` +- `validators.spoken_word_review` + +## Metadata Model + +Each prompt has metadata: +- `prompt_id` +- `prompt_version` +- `prompt_source` +- `embedded_path` +- `sha256` + +Current source/version values: +- `prompt_source = builtin` +- `prompt_version = v1` + +## Rendering + +`prompts.RenderUserSystem(promptID, data)` renders system/user templates. + +Template behavior: +- uses Go `text/template`; +- `missingkey=error` is enabled; +- output is trimmed. + +A shared hardening fragment is embedded once and referenced by prompt templates. + +## Prompt Context Inputs + +Shared prompt payload helpers: +- transcript section JSON (`internal/framework/promptcontext/MarshalTranscriptSectionJSON`) +- transcript description block (`TranscriptDescriptionBlock`) + +Modules and LLM validators provide typed data maps to render prompt assets. + +## Diagnostics Integration + +Prompt metadata is attached to proposal/validator diagnostics request metadata using `Metadata.DiagnosticsMap()`. + +## Key Tests + +- `internal/prompts/registry_test.go` +- `internal/framework/promptcontext/*_test.go` +- module and validator prompt builder tests diff --git a/docs/internal/validators.md b/docs/internal/validators.md new file mode 100644 index 0000000..bddc231 --- /dev/null +++ b/docs/internal/validators.md @@ -0,0 +1,72 @@ +# Audita Validators + +## Scope + +This document describes validator composition, execution order, and decision handling. + +## Ownership + +Built-in validator keys and chains: +- `internal/validators` + +Shared validator runtime mechanics: +- `internal/framework/validators` + +Execution-class metadata: +- `internal/validators/metadata` + +## Built-In Validator Keys + +Deterministic: +- `proposal_shape` +- `confidence_threshold` +- `original_text_presence` +- `non_empty_corrected_text` +- `no_effect` +- `protected_terms` + +LLM-backed: +- `spoken_form_plausibility` +- `meaning_reversal_review` +- `editorial_review` + +## Built-In Chains + +Module chains are defined in `internal/validators/chains.go`. +Glossary, homophones, spoken_word, and grammar each resolve a fixed ordered chain. + +## Runtime Execution + +For each module section: +1. run deterministic validators; +2. run LLM-backed validators; +3. record decisions and warnings; +4. carry only approved proposals forward. + +Decision cardinality is enforced: each candidate proposal must receive exactly one decision per validator. + +## LLM Validator Batching + +LLM validators: +- build canonical validation request payloads; +- batch by `validation_max_prompt_tokens`; +- call structured LLM client using response schema registry. + +Oversized single proposals are rejected with `validator_input_too_large`. +Malformed LLM validator responses are downgraded to warnings and rejected batch decisions. + +## Decision and Rejection Reporting + +Runner records: +- `validator_decisions` +- `validator_rejected` +- warning records (including malformed response warnings) + +Correction ledger classifies deterministic vs LLM validator decisions using canonical metadata classes. + +## Key Tests + +- `internal/validators/*_test.go` +- `internal/framework/validators/*_test.go` +- `internal/framework/processreport/correction_ledger_test.go` +- `internal/cli/run_test.go`