Migrate internal architecture docs to docs/internal

docs/internal/diagnostics-reporting.md

@@ -0,0 +1,79 @@
+# Audita Diagnostics and Reporting
+
+## Scope
+
+This document describes diagnostics artifacts, process report mapping, and correction ledger generation.
+
+## Run Directory Ownership
+
+`internal/core/diagnostics` owns run-directory creation, artifact writes, and retention decisions.
+
+Stable artifact names include:
+- `source-transcript.json`
+- `source-transcript-parsed.json`
+- `normalized-transcript.json`
+- `normalization-summary.json`
+- `chunking-summary.json`
+- `utilization-diagnostics.json`
+- `correction-ledger.json`
+- `invocation.json`
+- `effective-config.json`
+- `report.json`
+- `error.log` (failure)
+
+## Process Report Mapping
+
+`internal/framework/processreport` maps runner/CLI execution facts into `reporting.ProcessReport`.
+
+Report metadata fields include:
+- `report_schema_name` (`audita-process-report`)
+- `report_schema_version` (`v1`)
+- `output_schema`
+- `config_version` (when file config exists)
+
+The report includes:
+- top-level status/error phase/error message;
+- normalization/chunking summaries;
+- diagnostics metadata paths;
+- per-module results and module summary.
+
+## Correction Ledger
+
+`internal/framework/processreport/BuildCorrectionLedger` flattens run results into `correction-ledger.json` entries.
+
+Dispositions:
+- `applied`
+- `skipped`
+- `rejected`
+- `failed`
+
+Validator decisions are split into deterministic and LLM-backed groups using validator metadata classification.
+
+## Report Write Paths
+
+- run directory always attempts to write `report.json` when possible;
+- optional `--report-json` writes an external report file;
+- on failure paths, report writing is best-effort and does not mask primary run errors.
+
+## Retention Interaction
+
+Current retention behavior:
+- failed runs are retained;
+- `always` keeps successful runs;
+- `auto` removes only clean successful runs;
+- `never` currently retains successful runs in current implementation.
+
+## Redaction
+
+Redacted data expectations:
+- effective config artifact uses config redaction;
+- diagnostics payloads and surfaced errors use LLM secret redaction;
+- reports should not include raw API key values.
+
+## Key Tests
+
+- `internal/core/diagnostics/*_test.go`
+- `internal/framework/processreport/*_test.go`
+- `internal/core/reporting/report_test.go`
+- `internal/cli/run_test.go`
+- `cmd/audita/main_integration_test.go`

docs/internal/llm-runtime.md

@@ -0,0 +1,67 @@
+# Audita LLM Runtime
+
+## Scope
+
+This document describes the structured LLM runtime and scheduler behavior.
+
+## Client Boundary
+
+All runtime LLM calls go through `contracts.StructuredLLMClient`.
+
+Primary adapter:
+- `internal/framework/llm/OpenAICompatibleClient`
+
+## Request/Response Behavior
+
+The OpenAI-compatible adapter sends chat completions requests with:
+- model;
+- messages;
+- `response_format.type = json_schema`;
+- strict schema envelope (`name`, `schema`, `strict=true`).
+
+The response is decoded into the requested structured output target.
+
+## Response Schema Registry
+
+Structured response schemas are registered in `internal/framework/responseschema`:
+- `correction_set`
+- `validator_decision_set`
+
+Each schema includes stable diagnostics metadata (`id`, `version`, `name`, `sha256`).
+
+## Retries and Error Handling
+
+Adapter retries apply to retryable conditions (for example transport/decoding/retryable status classes) up to configured `max_retries`.
+
+Errors are sanitized to redact configured API-key values before surfacing.
+
+Malformed structured output detection is shared through `internal/framework/structuredoutput` and is used by:
+- proposal generation;
+- LLM-backed validators.
+
+## Scheduling and Concurrency
+
+`internal/framework/llm/Scheduler` provides FIFO, context-aware permit gating.
+
+Runner composes scheduler limits across:
+- total LLM concurrency;
+- proposal LLM concurrency;
+- validation LLM concurrency.
+
+Scheduler release is guarded to avoid permit leaks on cancellation/error.
+
+## Diagnostics and Redaction
+
+`internal/framework/llm/DiagnosticsWriter` writes request/response/error artifacts.
+
+Configured secrets are derived from `llm.ConfiguredSecrets(cfg)` and redacted from:
+- diagnostics payloads;
+- surfaced runtime/adapter errors.
+
+## Key Tests
+
+- `internal/framework/llm/openai_compatible_client_test.go`
+- `internal/framework/llm/scheduler_test.go`
+- `internal/framework/llm/diagnostics_test.go`
+- `internal/framework/responseschema/registry_test.go`
+- `internal/framework/structuredoutput/malformed_test.go`

docs/internal/modules.md

@@ -0,0 +1,58 @@
+# Audita Modules
+
+## Scope
+
+This document covers module contracts and built-in module packages.
+
+## Module Contract
+
+Modules implement `contracts.TranscriptModule`:
+- `Key()`
+- `ReplacementPolicy()`
+- `Validators()`
+- `Propose(ctx, req)`
+
+Runner resolves configured module specs to module instances through `internal/framework/modules`.
+
+## Built-In Modules
+
+Current module packages:
+- `internal/modules/glossary`
+- `internal/modules/homophones`
+- `internal/modules/spoken_word`
+- `internal/modules/grammar`
+
+Current replacement policies:
+- `glossary`: `replace_all`
+- `homophones`: `require_unique`
+- `spoken_word`: `require_unique`
+- `grammar`: `require_unique`
+
+## Proposal Generation Ownership
+
+Shared proposal-generation plumbing is centralized in:
+- `internal/framework/proposal_generation`
+
+Module packages own:
+- prompt selection (`internal/prompts` prompt IDs);
+- module-specific prompt payload construction.
+
+Shared prompt helpers live in `internal/framework/promptcontext`.
+
+## Validator Chain Ownership
+
+Built-in chains are resolved in `internal/validators` per module key.
+Module packages call the built-in chain resolver at construction.
+
+## Failure and Warning Behavior
+
+- module setup failures surface as `runner_setup` or module setup errors;
+- module runtime failures surface as `runner_execution` with partial module results preserved;
+- malformed structured proposal payloads are downgraded to warnings and section-level proposal rejection.
+
+## Key Tests
+
+- `internal/modules/*/module_test.go`
+- `internal/framework/modules/registry_test.go`
+- `internal/framework/proposal_generation/*_test.go`
+- `internal/cli/run_test.go` (pipeline/report integration)

docs/internal/output-schemas.md

@@ -0,0 +1,46 @@
+# Audita Output Schemas
+
+## Scope
+
+This document describes the implemented transcript output schema registry.
+
+## Registry Ownership
+
+Output schema registry is owned by `internal/core/outputschema`.
+
+Supported schema keys:
+- `bare-segments`
+- `audita-v1`
+
+## Schemas
+
+`bare-segments`:
+- top-level JSON array of transcript segments.
+
+`audita-v1`:
+- top-level JSON object with:
+  - `schema: "audita-v1"`
+  - `version: "v1"`
+  - `segments: [...]`
+
+Segment fields include `id`, `speaker`, `start`, `end`, `text`, and optional `categories`.
+
+## Validation and Resolution
+
+Config validation and runtime resolution both reject unsupported schema keys.
+
+Unknown schema keys fail with `unsupported output schema` before output emission.
+
+## Output Emission
+
+The selected schema is used by `audita process` when writing:
+- output file (`--output`) or
+- stdout (when no `--output`).
+
+Report metadata records selected `output_schema`.
+
+## Key Tests
+
+- `internal/core/outputschema/registry_test.go`
+- `internal/core/config/config_test.go`
+- `internal/cli/run_test.go`

docs/internal/overview.md

@@ -0,0 +1,92 @@
+# Audita Internal Overview
+
+## Scope
+
+This document is the internal architecture entry point for developers and coding agents.
+
+It summarizes:
+- package boundaries;
+- the main `process` execution path;
+- where to add new code safely.
+
+## Package Map
+
+CLI and command orchestration:
+- `internal/cli`
+
+Core deterministic components:
+- `internal/core/config`
+- `internal/core/schema`
+- `internal/core/normalization`
+- `internal/core/chunking`
+- `internal/core/outputschema`
+- `internal/core/diagnostics`
+- `internal/core/reporting`
+- `internal/core/modulecatalog`
+
+Framework orchestration and contracts:
+- `internal/framework/contracts`
+- `internal/framework/modules`
+- `internal/framework/proposal_generation`
+- `internal/framework/proposals`
+- `internal/framework/runner`
+- `internal/framework/validators`
+- `internal/framework/llm`
+- `internal/framework/responseschema`
+- `internal/framework/structuredoutput`
+- `internal/framework/processreport`
+- `internal/framework/promptcontext`
+- `internal/framework/stagename`
+
+Domain implementations:
+- `internal/modules/*`
+- `internal/validators/*`
+- `internal/prompts`
+
+## Main Execution Path (`audita process`)
+
+High-level flow:
+1. CLI loads effective config and validates CLI requirements.
+2. Run directory is created and invocation/effective config artifacts are written.
+3. Transcript/glossary files are loaded and parsed.
+4. Transcript is normalized and chunked.
+5. `runner.Run` executes configured module instances.
+6. Proposals are validated, applied deterministically, and serialized in selected output schema.
+7. Process report, utilization diagnostics, correction ledger, and retention decisions are finalized.
+
+## Boundary Summary
+
+- `internal/core/*` owns deterministic, reusable logic and persistence-independent rules.
+- `internal/framework/*` owns orchestration contracts and reusable runtime plumbing.
+- `internal/modules/*` owns module-specific proposal behavior and prompt usage.
+- `internal/validators/*` owns validator composition and built-in chain assembly.
+- `internal/prompts` owns embedded prompt assets and metadata registry.
+
+## Where To Add New Code
+
+Add config fields:
+- `internal/core/config`
+
+Add module behavior:
+- one package under `internal/modules/<module_key>`
+- registration/wiring through `internal/framework/modules` and config module list
+
+Add validators:
+- implementation under `internal/validators/<validator_key>`
+- registry/chain wiring in `internal/validators`
+
+Add runtime orchestration behavior:
+- `internal/framework/*` (runner/proposal/validator/LLM plumbing)
+
+Add CLI surface:
+- `internal/cli`
+
+## Related Internal Docs
+
+- [`docs/internal/pipeline.md`](pipeline.md)
+- [`docs/internal/modules.md`](modules.md)
+- [`docs/internal/validators.md`](validators.md)
+- [`docs/internal/llm-runtime.md`](llm-runtime.md)
+- [`docs/internal/diagnostics-reporting.md`](diagnostics-reporting.md)
+- [`docs/internal/prompts.md`](prompts.md)
+- [`docs/internal/output-schemas.md`](output-schemas.md)

docs/internal/pipeline.md

@@ -0,0 +1,79 @@
+# Audita Internal Pipeline
+
+## Scope
+
+This document describes the implemented `audita process` pipeline.
+
+## Inputs
+
+Pipeline inputs are:
+- effective config (`internal/core/config`);
+- transcript JSON (`internal/core/schema`);
+- glossary YAML (`internal/core/schema`).
+
+## Pipeline Phases
+
+1. Input loading and schema validation
+- transcript and glossary files are read and parsed.
+- schema failures stop the run with `transcript_schema` or `glossary_schema`.
+
+2. Normalization
+- canonical transcript segments are normalized by configured gap/duration/token settings.
+- normalization summary artifacts are written.
+
+3. Chunking
+- normalized transcript is chunked with configured max/min tokens and target sections.
+
+4. Module proposal generation
+- runner executes configured module instances in sequence.
+- each module proposes corrections per section.
+- per-section proposal generation can run concurrently.
+
+5. Validator filtering
+- validators run on candidate proposals before apply.
+- deterministic validators run before LLM-backed validators.
+- LLM validator inputs are batched by max prompt token limit.
+
+6. Deterministic apply
+- approved proposals are applied via replacement policy.
+- applied/skipped/rejected outcomes are recorded.
+
+7. Output and reporting
+- final transcript is serialized with selected output schema.
+- report, utilization diagnostics, and correction ledger are written.
+- retention policy is applied to run directory.
+
+## Runner Outputs
+
+`runner.Run` returns:
+- final transcript;
+- per-module results;
+- utilization diagnostics.
+
+CLI/reporting then map this into process report and diagnostics artifacts.
+
+## Failure Behavior
+
+Representative failure phases include:
+- `run_dir_creation`
+- `transcript_read`, `glossary_read`
+- `transcript_schema`, `glossary_schema`
+- `chunking`
+- `runner_setup`, `runner_execution`
+- `output_schema`, `serialization`, `output_write`, `stdout_write`
+
+When diagnostics are available, failure stderr includes diagnostics path.
+
+## Invariants
+
+- module execution order follows configured module sequence;
+- proposal/validator nondeterminism is isolated before deterministic apply;
+- proposal indices are assigned deterministically by section order;
+- output/report artifacts are generated from run results, not speculative state.
+
+## Key Tests
+
+- `internal/framework/runner/runner_test.go`
+- `internal/framework/proposal_generation/*_test.go`
+- `internal/cli/run_test.go`
+- `cmd/audita/main_integration_test.go`

docs/internal/prompts.md

@@ -0,0 +1,62 @@
+# Audita Prompt Registry
+
+## Scope
+
+This document describes embedded prompt assets, prompt metadata, and rendering behavior.
+
+## Registry Ownership
+
+Prompt registry lives in `internal/prompts` and embeds assets under `internal/prompts/assets/**`.
+
+Registered prompt IDs:
+- `modules.glossary.proposal`
+- `modules.homophones.proposal`
+- `modules.spoken_word.proposal`
+- `modules.grammar.proposal`
+- `validators.spoken_form_plausibility`
+- `validators.meaning_reversal_review`
+- `validators.editorial_review`
+- `validators.grammar_review`
+- `validators.spoken_word_review`
+
+## Metadata Model
+
+Each prompt has metadata:
+- `prompt_id`
+- `prompt_version`
+- `prompt_source`
+- `embedded_path`
+- `sha256`
+
+Current source/version values:
+- `prompt_source = builtin`
+- `prompt_version = v1`
+
+## Rendering
+
+`prompts.RenderUserSystem(promptID, data)` renders system/user templates.
+
+Template behavior:
+- uses Go `text/template`;
+- `missingkey=error` is enabled;
+- output is trimmed.
+
+A shared hardening fragment is embedded once and referenced by prompt templates.
+
+## Prompt Context Inputs
+
+Shared prompt payload helpers:
+- transcript section JSON (`internal/framework/promptcontext/MarshalTranscriptSectionJSON`)
+- transcript description block (`TranscriptDescriptionBlock`)
+
+Modules and LLM validators provide typed data maps to render prompt assets.
+
+## Diagnostics Integration
+
+Prompt metadata is attached to proposal/validator diagnostics request metadata using `Metadata.DiagnosticsMap()`.
+
+## Key Tests
+
+- `internal/prompts/registry_test.go`
+- `internal/framework/promptcontext/*_test.go`
+- module and validator prompt builder tests

docs/internal/validators.md

@@ -0,0 +1,72 @@
+# Audita Validators
+
+## Scope
+
+This document describes validator composition, execution order, and decision handling.
+
+## Ownership
+
+Built-in validator keys and chains:
+- `internal/validators`
+
+Shared validator runtime mechanics:
+- `internal/framework/validators`
+
+Execution-class metadata:
+- `internal/validators/metadata`
+
+## Built-In Validator Keys
+
+Deterministic:
+- `proposal_shape`
+- `confidence_threshold`
+- `original_text_presence`
+- `non_empty_corrected_text`
+- `no_effect`
+- `protected_terms`
+
+LLM-backed:
+- `spoken_form_plausibility`
+- `meaning_reversal_review`
+- `editorial_review`
+
+## Built-In Chains
+
+Module chains are defined in `internal/validators/chains.go`.
+Glossary, homophones, spoken_word, and grammar each resolve a fixed ordered chain.
+
+## Runtime Execution
+
+For each module section:
+1. run deterministic validators;
+2. run LLM-backed validators;
+3. record decisions and warnings;
+4. carry only approved proposals forward.
+
+Decision cardinality is enforced: each candidate proposal must receive exactly one decision per validator.
+
+## LLM Validator Batching
+
+LLM validators:
+- build canonical validation request payloads;
+- batch by `validation_max_prompt_tokens`;
+- call structured LLM client using response schema registry.
+
+Oversized single proposals are rejected with `validator_input_too_large`.
+Malformed LLM validator responses are downgraded to warnings and rejected batch decisions.
+
+## Decision and Rejection Reporting
+
+Runner records:
+- `validator_decisions`
+- `validator_rejected`
+- warning records (including malformed response warnings)
+
+Correction ledger classifies deterministic vs LLM validator decisions using canonical metadata classes.
+
+## Key Tests
+
+- `internal/validators/*_test.go`
+- `internal/framework/validators/*_test.go`
+- `internal/framework/processreport/correction_ledger_test.go`
+- `internal/cli/run_test.go`