Complete Phase 10 LLM validators

docs/architecture.md

@@ -26,18 +26,20 @@ Implemented today:
 - Bounded LLM scheduler/semaphore infrastructure with context-aware permit handling.
 - Runtime primary/validation LLM effective-config resolution helpers with validation inheritance.
 - Generic JSON prompt/response diagnostics writer primitives with secret redaction.
+- LLM-backed validator models, prompt builders, batching, and runtime execution.
+- Runner wiring for LLM validators via the internal structured LLM abstraction and scheduler hooks.
+- LLM validator diagnostics artifacts and report-level decision metadata paths.
 
 Not implemented in CLI runtime path today:
 - Real module execution pipeline (`glossary`, `homophones`, `spoken_word`, `grammar`).
 - Structured LLM proposal generation.
-- LLM-backed validators.
-- Runtime module/validator usage of the LLM scheduler infrastructure.
+- Shared module proposal-generation framework and module registry for real modules.
 - End-to-end transcript polishing with real module behavior.
 
 Phase sequencing note:
 - Phase 9 LLM infrastructure is complete (structured client, scheduler, effective config resolution, diagnostics primitives);
-- runtime wiring from modules/validators/runner into this LLM infrastructure remains future module/validator phase work;
-- LLM-backed validators remain Phase 10 work.
+- Phase 10 LLM-backed validator runtime integration is complete;
+- shared module proposal-generation and module registry work remain Phase 11.
 
 ## Actual Go package layout
 
@@ -93,6 +95,10 @@ internal/framework/runner/
 internal/framework/validators/
   models.go
   deterministic.go
+  llm_models.go
+  llm_prompt_builders.go
+  llm_batching.go
+  llm_validators.go
 
 internal/framework/llm/
   instructor_client.go
@@ -122,14 +128,14 @@ Current runtime flow (`internal/cli/run.go`):
 11. Write chunking summary artifact.
 12. Optionally execute runner modules sequentially when an injected module registry/factory is available (used by deterministic tests today).
 13. Output working transcript to `--output` file or stdout.
-14. Build process report (`phase` currently set to `phase8-validators`).
+14. Build process report (`phase` currently set to `phase10-llm-validators`).
 15. Optionally write `--report-json`; always write run-dir `report.json`.
 16. Apply work-dir retention.
 
 Important behavior details:
 - Glossary is validated but not yet used for real correction module logic.
 - Default production CLI behavior remains deterministic normalization/chunking output because no real module implementations are registered yet.
-- No LLM calls occur.
+- No real LLM calls occur in the default production runtime path because no real modules are registered yet.
 - Success path is generally quiet on stderr.
 - Source IDs are preserved into a canonical transcript before normalization; normalization then reassigns output IDs sequentially from `1`.
 
@@ -193,8 +199,8 @@ Current caveat:
 - API-key redaction in adapter-returned errors.
 
 Current runtime boundary:
-- the CLI/runner runtime path does not instantiate this adapter yet;
-- no production LLM requests are performed by `audita process`.
+- the default CLI runtime path still does not instantiate real production modules, so no default end-to-end LLM polishing occurs.
+- LLM calls are exercised only when test/injected modules and validators are provided.
 
 `internal/framework/llm` also provides:
 - a bounded `Scheduler` for controlled concurrent LLM calls with reliable permit release;
@@ -256,6 +262,24 @@ These primitives are wired into the production runner and report model. Real mod
 `internal/framework/runner` executes validator chains in order for each module and applies only validator-approved proposals.
 Validator rejections are reported distinctly from proposal-application skips.
 
+## Implemented LLM-backed validator infrastructure
+`internal/framework/validators` now includes LLM-backed validator support:
+- typed request/response models for structured LLM validation;
+- prompt builders for:
+  - spoken-form plausibility
+  - meaning reversal detection
+  - editorial review
+  - grammar review
+  - spoken-word review
+- deterministic batching by `validation_max_prompt_tokens`;
+- strict cardinality validation of structured LLM decisions (missing/duplicate/unknown indexes fail);
+- safe failure behavior for malformed/invalid structured responses.
+
+`internal/framework/runner` wires LLM validators into existing validator chains using:
+- the internal structured LLM client abstraction (`contracts.StructuredLLMClient`);
+- bounded scheduler hooks for validator call execution;
+- diagnostics writer hooks for machine-readable prompt/response artifacts with secret redaction.
+
 ## Reports and diagnostics (implemented)
 Current per-run artifacts include:
 - `source-transcript.json`
@@ -285,6 +309,7 @@ Current process reports also include:
 - module-level results (when runner modules execute), including applied/skipped proposal changes;
 - run-level module summary totals and failed module instance metadata.
 - module-level validator decisions and validator rejections.
+- optional decision-level diagnostic artifact paths for validator LLM interactions when available.
 
 Retention modes implemented in `ApplyRetention`:
 - `always`: keep all run directories.
@@ -296,7 +321,7 @@ Current runtime note:
 - real module execution is not implemented yet, so normal successful runs generally have no skipped corrections and `auto` typically removes clean successful run directories.
 
 Intentionally deferred to module/LLM phases:
-- module prompt/response diagnostics artifacts are not produced yet because module execution and LLM calls are not in the runtime path.
+- module proposal-generation prompt/response diagnostics remain tied to later real-module phases.
 
 ## Current tests and quality posture
 Implemented tests currently cover:
@@ -311,8 +336,9 @@ Implemented tests currently cover:
 - runner sequencing and failure behavior with deterministic fake modules (`internal/framework/runner/*_test.go`)
 - CLI runner integration through injected fake module factories (`internal/cli/run_test.go`)
 - validator models, cardinality enforcement, and deterministic validators (`internal/framework/validators/*_test.go`)
+- LLM-backed validator batching, prompt builders, structured-response safety, scheduler hooks, and diagnostics redaction (`internal/framework/validators/*_test.go`, `internal/framework/runner/*_test.go`)
 
-Not covered yet (because not implemented): real LLM validator/runtime integration and production module behavior.
+Not covered yet (because not implemented): shared module proposal generation, real module implementations, and full transcript-polishing runtime behavior.
 
 ## Intended final architecture (not yet implemented)
 The intended end-state still matches the rewrite plan:

docs/rewrite-notes.md

@@ -51,6 +51,10 @@ Implemented:
 - Validator cardinality enforcement (missing/duplicate/unknown proposal index errors).
 - Deterministic validator-chain execution in the production runner.
 - Module reports including validator decisions and validator rejections.
+- LLM-backed validator request/response models and prompt builders.
+- LLM validator batching by validation prompt-token budget.
+- LLM validator runtime integration through structured LLM client abstraction and scheduler hooks.
+- LLM validator prompt/response diagnostics artifact wiring with secret redaction.
 - Broad deterministic and CLI/subprocess test coverage for implemented phases through `go test ./...`.
 - Internal typed structured LLM contract (`StructuredLLMClient.CompleteStructured(ctx, req, out)`).
 - `internal/framework/llm` instructor-go-backed adapter with:
@@ -67,9 +71,7 @@ Implemented:
 
 Not yet implemented in runtime pipeline:
 - Real correction modules.
-- Runtime wiring from production runner/modules into the structured LLM adapter.
-- LLM-backed validators.
-- Module/validator call-site wiring to emit LLM prompt/response diagnostics artifacts.
+- Shared module proposal generation and module registry wiring.
 - End-to-end transcript polishing behavior.
 
 ## Completed phases
@@ -208,7 +210,7 @@ Not implemented in Phase 8 (by design):
 
 ## Remaining work plan
 
-Next recommended phase: **Phase 10 (LLM-backed validators)**.
+Next recommended phase: **Phase 11 (shared LLM proposal generation framework and module registry)**.
 
 ## Phase 9: Structured LLM client and scheduler infrastructure
 
@@ -275,45 +277,30 @@ Met:
 
 ## Phase 10: LLM-backed validators
 
-### Purpose
+Completed.
 
-Implement the LLM-backed validator layer used by the Python implementation, and wire it into the runtime validator framework.
-
-### Scope
-
-Implement:
-- LLM validator request and response models.
-- Shared batching logic for validation prompts using validation token limits.
-- Prompt builders for LLM validators.
-- LLM-backed validation categories needed for parity, such as:
+Implemented:
+- LLM-backed validator request/response models in `internal/framework/validators`.
+- Prompt builders for:
   - spoken-form plausibility
   - meaning reversal detection
   - editorial review
   - grammar review
   - spoken-word review
-- Validator prompt/response diagnostics.
-- Validation LLM scheduler usage.
-- Validator error handling and report integration.
-- Fake LLM tests for approval, rejection, malformed output, missing decision, duplicate decision, and retry cases.
+- Deterministic batching by `validation_max_prompt_tokens` with stable ordering and no drop/dup behavior.
+- LLM validator execution through the internal structured client abstraction (no direct provider calls in validator code).
+- Scheduler/concurrency hooks for LLM validator calls.
+- Prompt/response diagnostics artifact writing for LLM validator batches using Phase 9 diagnostics primitives.
+- Secret redaction in validator LLM diagnostics artifacts.
+- Strict structured-response safety and cardinality checks (missing/duplicate/unknown indexes fail closed).
+- Runner/report integration so LLM validator decisions and rejections appear in module reports.
+- Fake-module and fake-client tests for approval/rejection, malformed output, cardinality errors, batching, scheduler usage, and diagnostics redaction.
 
-Do not implement:
-- Real correction modules, except for minimal fake/test modules needed to exercise validators.
-- Full default pipeline behavior.
+Not implemented in Phase 10 (by design):
+- Real correction modules (`glossary`, `homophones`, `spoken_word`, `grammar`).
+- Shared module proposal generation and module registry work (Phase 11).
 - Domain proposal prompts.
-
-### Expected behavior at end of phase
-
-The runner can execute a mixed deterministic + LLM validator chain against proposals produced by fake modules. LLM validators use the structured LLM client and write diagnostics.
-
-### Definition of done
-
-- LLM-backed validators are implemented.
-- Validator batching respects configured token limits.
-- Validator cardinality rules are enforced for LLM validator output.
-- Prompt/response diagnostics are written for LLM validator calls.
-- Validator results appear in module reports.
-- Fake LLM tests cover success, rejection, malformed output, and retry behavior.
-- `go test ./...` passes.
+- Default CLI end-to-end transcript polishing behavior.
 
 ## Phase 11: Shared LLM proposal generation framework and module registry