Complete Phase 10 LLM validators

This commit is contained in:
2026-05-12 01:25:52 +00:00
parent 6d9a4bd017
commit 12202508bf
14 changed files with 1213 additions and 82 deletions

View File

@@ -51,6 +51,10 @@ Implemented:
- Validator cardinality enforcement (missing/duplicate/unknown proposal index errors).
- Deterministic validator-chain execution in the production runner.
- Module reports including validator decisions and validator rejections.
- LLM-backed validator request/response models and prompt builders.
- LLM validator batching by validation prompt-token budget.
- LLM validator runtime integration through structured LLM client abstraction and scheduler hooks.
- LLM validator prompt/response diagnostics artifact wiring with secret redaction.
- Broad deterministic and CLI/subprocess test coverage for implemented phases through `go test ./...`.
- Internal typed structured LLM contract (`StructuredLLMClient.CompleteStructured(ctx, req, out)`).
- `internal/framework/llm` instructor-go-backed adapter with:
@@ -67,9 +71,7 @@ Implemented:
Not yet implemented in runtime pipeline:
- Real correction modules.
- Runtime wiring from production runner/modules into the structured LLM adapter.
- LLM-backed validators.
- Module/validator call-site wiring to emit LLM prompt/response diagnostics artifacts.
- Shared module proposal generation and module registry wiring.
- End-to-end transcript polishing behavior.
## Completed phases
@@ -208,7 +210,7 @@ Not implemented in Phase 8 (by design):
## Remaining work plan
Next recommended phase: **Phase 10 (LLM-backed validators)**.
Next recommended phase: **Phase 11 (shared LLM proposal generation framework and module registry)**.
## Phase 9: Structured LLM client and scheduler infrastructure
@@ -275,45 +277,30 @@ Met:
## Phase 10: LLM-backed validators
### Purpose
Completed.
Implement the LLM-backed validator layer used by the Python implementation, and wire it into the runtime validator framework.
### Scope
Implement:
- LLM validator request and response models.
- Shared batching logic for validation prompts using validation token limits.
- Prompt builders for LLM validators.
- LLM-backed validation categories needed for parity, such as:
Implemented:
- LLM-backed validator request/response models in `internal/framework/validators`.
- Prompt builders for:
- spoken-form plausibility
- meaning reversal detection
- editorial review
- grammar review
- spoken-word review
- Validator prompt/response diagnostics.
- Validation LLM scheduler usage.
- Validator error handling and report integration.
- Fake LLM tests for approval, rejection, malformed output, missing decision, duplicate decision, and retry cases.
- Deterministic batching by `validation_max_prompt_tokens` with stable ordering and no drop/dup behavior.
- LLM validator execution through the internal structured client abstraction (no direct provider calls in validator code).
- Scheduler/concurrency hooks for LLM validator calls.
- Prompt/response diagnostics artifact writing for LLM validator batches using Phase 9 diagnostics primitives.
- Secret redaction in validator LLM diagnostics artifacts.
- Strict structured-response safety and cardinality checks (missing/duplicate/unknown indexes fail closed).
- Runner/report integration so LLM validator decisions and rejections appear in module reports.
- Fake-module and fake-client tests for approval/rejection, malformed output, cardinality errors, batching, scheduler usage, and diagnostics redaction.
Do not implement:
- Real correction modules, except for minimal fake/test modules needed to exercise validators.
- Full default pipeline behavior.
Not implemented in Phase 10 (by design):
- Real correction modules (`glossary`, `homophones`, `spoken_word`, `grammar`).
- Shared module proposal generation and module registry work (Phase 11).
- Domain proposal prompts.
### Expected behavior at end of phase
The runner can execute a mixed deterministic + LLM validator chain against proposals produced by fake modules. LLM validators use the structured LLM client and write diagnostics.
### Definition of done
- LLM-backed validators are implemented.
- Validator batching respects configured token limits.
- Validator cardinality rules are enforced for LLM validator output.
- Prompt/response diagnostics are written for LLM validator calls.
- Validator results appear in module reports.
- Fake LLM tests cover success, rejection, malformed output, and retry behavior.
- `go test ./...` passes.
- Default CLI end-to-end transcript polishing behavior.
## Phase 11: Shared LLM proposal generation framework and module registry