Complete Phase 10 LLM validators
This commit is contained in:
@@ -51,6 +51,10 @@ Implemented:
|
||||
- Validator cardinality enforcement (missing/duplicate/unknown proposal index errors).
|
||||
- Deterministic validator-chain execution in the production runner.
|
||||
- Module reports including validator decisions and validator rejections.
|
||||
- LLM-backed validator request/response models and prompt builders.
|
||||
- LLM validator batching by validation prompt-token budget.
|
||||
- LLM validator runtime integration through structured LLM client abstraction and scheduler hooks.
|
||||
- LLM validator prompt/response diagnostics artifact wiring with secret redaction.
|
||||
- Broad deterministic and CLI/subprocess test coverage for implemented phases through `go test ./...`.
|
||||
- Internal typed structured LLM contract (`StructuredLLMClient.CompleteStructured(ctx, req, out)`).
|
||||
- `internal/framework/llm` instructor-go-backed adapter with:
|
||||
@@ -67,9 +71,7 @@ Implemented:
|
||||
|
||||
Not yet implemented in runtime pipeline:
|
||||
- Real correction modules.
|
||||
- Runtime wiring from production runner/modules into the structured LLM adapter.
|
||||
- LLM-backed validators.
|
||||
- Module/validator call-site wiring to emit LLM prompt/response diagnostics artifacts.
|
||||
- Shared module proposal generation and module registry wiring.
|
||||
- End-to-end transcript polishing behavior.
|
||||
|
||||
## Completed phases
|
||||
@@ -208,7 +210,7 @@ Not implemented in Phase 8 (by design):
|
||||
|
||||
## Remaining work plan
|
||||
|
||||
Next recommended phase: **Phase 10 (LLM-backed validators)**.
|
||||
Next recommended phase: **Phase 11 (shared LLM proposal generation framework and module registry)**.
|
||||
|
||||
## Phase 9: Structured LLM client and scheduler infrastructure
|
||||
|
||||
@@ -275,45 +277,30 @@ Met:
|
||||
|
||||
## Phase 10: LLM-backed validators
|
||||
|
||||
### Purpose
|
||||
Completed.
|
||||
|
||||
Implement the LLM-backed validator layer used by the Python implementation, and wire it into the runtime validator framework.
|
||||
|
||||
### Scope
|
||||
|
||||
Implement:
|
||||
- LLM validator request and response models.
|
||||
- Shared batching logic for validation prompts using validation token limits.
|
||||
- Prompt builders for LLM validators.
|
||||
- LLM-backed validation categories needed for parity, such as:
|
||||
Implemented:
|
||||
- LLM-backed validator request/response models in `internal/framework/validators`.
|
||||
- Prompt builders for:
|
||||
- spoken-form plausibility
|
||||
- meaning reversal detection
|
||||
- editorial review
|
||||
- grammar review
|
||||
- spoken-word review
|
||||
- Validator prompt/response diagnostics.
|
||||
- Validation LLM scheduler usage.
|
||||
- Validator error handling and report integration.
|
||||
- Fake LLM tests for approval, rejection, malformed output, missing decision, duplicate decision, and retry cases.
|
||||
- Deterministic batching by `validation_max_prompt_tokens` with stable ordering and no drop/dup behavior.
|
||||
- LLM validator execution through the internal structured client abstraction (no direct provider calls in validator code).
|
||||
- Scheduler/concurrency hooks for LLM validator calls.
|
||||
- Prompt/response diagnostics artifact writing for LLM validator batches using Phase 9 diagnostics primitives.
|
||||
- Secret redaction in validator LLM diagnostics artifacts.
|
||||
- Strict structured-response safety and cardinality checks (missing/duplicate/unknown indexes fail closed).
|
||||
- Runner/report integration so LLM validator decisions and rejections appear in module reports.
|
||||
- Fake-module and fake-client tests for approval/rejection, malformed output, cardinality errors, batching, scheduler usage, and diagnostics redaction.
|
||||
|
||||
Do not implement:
|
||||
- Real correction modules, except for minimal fake/test modules needed to exercise validators.
|
||||
- Full default pipeline behavior.
|
||||
Not implemented in Phase 10 (by design):
|
||||
- Real correction modules (`glossary`, `homophones`, `spoken_word`, `grammar`).
|
||||
- Shared module proposal generation and module registry work (Phase 11).
|
||||
- Domain proposal prompts.
|
||||
|
||||
### Expected behavior at end of phase
|
||||
|
||||
The runner can execute a mixed deterministic + LLM validator chain against proposals produced by fake modules. LLM validators use the structured LLM client and write diagnostics.
|
||||
|
||||
### Definition of done
|
||||
|
||||
- LLM-backed validators are implemented.
|
||||
- Validator batching respects configured token limits.
|
||||
- Validator cardinality rules are enforced for LLM validator output.
|
||||
- Prompt/response diagnostics are written for LLM validator calls.
|
||||
- Validator results appear in module reports.
|
||||
- Fake LLM tests cover success, rejection, malformed output, and retry behavior.
|
||||
- `go test ./...` passes.
|
||||
- Default CLI end-to-end transcript polishing behavior.
|
||||
|
||||
## Phase 11: Shared LLM proposal generation framework and module registry
|
||||
|
||||
|
||||
Reference in New Issue
Block a user