Complete Phase 8 deterministic validators
This commit is contained in:
@@ -19,14 +19,21 @@ Implemented today:
|
||||
- Framework foundation packages for contracts and proposal application.
|
||||
- Production runner orchestration package with deterministic sequential module execution.
|
||||
- Module-level report structures with applied/skipped change records.
|
||||
- Runtime validator models and deterministic validators.
|
||||
- Deterministic validator-chain execution in the runner with cardinality enforcement.
|
||||
- Module-level validator decision/rejection reporting.
|
||||
|
||||
Not implemented in CLI runtime path today:
|
||||
- Real module execution pipeline (`glossary`, `homophones`, `spoken_word`, `grammar`).
|
||||
- Structured LLM proposal generation.
|
||||
- Validator chain execution.
|
||||
- LLM-backed validators.
|
||||
- Production LLM scheduler behavior.
|
||||
- End-to-end transcript polishing with real module behavior.
|
||||
|
||||
Phase sequencing note:
|
||||
- structured LLM client and scheduler infrastructure remain Phase 9 work;
|
||||
- LLM-backed validators remain Phase 10 work.
|
||||
|
||||
## Actual Go package layout
|
||||
|
||||
```text
|
||||
@@ -77,6 +84,10 @@ internal/framework/proposals/
|
||||
|
||||
internal/framework/runner/
|
||||
runner.go
|
||||
|
||||
internal/framework/validators/
|
||||
models.go
|
||||
deterministic.go
|
||||
```
|
||||
|
||||
## Current CLI behavior
|
||||
@@ -100,7 +111,7 @@ Current runtime flow (`internal/cli/run.go`):
|
||||
11. Write chunking summary artifact.
|
||||
12. Optionally execute runner modules sequentially when an injected module registry/factory is available (used by deterministic tests today).
|
||||
13. Output working transcript to `--output` file or stdout.
|
||||
14. Build process report (`phase` currently set to `phase7-runner`).
|
||||
14. Build process report (`phase` currently set to `phase8-validators`).
|
||||
15. Optionally write `--report-json`; always write run-dir `report.json`.
|
||||
16. Apply work-dir retention.
|
||||
|
||||
@@ -194,6 +205,24 @@ Current behavior details:
|
||||
|
||||
These primitives are wired into the production runner and report model. Real module implementations are still pending.
|
||||
|
||||
## Implemented validator runtime infrastructure
|
||||
`internal/framework/validators` provides deterministic validator infrastructure:
|
||||
- runtime validation request/result models;
|
||||
- stable validator reason codes;
|
||||
- cardinality enforcement for validator decisions:
|
||||
- missing proposal indexes fail
|
||||
- duplicate proposal indexes fail
|
||||
- unknown proposal indexes fail
|
||||
- deterministic validators:
|
||||
- confidence threshold by module key/config threshold
|
||||
- original-text presence against current working transcript
|
||||
- non-empty corrected text
|
||||
- identical/no-effect rejection
|
||||
- conservative protected glossary-term guard for non-glossary modules
|
||||
|
||||
`internal/framework/runner` executes validator chains in order for each module and applies only validator-approved proposals.
|
||||
Validator rejections are reported distinctly from proposal-application skips.
|
||||
|
||||
## Reports and diagnostics (implemented)
|
||||
Current per-run artifacts include:
|
||||
- `source-transcript.json`
|
||||
@@ -222,6 +251,7 @@ Current process reports include diagnostics metadata references for:
|
||||
Current process reports also include:
|
||||
- module-level results (when runner modules execute), including applied/skipped proposal changes;
|
||||
- run-level module summary totals and failed module instance metadata.
|
||||
- module-level validator decisions and validator rejections.
|
||||
|
||||
Retention modes implemented in `ApplyRetention`:
|
||||
- `always`: keep all run directories.
|
||||
@@ -247,8 +277,9 @@ Implemented tests currently cover:
|
||||
- contracts/foundation composition tests (`internal/framework/contracts/*_test.go`)
|
||||
- runner sequencing and failure behavior with deterministic fake modules (`internal/framework/runner/*_test.go`)
|
||||
- CLI runner integration through injected fake module factories (`internal/cli/run_test.go`)
|
||||
- validator models, cardinality enforcement, and deterministic validators (`internal/framework/validators/*_test.go`)
|
||||
|
||||
Not covered yet (because not implemented): validator runtime flow with approvals/rejections and real LLM integration.
|
||||
Not covered yet (because not implemented): real LLM validator/runtime integration and production module behavior.
|
||||
|
||||
## Intended final architecture (not yet implemented)
|
||||
The intended end-state still matches the rewrite plan:
|
||||
|
||||
@@ -47,12 +47,16 @@ Implemented:
|
||||
- Production runner orchestration over a mutable working transcript.
|
||||
- Module-level report structures and run-level module summaries.
|
||||
- CLI runner integration point via injectable module factory/registry (used by deterministic tests).
|
||||
- Runtime validator models and deterministic validator implementations.
|
||||
- Validator cardinality enforcement (missing/duplicate/unknown proposal index errors).
|
||||
- Deterministic validator-chain execution in the production runner.
|
||||
- Module reports including validator decisions and validator rejections.
|
||||
- Broad deterministic and CLI/subprocess test coverage for implemented phases through `go test ./...`.
|
||||
|
||||
Not yet implemented in runtime pipeline:
|
||||
- Real correction modules.
|
||||
- Validator-chain execution.
|
||||
- Structured LLM client integration.
|
||||
- LLM-backed validators.
|
||||
- Prompt/response diagnostics for LLM calls.
|
||||
- End-to-end transcript polishing behavior.
|
||||
|
||||
@@ -161,21 +165,12 @@ Not implemented in Phase 7 (by design):
|
||||
Current runtime behavior note:
|
||||
- Default user-facing CLI behavior remains deterministic normalization/chunking output unless test-only module injection is used during tests.
|
||||
|
||||
## Remaining work plan
|
||||
|
||||
Next recommended phase: **Phase 8 (runtime validator framework and deterministic validators)**.
|
||||
|
||||
## Phase 8: Runtime validator framework and deterministic validators
|
||||
Completed.
|
||||
|
||||
### Purpose
|
||||
|
||||
Wire validator-chain execution into the runner using deterministic validators first. This phase establishes the safety model before any real LLM proposal generation is introduced.
|
||||
|
||||
### Scope
|
||||
|
||||
Implement:
|
||||
- Runtime validator interfaces if the existing contracts need refinement.
|
||||
- Validation request/result models.
|
||||
Implemented:
|
||||
- Runtime validator request/result models in `internal/framework/validators`.
|
||||
- Deterministic validator reason codes for stable reporting.
|
||||
- Validator cardinality enforcement:
|
||||
- one decision per candidate proposal index
|
||||
- missing indexes are errors
|
||||
@@ -183,33 +178,25 @@ Implement:
|
||||
- unknown indexes are errors
|
||||
- Deterministic validators:
|
||||
- confidence threshold
|
||||
- original-text presence
|
||||
- original-text presence against working transcript
|
||||
- non-empty correction
|
||||
- identical text/no-effect rejection
|
||||
- protected glossary term logic, if it can be implemented deterministically from current glossary schema
|
||||
- Validator ordering.
|
||||
- Runner integration so candidate proposals pass through validators before application.
|
||||
- Module report fields for validator approvals/rejections.
|
||||
- identical/no-effect rejection
|
||||
- conservative protected glossary-term guard
|
||||
- Ordered validator-chain execution in the production runner.
|
||||
- Runner behavior where only validator-approved proposals proceed to proposal application.
|
||||
- Module-level reporting of validator decisions and validator rejections, distinct from application-level skips.
|
||||
- Deterministic fake-module tests covering approvals, rejections, validator order/filtering, and cardinality failure pipeline-stop behavior.
|
||||
|
||||
Do not implement:
|
||||
- LLM-backed validators.
|
||||
- Real modules.
|
||||
- Real LLM proposal generation.
|
||||
Not implemented in Phase 8 (by design):
|
||||
- LLM-backed validators (Phase 10).
|
||||
- Structured LLM client implementation or scheduler behavior (Phase 9).
|
||||
- Real correction modules.
|
||||
- Prompt/response diagnostics.
|
||||
- End-to-end transcript polishing.
|
||||
|
||||
### Expected behavior at end of phase
|
||||
## Remaining work plan
|
||||
|
||||
Fake modules can generate deterministic proposals, those proposals can be filtered by deterministic validators, and only approved proposals are applied.
|
||||
|
||||
### Definition of done
|
||||
|
||||
- Validator chains run in the production runner.
|
||||
- Deterministic validators are implemented and tested.
|
||||
- Validator cardinality enforcement is tested.
|
||||
- Rejected proposals appear in module reports with stable reasons.
|
||||
- Approved proposals are applied through existing proposal application semantics.
|
||||
- No real LLM calls occur.
|
||||
- `go test ./...` passes.
|
||||
Next recommended phase: **Phase 9 (structured LLM client and scheduler infrastructure)**.
|
||||
|
||||
## Phase 9: Structured LLM client and scheduler infrastructure
|
||||
|
||||
|
||||
Reference in New Issue
Block a user