Complete Phase 8 deterministic validators
This commit is contained in:
@@ -19,14 +19,21 @@ Implemented today:
|
|||||||
- Framework foundation packages for contracts and proposal application.
|
- Framework foundation packages for contracts and proposal application.
|
||||||
- Production runner orchestration package with deterministic sequential module execution.
|
- Production runner orchestration package with deterministic sequential module execution.
|
||||||
- Module-level report structures with applied/skipped change records.
|
- Module-level report structures with applied/skipped change records.
|
||||||
|
- Runtime validator models and deterministic validators.
|
||||||
|
- Deterministic validator-chain execution in the runner with cardinality enforcement.
|
||||||
|
- Module-level validator decision/rejection reporting.
|
||||||
|
|
||||||
Not implemented in CLI runtime path today:
|
Not implemented in CLI runtime path today:
|
||||||
- Real module execution pipeline (`glossary`, `homophones`, `spoken_word`, `grammar`).
|
- Real module execution pipeline (`glossary`, `homophones`, `spoken_word`, `grammar`).
|
||||||
- Structured LLM proposal generation.
|
- Structured LLM proposal generation.
|
||||||
- Validator chain execution.
|
- LLM-backed validators.
|
||||||
- Production LLM scheduler behavior.
|
- Production LLM scheduler behavior.
|
||||||
- End-to-end transcript polishing with real module behavior.
|
- End-to-end transcript polishing with real module behavior.
|
||||||
|
|
||||||
|
Phase sequencing note:
|
||||||
|
- structured LLM client and scheduler infrastructure remain Phase 9 work;
|
||||||
|
- LLM-backed validators remain Phase 10 work.
|
||||||
|
|
||||||
## Actual Go package layout
|
## Actual Go package layout
|
||||||
|
|
||||||
```text
|
```text
|
||||||
@@ -77,6 +84,10 @@ internal/framework/proposals/
|
|||||||
|
|
||||||
internal/framework/runner/
|
internal/framework/runner/
|
||||||
runner.go
|
runner.go
|
||||||
|
|
||||||
|
internal/framework/validators/
|
||||||
|
models.go
|
||||||
|
deterministic.go
|
||||||
```
|
```
|
||||||
|
|
||||||
## Current CLI behavior
|
## Current CLI behavior
|
||||||
@@ -100,7 +111,7 @@ Current runtime flow (`internal/cli/run.go`):
|
|||||||
11. Write chunking summary artifact.
|
11. Write chunking summary artifact.
|
||||||
12. Optionally execute runner modules sequentially when an injected module registry/factory is available (used by deterministic tests today).
|
12. Optionally execute runner modules sequentially when an injected module registry/factory is available (used by deterministic tests today).
|
||||||
13. Output working transcript to `--output` file or stdout.
|
13. Output working transcript to `--output` file or stdout.
|
||||||
14. Build process report (`phase` currently set to `phase7-runner`).
|
14. Build process report (`phase` currently set to `phase8-validators`).
|
||||||
15. Optionally write `--report-json`; always write run-dir `report.json`.
|
15. Optionally write `--report-json`; always write run-dir `report.json`.
|
||||||
16. Apply work-dir retention.
|
16. Apply work-dir retention.
|
||||||
|
|
||||||
@@ -194,6 +205,24 @@ Current behavior details:
|
|||||||
|
|
||||||
These primitives are wired into the production runner and report model. Real module implementations are still pending.
|
These primitives are wired into the production runner and report model. Real module implementations are still pending.
|
||||||
|
|
||||||
|
## Implemented validator runtime infrastructure
|
||||||
|
`internal/framework/validators` provides deterministic validator infrastructure:
|
||||||
|
- runtime validation request/result models;
|
||||||
|
- stable validator reason codes;
|
||||||
|
- cardinality enforcement for validator decisions:
|
||||||
|
- missing proposal indexes fail
|
||||||
|
- duplicate proposal indexes fail
|
||||||
|
- unknown proposal indexes fail
|
||||||
|
- deterministic validators:
|
||||||
|
- confidence threshold by module key/config threshold
|
||||||
|
- original-text presence against current working transcript
|
||||||
|
- non-empty corrected text
|
||||||
|
- identical/no-effect rejection
|
||||||
|
- conservative protected glossary-term guard for non-glossary modules
|
||||||
|
|
||||||
|
`internal/framework/runner` executes validator chains in order for each module and applies only validator-approved proposals.
|
||||||
|
Validator rejections are reported distinctly from proposal-application skips.
|
||||||
|
|
||||||
## Reports and diagnostics (implemented)
|
## Reports and diagnostics (implemented)
|
||||||
Current per-run artifacts include:
|
Current per-run artifacts include:
|
||||||
- `source-transcript.json`
|
- `source-transcript.json`
|
||||||
@@ -222,6 +251,7 @@ Current process reports include diagnostics metadata references for:
|
|||||||
Current process reports also include:
|
Current process reports also include:
|
||||||
- module-level results (when runner modules execute), including applied/skipped proposal changes;
|
- module-level results (when runner modules execute), including applied/skipped proposal changes;
|
||||||
- run-level module summary totals and failed module instance metadata.
|
- run-level module summary totals and failed module instance metadata.
|
||||||
|
- module-level validator decisions and validator rejections.
|
||||||
|
|
||||||
Retention modes implemented in `ApplyRetention`:
|
Retention modes implemented in `ApplyRetention`:
|
||||||
- `always`: keep all run directories.
|
- `always`: keep all run directories.
|
||||||
@@ -247,8 +277,9 @@ Implemented tests currently cover:
|
|||||||
- contracts/foundation composition tests (`internal/framework/contracts/*_test.go`)
|
- contracts/foundation composition tests (`internal/framework/contracts/*_test.go`)
|
||||||
- runner sequencing and failure behavior with deterministic fake modules (`internal/framework/runner/*_test.go`)
|
- runner sequencing and failure behavior with deterministic fake modules (`internal/framework/runner/*_test.go`)
|
||||||
- CLI runner integration through injected fake module factories (`internal/cli/run_test.go`)
|
- CLI runner integration through injected fake module factories (`internal/cli/run_test.go`)
|
||||||
|
- validator models, cardinality enforcement, and deterministic validators (`internal/framework/validators/*_test.go`)
|
||||||
|
|
||||||
Not covered yet (because not implemented): validator runtime flow with approvals/rejections and real LLM integration.
|
Not covered yet (because not implemented): real LLM validator/runtime integration and production module behavior.
|
||||||
|
|
||||||
## Intended final architecture (not yet implemented)
|
## Intended final architecture (not yet implemented)
|
||||||
The intended end-state still matches the rewrite plan:
|
The intended end-state still matches the rewrite plan:
|
||||||
|
|||||||
@@ -47,12 +47,16 @@ Implemented:
|
|||||||
- Production runner orchestration over a mutable working transcript.
|
- Production runner orchestration over a mutable working transcript.
|
||||||
- Module-level report structures and run-level module summaries.
|
- Module-level report structures and run-level module summaries.
|
||||||
- CLI runner integration point via injectable module factory/registry (used by deterministic tests).
|
- CLI runner integration point via injectable module factory/registry (used by deterministic tests).
|
||||||
|
- Runtime validator models and deterministic validator implementations.
|
||||||
|
- Validator cardinality enforcement (missing/duplicate/unknown proposal index errors).
|
||||||
|
- Deterministic validator-chain execution in the production runner.
|
||||||
|
- Module reports including validator decisions and validator rejections.
|
||||||
- Broad deterministic and CLI/subprocess test coverage for implemented phases through `go test ./...`.
|
- Broad deterministic and CLI/subprocess test coverage for implemented phases through `go test ./...`.
|
||||||
|
|
||||||
Not yet implemented in runtime pipeline:
|
Not yet implemented in runtime pipeline:
|
||||||
- Real correction modules.
|
- Real correction modules.
|
||||||
- Validator-chain execution.
|
|
||||||
- Structured LLM client integration.
|
- Structured LLM client integration.
|
||||||
|
- LLM-backed validators.
|
||||||
- Prompt/response diagnostics for LLM calls.
|
- Prompt/response diagnostics for LLM calls.
|
||||||
- End-to-end transcript polishing behavior.
|
- End-to-end transcript polishing behavior.
|
||||||
|
|
||||||
@@ -161,21 +165,12 @@ Not implemented in Phase 7 (by design):
|
|||||||
Current runtime behavior note:
|
Current runtime behavior note:
|
||||||
- Default user-facing CLI behavior remains deterministic normalization/chunking output unless test-only module injection is used during tests.
|
- Default user-facing CLI behavior remains deterministic normalization/chunking output unless test-only module injection is used during tests.
|
||||||
|
|
||||||
## Remaining work plan
|
|
||||||
|
|
||||||
Next recommended phase: **Phase 8 (runtime validator framework and deterministic validators)**.
|
|
||||||
|
|
||||||
## Phase 8: Runtime validator framework and deterministic validators
|
## Phase 8: Runtime validator framework and deterministic validators
|
||||||
|
Completed.
|
||||||
|
|
||||||
### Purpose
|
Implemented:
|
||||||
|
- Runtime validator request/result models in `internal/framework/validators`.
|
||||||
Wire validator-chain execution into the runner using deterministic validators first. This phase establishes the safety model before any real LLM proposal generation is introduced.
|
- Deterministic validator reason codes for stable reporting.
|
||||||
|
|
||||||
### Scope
|
|
||||||
|
|
||||||
Implement:
|
|
||||||
- Runtime validator interfaces if the existing contracts need refinement.
|
|
||||||
- Validation request/result models.
|
|
||||||
- Validator cardinality enforcement:
|
- Validator cardinality enforcement:
|
||||||
- one decision per candidate proposal index
|
- one decision per candidate proposal index
|
||||||
- missing indexes are errors
|
- missing indexes are errors
|
||||||
@@ -183,33 +178,25 @@ Implement:
|
|||||||
- unknown indexes are errors
|
- unknown indexes are errors
|
||||||
- Deterministic validators:
|
- Deterministic validators:
|
||||||
- confidence threshold
|
- confidence threshold
|
||||||
- original-text presence
|
- original-text presence against working transcript
|
||||||
- non-empty correction
|
- non-empty correction
|
||||||
- identical text/no-effect rejection
|
- identical/no-effect rejection
|
||||||
- protected glossary term logic, if it can be implemented deterministically from current glossary schema
|
- conservative protected glossary-term guard
|
||||||
- Validator ordering.
|
- Ordered validator-chain execution in the production runner.
|
||||||
- Runner integration so candidate proposals pass through validators before application.
|
- Runner behavior where only validator-approved proposals proceed to proposal application.
|
||||||
- Module report fields for validator approvals/rejections.
|
- Module-level reporting of validator decisions and validator rejections, distinct from application-level skips.
|
||||||
|
- Deterministic fake-module tests covering approvals, rejections, validator order/filtering, and cardinality failure pipeline-stop behavior.
|
||||||
|
|
||||||
Do not implement:
|
Not implemented in Phase 8 (by design):
|
||||||
- LLM-backed validators.
|
- LLM-backed validators (Phase 10).
|
||||||
- Real modules.
|
- Structured LLM client implementation or scheduler behavior (Phase 9).
|
||||||
- Real LLM proposal generation.
|
- Real correction modules.
|
||||||
- Prompt/response diagnostics.
|
- Prompt/response diagnostics.
|
||||||
|
- End-to-end transcript polishing.
|
||||||
|
|
||||||
### Expected behavior at end of phase
|
## Remaining work plan
|
||||||
|
|
||||||
Fake modules can generate deterministic proposals, those proposals can be filtered by deterministic validators, and only approved proposals are applied.
|
Next recommended phase: **Phase 9 (structured LLM client and scheduler infrastructure)**.
|
||||||
|
|
||||||
### Definition of done
|
|
||||||
|
|
||||||
- Validator chains run in the production runner.
|
|
||||||
- Deterministic validators are implemented and tested.
|
|
||||||
- Validator cardinality enforcement is tested.
|
|
||||||
- Rejected proposals appear in module reports with stable reasons.
|
|
||||||
- Approved proposals are applied through existing proposal application semantics.
|
|
||||||
- No real LLM calls occur.
|
|
||||||
- `go test ./...` passes.
|
|
||||||
|
|
||||||
## Phase 9: Structured LLM client and scheduler infrastructure
|
## Phase 9: Structured LLM client and scheduler infrastructure
|
||||||
|
|
||||||
|
|||||||
@@ -361,7 +361,7 @@ func runProcess(args []string, stdout, stderr io.Writer) int {
|
|||||||
hasSkippedCorrections := false
|
hasSkippedCorrections := false
|
||||||
if runOutput != nil {
|
if runOutput != nil {
|
||||||
for _, mr := range runOutput.ModuleResults {
|
for _, mr := range runOutput.ModuleResults {
|
||||||
if len(mr.SkippedChanges) > 0 {
|
if len(mr.SkippedChanges) > 0 || len(mr.ValidatorRejected) > 0 {
|
||||||
hasSkippedCorrections = true
|
hasSkippedCorrections = true
|
||||||
break
|
break
|
||||||
}
|
}
|
||||||
@@ -395,7 +395,7 @@ func extractErrorPhase(err error) (phase string, message string) {
|
|||||||
|
|
||||||
func buildProcessReport(status string, inv processInvocation, runDir *diagnostics.RunDirectory, startedAt, completedAt time.Time, errorMessage string, errorPhase string, normalizationSummary *normalization.NormalizationSummary, chunkingSummary *chunking.Summary, runOutput *runner.RunOutput) reporting.ProcessReport {
|
func buildProcessReport(status string, inv processInvocation, runDir *diagnostics.RunDirectory, startedAt, completedAt time.Time, errorMessage string, errorPhase string, normalizationSummary *normalization.NormalizationSummary, chunkingSummary *chunking.Summary, runOutput *runner.RunOutput) reporting.ProcessReport {
|
||||||
report := reporting.ProcessReport{
|
report := reporting.ProcessReport{
|
||||||
Phase: "phase7-runner",
|
Phase: "phase8-validators",
|
||||||
Status: status,
|
Status: status,
|
||||||
Operation: "process",
|
Operation: "process",
|
||||||
TranscriptPath: inv.TranscriptPath,
|
TranscriptPath: inv.TranscriptPath,
|
||||||
@@ -466,6 +466,8 @@ func buildModuleReporting(runOutput *runner.RunOutput) (*reporting.ModulesSummar
|
|||||||
ReplacementPolicy: string(r.ReplacementPolicy),
|
ReplacementPolicy: string(r.ReplacementPolicy),
|
||||||
Status: r.Status,
|
Status: r.Status,
|
||||||
ProposalCount: r.ProposalCount,
|
ProposalCount: r.ProposalCount,
|
||||||
|
ValidatorDecisions: mapValidatorDecisions(r.ValidatorDecisions),
|
||||||
|
ValidatorRejected: mapValidatorRejected(r.ValidatorRejected),
|
||||||
AppliedChanges: r.AppliedChanges,
|
AppliedChanges: r.AppliedChanges,
|
||||||
SkippedChanges: r.SkippedChanges,
|
SkippedChanges: r.SkippedChanges,
|
||||||
ErrorMessage: r.ErrorMessage,
|
ErrorMessage: r.ErrorMessage,
|
||||||
@@ -473,7 +475,7 @@ func buildModuleReporting(runOutput *runner.RunOutput) (*reporting.ModulesSummar
|
|||||||
CompletedAt: &completedAt,
|
CompletedAt: &completedAt,
|
||||||
})
|
})
|
||||||
summary.TotalAppliedChanges += len(r.AppliedChanges)
|
summary.TotalAppliedChanges += len(r.AppliedChanges)
|
||||||
summary.TotalSkippedChanges += len(r.SkippedChanges)
|
summary.TotalSkippedChanges += len(r.SkippedChanges) + len(r.ValidatorRejected)
|
||||||
if r.Status == runner.ModuleStatusFailed && summary.FailedModuleInstance == "" {
|
if r.Status == runner.ModuleStatusFailed && summary.FailedModuleInstance == "" {
|
||||||
summary.FailedModuleInstance = r.ModuleInstance
|
summary.FailedModuleInstance = r.ModuleInstance
|
||||||
}
|
}
|
||||||
@@ -482,6 +484,44 @@ func buildModuleReporting(runOutput *runner.RunOutput) (*reporting.ModulesSummar
|
|||||||
return summary, moduleReports
|
return summary, moduleReports
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func mapValidatorDecisions(in []runner.ValidatorDecisionRecord) []reporting.ValidatorDecisionReport {
|
||||||
|
if len(in) == 0 {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
out := make([]reporting.ValidatorDecisionReport, len(in))
|
||||||
|
for i, d := range in {
|
||||||
|
out[i] = reporting.ValidatorDecisionReport{
|
||||||
|
ValidatorName: d.ValidatorName,
|
||||||
|
ProposalIndex: d.ProposalIndex,
|
||||||
|
Approved: d.Approved,
|
||||||
|
ReasonCode: d.ReasonCode,
|
||||||
|
Message: d.Message,
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return out
|
||||||
|
}
|
||||||
|
|
||||||
|
func mapValidatorRejected(in []runner.ValidatorRejectedChange) []reporting.ValidatorRejectedReport {
|
||||||
|
if len(in) == 0 {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
out := make([]reporting.ValidatorRejectedReport, len(in))
|
||||||
|
for i, d := range in {
|
||||||
|
out[i] = reporting.ValidatorRejectedReport{
|
||||||
|
ValidatorName: d.ValidatorName,
|
||||||
|
ProposalIndex: d.ProposalIndex,
|
||||||
|
ModuleKey: d.ModuleKey,
|
||||||
|
ModuleInstance: d.ModuleInstance,
|
||||||
|
TargetSegmentID: d.TargetSegmentID,
|
||||||
|
OriginalText: d.OriginalText,
|
||||||
|
CorrectedText: d.CorrectedText,
|
||||||
|
ReasonCode: d.ReasonCode,
|
||||||
|
Message: d.Message,
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return out
|
||||||
|
}
|
||||||
|
|
||||||
type processFlags struct {
|
type processFlags struct {
|
||||||
glossaryPath *string
|
glossaryPath *string
|
||||||
outputPath *string
|
outputPath *string
|
||||||
|
|||||||
@@ -15,6 +15,7 @@ import (
|
|||||||
"gitea.maximumdirect.net/eric/audita/internal/core/schema"
|
"gitea.maximumdirect.net/eric/audita/internal/core/schema"
|
||||||
"gitea.maximumdirect.net/eric/audita/internal/framework/contracts"
|
"gitea.maximumdirect.net/eric/audita/internal/framework/contracts"
|
||||||
"gitea.maximumdirect.net/eric/audita/internal/framework/proposals"
|
"gitea.maximumdirect.net/eric/audita/internal/framework/proposals"
|
||||||
|
"gitea.maximumdirect.net/eric/audita/internal/framework/validators"
|
||||||
)
|
)
|
||||||
|
|
||||||
func TestRunRootHelp(t *testing.T) {
|
func TestRunRootHelp(t *testing.T) {
|
||||||
@@ -613,8 +614,8 @@ func TestRunProcessReportJSONIncludesChunkingSummary(t *testing.T) {
|
|||||||
if report.Chunking.MaxSectionTokens == 0 {
|
if report.Chunking.MaxSectionTokens == 0 {
|
||||||
t.Errorf("expected max_section_tokens in report")
|
t.Errorf("expected max_section_tokens in report")
|
||||||
}
|
}
|
||||||
if report.Phase != "phase7-runner" {
|
if report.Phase != "phase8-validators" {
|
||||||
t.Errorf("expected phase 'phase7-runner', got %q", report.Phase)
|
t.Errorf("expected phase 'phase8-validators', got %q", report.Phase)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -633,12 +634,13 @@ func (f fakeModuleFactory) ModuleForSpec(spec contracts.ModuleRunSpec) (contract
|
|||||||
type fakeModule struct {
|
type fakeModule struct {
|
||||||
key string
|
key string
|
||||||
policy proposals.ReplacementPolicy
|
policy proposals.ReplacementPolicy
|
||||||
|
validators []contracts.Validator
|
||||||
proposeF func(req contracts.ProposalRequest) ([]proposals.CorrectionProposal, error)
|
proposeF func(req contracts.ProposalRequest) ([]proposals.CorrectionProposal, error)
|
||||||
}
|
}
|
||||||
|
|
||||||
func (m fakeModule) Key() string { return m.key }
|
func (m fakeModule) Key() string { return m.key }
|
||||||
func (m fakeModule) ReplacementPolicy() proposals.ReplacementPolicy { return m.policy }
|
func (m fakeModule) ReplacementPolicy() proposals.ReplacementPolicy { return m.policy }
|
||||||
func (m fakeModule) Validators() []contracts.Validator { return nil }
|
func (m fakeModule) Validators() []contracts.Validator { return m.validators }
|
||||||
func (m fakeModule) Propose(ctx context.Context, req contracts.ProposalRequest) ([]proposals.CorrectionProposal, error) {
|
func (m fakeModule) Propose(ctx context.Context, req contracts.ProposalRequest) ([]proposals.CorrectionProposal, error) {
|
||||||
if m.proposeF == nil {
|
if m.proposeF == nil {
|
||||||
return nil, nil
|
return nil, nil
|
||||||
@@ -646,14 +648,32 @@ func (m fakeModule) Propose(ctx context.Context, req contracts.ProposalRequest)
|
|||||||
return m.proposeF(req)
|
return m.proposeF(req)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
type fakeValidator struct {
|
||||||
|
name string
|
||||||
|
validateF func(req contracts.ValidationRequest) (validators.Result, error)
|
||||||
|
}
|
||||||
|
|
||||||
|
func (v fakeValidator) Name() string { return v.name }
|
||||||
|
func (v fakeValidator) Validate(ctx context.Context, req contracts.ValidationRequest) (validators.Result, error) {
|
||||||
|
_ = ctx
|
||||||
|
return v.validateF(req)
|
||||||
|
}
|
||||||
|
|
||||||
func TestRunProcessInjectedFactoryExecutesRunnerAndReportsModules(t *testing.T) {
|
func TestRunProcessInjectedFactoryExecutesRunnerAndReportsModules(t *testing.T) {
|
||||||
|
allow := fakeValidator{name: "allow", validateF: func(req contracts.ValidationRequest) (validators.Result, error) {
|
||||||
|
decisions := make([]validators.Decision, len(req.CandidateProposal))
|
||||||
|
for i, p := range req.CandidateProposal {
|
||||||
|
decisions[i] = validators.Decision{ProposalIndex: p.ProposalIndex, Approved: true, ReasonCode: validators.ReasonApproved, Message: "approved"}
|
||||||
|
}
|
||||||
|
return validators.Result{ValidatorName: "allow", Decisions: decisions}, nil
|
||||||
|
}}
|
||||||
processModuleFactory = fakeModuleFactory{modules: map[string]contracts.TranscriptModule{
|
processModuleFactory = fakeModuleFactory{modules: map[string]contracts.TranscriptModule{
|
||||||
"m1": fakeModule{key: "m1", policy: proposals.ReplacementPolicyRequireUnique, proposeF: func(req contracts.ProposalRequest) ([]proposals.CorrectionProposal, error) {
|
"m1": fakeModule{key: "m1", policy: proposals.ReplacementPolicyRequireUnique, validators: []contracts.Validator{allow}, proposeF: func(req contracts.ProposalRequest) ([]proposals.CorrectionProposal, error) {
|
||||||
return []proposals.CorrectionProposal{
|
return []proposals.CorrectionProposal{
|
||||||
{TargetSegmentID: 1, OriginalText: "Hello", CorrectedText: "Hi", Confidence: 1},
|
{TargetSegmentID: 1, OriginalText: "Hello", CorrectedText: "Hi", Confidence: 1},
|
||||||
}, nil
|
}, nil
|
||||||
}},
|
}},
|
||||||
"m2": fakeModule{key: "m2", policy: proposals.ReplacementPolicyRequireUnique, proposeF: func(req contracts.ProposalRequest) ([]proposals.CorrectionProposal, error) {
|
"m2": fakeModule{key: "m2", policy: proposals.ReplacementPolicyRequireUnique, validators: []contracts.Validator{allow}, proposeF: func(req contracts.ProposalRequest) ([]proposals.CorrectionProposal, error) {
|
||||||
if req.WorkingTranscript.Segments[0].Text != "Hi world" {
|
if req.WorkingTranscript.Segments[0].Text != "Hi world" {
|
||||||
t.Fatalf("expected module 2 to see module 1 changes, got %q", req.WorkingTranscript.Segments[0].Text)
|
t.Fatalf("expected module 2 to see module 1 changes, got %q", req.WorkingTranscript.Segments[0].Text)
|
||||||
}
|
}
|
||||||
@@ -711,11 +731,17 @@ func TestRunProcessInjectedFactoryExecutesRunnerAndReportsModules(t *testing.T)
|
|||||||
if len(report.ModuleResults[1].SkippedChanges) != 1 {
|
if len(report.ModuleResults[1].SkippedChanges) != 1 {
|
||||||
t.Fatalf("expected skipped change for module 2, got %+v", report.ModuleResults[1].SkippedChanges)
|
t.Fatalf("expected skipped change for module 2, got %+v", report.ModuleResults[1].SkippedChanges)
|
||||||
}
|
}
|
||||||
|
if len(report.ModuleResults[0].ValidatorDecisions) == 0 {
|
||||||
|
t.Fatalf("expected validator decisions in module report")
|
||||||
|
}
|
||||||
|
|
||||||
runDirReport := readProcessReport(t, filepath.Join(onlyRunDir(t, workDir), "report.json"))
|
runDirReport := readProcessReport(t, filepath.Join(onlyRunDir(t, workDir), "report.json"))
|
||||||
if len(runDirReport.ModuleResults) != 2 {
|
if len(runDirReport.ModuleResults) != 2 {
|
||||||
t.Fatalf("expected module results in run-dir report")
|
t.Fatalf("expected module results in run-dir report")
|
||||||
}
|
}
|
||||||
|
if len(runDirReport.ModuleResults[1].ValidatorDecisions) == 0 {
|
||||||
|
t.Fatalf("expected validator decisions in run-dir report")
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
func TestRunProcessInjectedFactorySkippedKeepsAutoRetention(t *testing.T) {
|
func TestRunProcessInjectedFactorySkippedKeepsAutoRetention(t *testing.T) {
|
||||||
|
|||||||
@@ -38,6 +38,8 @@ type ModuleReport struct {
|
|||||||
ReplacementPolicy string `json:"replacement_policy,omitempty"`
|
ReplacementPolicy string `json:"replacement_policy,omitempty"`
|
||||||
Status string `json:"status"`
|
Status string `json:"status"`
|
||||||
ProposalCount int `json:"proposal_count"`
|
ProposalCount int `json:"proposal_count"`
|
||||||
|
ValidatorDecisions []ValidatorDecisionReport `json:"validator_decisions,omitempty"`
|
||||||
|
ValidatorRejected []ValidatorRejectedReport `json:"validator_rejected,omitempty"`
|
||||||
AppliedChanges []proposals.AppliedChange `json:"applied_changes,omitempty"`
|
AppliedChanges []proposals.AppliedChange `json:"applied_changes,omitempty"`
|
||||||
SkippedChanges []proposals.SkippedChange `json:"skipped_changes,omitempty"`
|
SkippedChanges []proposals.SkippedChange `json:"skipped_changes,omitempty"`
|
||||||
ErrorMessage string `json:"error_message,omitempty"`
|
ErrorMessage string `json:"error_message,omitempty"`
|
||||||
@@ -45,6 +47,26 @@ type ModuleReport struct {
|
|||||||
CompletedAt *time.Time `json:"completed_at,omitempty"`
|
CompletedAt *time.Time `json:"completed_at,omitempty"`
|
||||||
}
|
}
|
||||||
|
|
||||||
|
type ValidatorDecisionReport struct {
|
||||||
|
ValidatorName string `json:"validator_name"`
|
||||||
|
ProposalIndex int `json:"proposal_index"`
|
||||||
|
Approved bool `json:"approved"`
|
||||||
|
ReasonCode string `json:"reason_code"`
|
||||||
|
Message string `json:"message"`
|
||||||
|
}
|
||||||
|
|
||||||
|
type ValidatorRejectedReport struct {
|
||||||
|
ValidatorName string `json:"validator_name"`
|
||||||
|
ProposalIndex int `json:"proposal_index"`
|
||||||
|
ModuleKey string `json:"module_key"`
|
||||||
|
ModuleInstance string `json:"module_instance"`
|
||||||
|
TargetSegmentID int `json:"target_segment_id"`
|
||||||
|
OriginalText string `json:"original_text"`
|
||||||
|
CorrectedText string `json:"corrected_text"`
|
||||||
|
ReasonCode string `json:"reason_code"`
|
||||||
|
Message string `json:"message"`
|
||||||
|
}
|
||||||
|
|
||||||
type ModulesSummary struct {
|
type ModulesSummary struct {
|
||||||
ModuleCount int `json:"module_count"`
|
ModuleCount int `json:"module_count"`
|
||||||
TotalAppliedChanges int `json:"total_applied_changes"`
|
TotalAppliedChanges int `json:"total_applied_changes"`
|
||||||
|
|||||||
@@ -11,7 +11,7 @@ import (
|
|||||||
func TestProcessReportModuleResultsJSONSuccessAndSkipped(t *testing.T) {
|
func TestProcessReportModuleResultsJSONSuccessAndSkipped(t *testing.T) {
|
||||||
now := time.Now().UTC()
|
now := time.Now().UTC()
|
||||||
report := ProcessReport{
|
report := ProcessReport{
|
||||||
Phase: "phase7-runner",
|
Phase: "phase8-validators",
|
||||||
Status: "success",
|
Status: "success",
|
||||||
ModuleResults: []ModuleReport{
|
ModuleResults: []ModuleReport{
|
||||||
{
|
{
|
||||||
@@ -20,6 +20,24 @@ func TestProcessReportModuleResultsJSONSuccessAndSkipped(t *testing.T) {
|
|||||||
ReplacementPolicy: "require_unique",
|
ReplacementPolicy: "require_unique",
|
||||||
Status: "success",
|
Status: "success",
|
||||||
ProposalCount: 2,
|
ProposalCount: 2,
|
||||||
|
ValidatorDecisions: []ValidatorDecisionReport{{
|
||||||
|
ValidatorName: "confidence_threshold",
|
||||||
|
ProposalIndex: 0,
|
||||||
|
Approved: true,
|
||||||
|
ReasonCode: "approved",
|
||||||
|
Message: "approved",
|
||||||
|
}},
|
||||||
|
ValidatorRejected: []ValidatorRejectedReport{{
|
||||||
|
ValidatorName: "confidence_threshold",
|
||||||
|
ProposalIndex: 1,
|
||||||
|
ModuleKey: "grammar",
|
||||||
|
ModuleInstance: "grammar_1",
|
||||||
|
TargetSegmentID: 1,
|
||||||
|
OriginalText: "bad",
|
||||||
|
CorrectedText: "worse",
|
||||||
|
ReasonCode: "low_confidence",
|
||||||
|
Message: "rejected",
|
||||||
|
}},
|
||||||
AppliedChanges: []proposals.AppliedChange{{
|
AppliedChanges: []proposals.AppliedChange{{
|
||||||
ProposalIndex: 0,
|
ProposalIndex: 0,
|
||||||
ModuleKey: "grammar",
|
ModuleKey: "grammar",
|
||||||
@@ -51,6 +69,9 @@ func TestProcessReportModuleResultsJSONSuccessAndSkipped(t *testing.T) {
|
|||||||
if len(parsed.ModuleResults) != 1 {
|
if len(parsed.ModuleResults) != 1 {
|
||||||
t.Fatalf("expected one module result, got %d", len(parsed.ModuleResults))
|
t.Fatalf("expected one module result, got %d", len(parsed.ModuleResults))
|
||||||
}
|
}
|
||||||
|
if len(parsed.ModuleResults[0].ValidatorDecisions) != 1 || len(parsed.ModuleResults[0].ValidatorRejected) != 1 {
|
||||||
|
t.Fatalf("expected validator result payloads in module report")
|
||||||
|
}
|
||||||
if parsed.ModulesSummary == nil || parsed.ModulesSummary.TotalSkippedChanges != 1 {
|
if parsed.ModulesSummary == nil || parsed.ModulesSummary.TotalSkippedChanges != 1 {
|
||||||
t.Fatalf("unexpected module summary: %+v", parsed.ModulesSummary)
|
t.Fatalf("unexpected module summary: %+v", parsed.ModulesSummary)
|
||||||
}
|
}
|
||||||
@@ -59,7 +80,7 @@ func TestProcessReportModuleResultsJSONSuccessAndSkipped(t *testing.T) {
|
|||||||
func TestProcessReportModuleResultsJSONFailedModule(t *testing.T) {
|
func TestProcessReportModuleResultsJSONFailedModule(t *testing.T) {
|
||||||
now := time.Now().UTC()
|
now := time.Now().UTC()
|
||||||
report := ProcessReport{
|
report := ProcessReport{
|
||||||
Phase: "phase7-runner",
|
Phase: "phase8-validators",
|
||||||
Status: "failed",
|
Status: "failed",
|
||||||
ModuleResults: []ModuleReport{
|
ModuleResults: []ModuleReport{
|
||||||
{
|
{
|
||||||
|
|||||||
@@ -9,6 +9,7 @@ import (
|
|||||||
"gitea.maximumdirect.net/eric/audita/internal/core/config"
|
"gitea.maximumdirect.net/eric/audita/internal/core/config"
|
||||||
"gitea.maximumdirect.net/eric/audita/internal/core/schema"
|
"gitea.maximumdirect.net/eric/audita/internal/core/schema"
|
||||||
"gitea.maximumdirect.net/eric/audita/internal/framework/proposals"
|
"gitea.maximumdirect.net/eric/audita/internal/framework/proposals"
|
||||||
|
"gitea.maximumdirect.net/eric/audita/internal/framework/validators"
|
||||||
)
|
)
|
||||||
|
|
||||||
// StructuredLLMClient provides provider-agnostic structured completion.
|
// StructuredLLMClient provides provider-agnostic structured completion.
|
||||||
@@ -27,7 +28,7 @@ type TranscriptModule interface {
|
|||||||
// Validator evaluates candidate proposals and returns one decision per proposal index.
|
// Validator evaluates candidate proposals and returns one decision per proposal index.
|
||||||
type Validator interface {
|
type Validator interface {
|
||||||
Name() string
|
Name() string
|
||||||
Validate(ctx context.Context, req ValidationRequest) ([]ValidationDecision, error)
|
Validate(ctx context.Context, req ValidationRequest) (validators.Result, error)
|
||||||
}
|
}
|
||||||
|
|
||||||
// StructuredCompletionRequest is a transport-neutral structured completion request.
|
// StructuredCompletionRequest is a transport-neutral structured completion request.
|
||||||
@@ -90,19 +91,7 @@ type ProposalRequest struct {
|
|||||||
}
|
}
|
||||||
|
|
||||||
// ValidationRequest is the input to validator execution.
|
// ValidationRequest is the input to validator execution.
|
||||||
type ValidationRequest struct {
|
type ValidationRequest = validators.Request
|
||||||
ExecutionContext
|
|
||||||
RunSpec ModuleRunSpec `json:"run_spec"`
|
|
||||||
CandidateProposals []proposals.EnrichedCorrectionProposal `json:"candidate_proposals"`
|
|
||||||
}
|
|
||||||
|
|
||||||
// ValidationDecision is one validator decision for one proposal index.
|
|
||||||
type ValidationDecision struct {
|
|
||||||
ProposalIndex int `json:"proposal_index"`
|
|
||||||
Approved bool `json:"approved"`
|
|
||||||
Confidence *float64 `json:"confidence,omitempty"`
|
|
||||||
Reason string `json:"reason,omitempty"`
|
|
||||||
}
|
|
||||||
|
|
||||||
// ResolveModuleRunSpecs deterministically resolves instance names from logical keys.
|
// ResolveModuleRunSpecs deterministically resolves instance names from logical keys.
|
||||||
// Repeated keys are suffixed with _<n> (1-based), while singleton keys keep their raw key.
|
// Repeated keys are suffixed with _<n> (1-based), while singleton keys keep their raw key.
|
||||||
|
|||||||
@@ -8,6 +8,7 @@ import (
|
|||||||
|
|
||||||
"gitea.maximumdirect.net/eric/audita/internal/core/chunking"
|
"gitea.maximumdirect.net/eric/audita/internal/core/chunking"
|
||||||
"gitea.maximumdirect.net/eric/audita/internal/framework/proposals"
|
"gitea.maximumdirect.net/eric/audita/internal/framework/proposals"
|
||||||
|
"gitea.maximumdirect.net/eric/audita/internal/framework/validators"
|
||||||
)
|
)
|
||||||
|
|
||||||
type fakeLLMClient struct{}
|
type fakeLLMClient struct{}
|
||||||
@@ -22,13 +23,13 @@ type fakeValidator struct{}
|
|||||||
|
|
||||||
func (f *fakeValidator) Name() string { return "fake-validator" }
|
func (f *fakeValidator) Name() string { return "fake-validator" }
|
||||||
|
|
||||||
func (f *fakeValidator) Validate(ctx context.Context, req ValidationRequest) ([]ValidationDecision, error) {
|
func (f *fakeValidator) Validate(ctx context.Context, req ValidationRequest) (validators.Result, error) {
|
||||||
_ = ctx
|
_ = ctx
|
||||||
decisions := make([]ValidationDecision, len(req.CandidateProposals))
|
decisions := make([]validators.Decision, len(req.CandidateProposal))
|
||||||
for i, proposal := range req.CandidateProposals {
|
for i, proposal := range req.CandidateProposal {
|
||||||
decisions[i] = ValidationDecision{ProposalIndex: proposal.ProposalIndex, Approved: true}
|
decisions[i] = validators.Decision{ProposalIndex: proposal.ProposalIndex, Approved: true, ReasonCode: validators.ReasonApproved, Message: "approved"}
|
||||||
}
|
}
|
||||||
return decisions, nil
|
return validators.Result{ValidatorName: f.Name(), Decisions: decisions}, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
type fakeModule struct{}
|
type fakeModule struct{}
|
||||||
|
|||||||
@@ -9,6 +9,7 @@ import (
|
|||||||
"gitea.maximumdirect.net/eric/audita/internal/core/schema"
|
"gitea.maximumdirect.net/eric/audita/internal/core/schema"
|
||||||
"gitea.maximumdirect.net/eric/audita/internal/framework/contracts"
|
"gitea.maximumdirect.net/eric/audita/internal/framework/contracts"
|
||||||
"gitea.maximumdirect.net/eric/audita/internal/framework/proposals"
|
"gitea.maximumdirect.net/eric/audita/internal/framework/proposals"
|
||||||
|
"gitea.maximumdirect.net/eric/audita/internal/framework/validators"
|
||||||
)
|
)
|
||||||
|
|
||||||
const (
|
const (
|
||||||
@@ -33,6 +34,8 @@ type ModuleResult struct {
|
|||||||
ReplacementPolicy proposals.ReplacementPolicy `json:"replacement_policy"`
|
ReplacementPolicy proposals.ReplacementPolicy `json:"replacement_policy"`
|
||||||
Status string `json:"status"`
|
Status string `json:"status"`
|
||||||
ProposalCount int `json:"proposal_count"`
|
ProposalCount int `json:"proposal_count"`
|
||||||
|
ValidatorDecisions []ValidatorDecisionRecord `json:"validator_decisions,omitempty"`
|
||||||
|
ValidatorRejected []ValidatorRejectedChange `json:"validator_rejected,omitempty"`
|
||||||
AppliedChanges []proposals.AppliedChange `json:"applied_changes,omitempty"`
|
AppliedChanges []proposals.AppliedChange `json:"applied_changes,omitempty"`
|
||||||
SkippedChanges []proposals.SkippedChange `json:"skipped_changes,omitempty"`
|
SkippedChanges []proposals.SkippedChange `json:"skipped_changes,omitempty"`
|
||||||
ErrorMessage string `json:"error_message,omitempty"`
|
ErrorMessage string `json:"error_message,omitempty"`
|
||||||
@@ -40,6 +43,26 @@ type ModuleResult struct {
|
|||||||
CompletedAt time.Time `json:"completed_at"`
|
CompletedAt time.Time `json:"completed_at"`
|
||||||
}
|
}
|
||||||
|
|
||||||
|
type ValidatorDecisionRecord struct {
|
||||||
|
ValidatorName string `json:"validator_name"`
|
||||||
|
ProposalIndex int `json:"proposal_index"`
|
||||||
|
Approved bool `json:"approved"`
|
||||||
|
ReasonCode string `json:"reason_code"`
|
||||||
|
Message string `json:"message"`
|
||||||
|
}
|
||||||
|
|
||||||
|
type ValidatorRejectedChange struct {
|
||||||
|
ValidatorName string `json:"validator_name"`
|
||||||
|
ProposalIndex int `json:"proposal_index"`
|
||||||
|
ModuleKey string `json:"module_key"`
|
||||||
|
ModuleInstance string `json:"module_instance"`
|
||||||
|
TargetSegmentID int `json:"target_segment_id"`
|
||||||
|
OriginalText string `json:"original_text"`
|
||||||
|
CorrectedText string `json:"corrected_text"`
|
||||||
|
ReasonCode string `json:"reason_code"`
|
||||||
|
Message string `json:"message"`
|
||||||
|
}
|
||||||
|
|
||||||
// RunInput is the deterministic runner input.
|
// RunInput is the deterministic runner input.
|
||||||
type RunInput struct {
|
type RunInput struct {
|
||||||
Config *config.Config
|
Config *config.Config
|
||||||
@@ -121,7 +144,86 @@ func (r *Runner) Run(ctx context.Context, input RunInput) (RunOutput, error) {
|
|||||||
})
|
})
|
||||||
}
|
}
|
||||||
|
|
||||||
applyResult := proposals.ApplyProposals(working, enriched, policy)
|
validatorDecisions := make([]ValidatorDecisionRecord, 0)
|
||||||
|
validatorRejected := make([]ValidatorRejectedChange, 0)
|
||||||
|
eligible := enriched
|
||||||
|
for _, validator := range module.Validators() {
|
||||||
|
vResult, vErr := validator.Validate(ctx, contracts.ValidationRequest{
|
||||||
|
WorkingTranscript: working,
|
||||||
|
CandidateProposal: eligible,
|
||||||
|
ModuleKey: spec.ModuleKey,
|
||||||
|
ModuleInstance: spec.InstanceName,
|
||||||
|
ReplacementPolicy: policy,
|
||||||
|
Glossary: input.Glossary,
|
||||||
|
Config: input.Config,
|
||||||
|
})
|
||||||
|
if vErr != nil {
|
||||||
|
failed := ModuleResult{
|
||||||
|
ModuleKey: spec.ModuleKey,
|
||||||
|
ModuleInstance: spec.InstanceName,
|
||||||
|
ReplacementPolicy: policy,
|
||||||
|
Status: ModuleStatusFailed,
|
||||||
|
ProposalCount: len(enriched),
|
||||||
|
ValidatorDecisions: validatorDecisions,
|
||||||
|
ValidatorRejected: validatorRejected,
|
||||||
|
ErrorMessage: vErr.Error(),
|
||||||
|
StartedAt: startedAt,
|
||||||
|
CompletedAt: time.Now().UTC(),
|
||||||
|
}
|
||||||
|
results = append(results, failed)
|
||||||
|
return RunOutput{FinalTranscript: working, ModuleResults: results}, fmt.Errorf("module %q validator %q failed: %w", spec.InstanceName, validator.Name(), vErr)
|
||||||
|
}
|
||||||
|
if err := validators.EnforceDecisionCardinality(eligible, vResult.Decisions); err != nil {
|
||||||
|
failed := ModuleResult{
|
||||||
|
ModuleKey: spec.ModuleKey,
|
||||||
|
ModuleInstance: spec.InstanceName,
|
||||||
|
ReplacementPolicy: policy,
|
||||||
|
Status: ModuleStatusFailed,
|
||||||
|
ProposalCount: len(enriched),
|
||||||
|
ValidatorDecisions: validatorDecisions,
|
||||||
|
ValidatorRejected: validatorRejected,
|
||||||
|
ErrorMessage: err.Error(),
|
||||||
|
StartedAt: startedAt,
|
||||||
|
CompletedAt: time.Now().UTC(),
|
||||||
|
}
|
||||||
|
results = append(results, failed)
|
||||||
|
return RunOutput{FinalTranscript: working, ModuleResults: results}, fmt.Errorf("module %q validator %q cardinality failed: %w", spec.InstanceName, validator.Name(), err)
|
||||||
|
}
|
||||||
|
|
||||||
|
nextEligible := make([]proposals.EnrichedCorrectionProposal, 0, len(eligible))
|
||||||
|
byIndex := make(map[int]proposals.EnrichedCorrectionProposal, len(eligible))
|
||||||
|
for _, p := range eligible {
|
||||||
|
byIndex[p.ProposalIndex] = p
|
||||||
|
}
|
||||||
|
for _, d := range vResult.Decisions {
|
||||||
|
validatorDecisions = append(validatorDecisions, ValidatorDecisionRecord{
|
||||||
|
ValidatorName: validator.Name(),
|
||||||
|
ProposalIndex: d.ProposalIndex,
|
||||||
|
Approved: d.Approved,
|
||||||
|
ReasonCode: d.ReasonCode,
|
||||||
|
Message: d.Message,
|
||||||
|
})
|
||||||
|
if d.Approved {
|
||||||
|
nextEligible = append(nextEligible, byIndex[d.ProposalIndex])
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
p := byIndex[d.ProposalIndex]
|
||||||
|
validatorRejected = append(validatorRejected, ValidatorRejectedChange{
|
||||||
|
ValidatorName: validator.Name(),
|
||||||
|
ProposalIndex: p.ProposalIndex,
|
||||||
|
ModuleKey: p.ModuleKey,
|
||||||
|
ModuleInstance: p.ModuleInstance,
|
||||||
|
TargetSegmentID: p.TargetSegmentID,
|
||||||
|
OriginalText: p.OriginalText,
|
||||||
|
CorrectedText: p.CorrectedText,
|
||||||
|
ReasonCode: d.ReasonCode,
|
||||||
|
Message: d.Message,
|
||||||
|
})
|
||||||
|
}
|
||||||
|
eligible = nextEligible
|
||||||
|
}
|
||||||
|
|
||||||
|
applyResult := proposals.ApplyProposals(working, eligible, policy)
|
||||||
working = applyResult.Transcript
|
working = applyResult.Transcript
|
||||||
|
|
||||||
results = append(results, ModuleResult{
|
results = append(results, ModuleResult{
|
||||||
@@ -130,6 +232,8 @@ func (r *Runner) Run(ctx context.Context, input RunInput) (RunOutput, error) {
|
|||||||
ReplacementPolicy: policy,
|
ReplacementPolicy: policy,
|
||||||
Status: ModuleStatusSuccess,
|
Status: ModuleStatusSuccess,
|
||||||
ProposalCount: len(enriched),
|
ProposalCount: len(enriched),
|
||||||
|
ValidatorDecisions: validatorDecisions,
|
||||||
|
ValidatorRejected: validatorRejected,
|
||||||
AppliedChanges: applyResult.Applied,
|
AppliedChanges: applyResult.Applied,
|
||||||
SkippedChanges: applyResult.Skipped,
|
SkippedChanges: applyResult.Skipped,
|
||||||
StartedAt: startedAt,
|
StartedAt: startedAt,
|
||||||
|
|||||||
@@ -9,6 +9,7 @@ import (
|
|||||||
"gitea.maximumdirect.net/eric/audita/internal/core/schema"
|
"gitea.maximumdirect.net/eric/audita/internal/core/schema"
|
||||||
"gitea.maximumdirect.net/eric/audita/internal/framework/contracts"
|
"gitea.maximumdirect.net/eric/audita/internal/framework/contracts"
|
||||||
"gitea.maximumdirect.net/eric/audita/internal/framework/proposals"
|
"gitea.maximumdirect.net/eric/audita/internal/framework/proposals"
|
||||||
|
"gitea.maximumdirect.net/eric/audita/internal/framework/validators"
|
||||||
)
|
)
|
||||||
|
|
||||||
type fakeFactory struct {
|
type fakeFactory struct {
|
||||||
@@ -26,12 +27,13 @@ func (f fakeFactory) ModuleForSpec(spec contracts.ModuleRunSpec) (contracts.Tran
|
|||||||
type fakeModule struct {
|
type fakeModule struct {
|
||||||
key string
|
key string
|
||||||
policy proposals.ReplacementPolicy
|
policy proposals.ReplacementPolicy
|
||||||
|
validators []contracts.Validator
|
||||||
proposeF func(req contracts.ProposalRequest) ([]proposals.CorrectionProposal, error)
|
proposeF func(req contracts.ProposalRequest) ([]proposals.CorrectionProposal, error)
|
||||||
}
|
}
|
||||||
|
|
||||||
func (m fakeModule) Key() string { return m.key }
|
func (m fakeModule) Key() string { return m.key }
|
||||||
func (m fakeModule) ReplacementPolicy() proposals.ReplacementPolicy { return m.policy }
|
func (m fakeModule) ReplacementPolicy() proposals.ReplacementPolicy { return m.policy }
|
||||||
func (m fakeModule) Validators() []contracts.Validator { return nil }
|
func (m fakeModule) Validators() []contracts.Validator { return m.validators }
|
||||||
func (m fakeModule) Propose(ctx context.Context, req contracts.ProposalRequest) ([]proposals.CorrectionProposal, error) {
|
func (m fakeModule) Propose(ctx context.Context, req contracts.ProposalRequest) ([]proposals.CorrectionProposal, error) {
|
||||||
if m.proposeF == nil {
|
if m.proposeF == nil {
|
||||||
return nil, nil
|
return nil, nil
|
||||||
@@ -39,6 +41,17 @@ func (m fakeModule) Propose(ctx context.Context, req contracts.ProposalRequest)
|
|||||||
return m.proposeF(req)
|
return m.proposeF(req)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
type fakeValidator struct {
|
||||||
|
name string
|
||||||
|
validateF func(req contracts.ValidationRequest) (validators.Result, error)
|
||||||
|
}
|
||||||
|
|
||||||
|
func (v fakeValidator) Name() string { return v.name }
|
||||||
|
func (v fakeValidator) Validate(ctx context.Context, req contracts.ValidationRequest) (validators.Result, error) {
|
||||||
|
_ = ctx
|
||||||
|
return v.validateF(req)
|
||||||
|
}
|
||||||
|
|
||||||
func TestRunnerOneModuleAppliesProposal(t *testing.T) {
|
func TestRunnerOneModuleAppliesProposal(t *testing.T) {
|
||||||
transcript := &schema.Transcript{Segments: []schema.Segment{{ID: 1, Speaker: "A", Start: 0, End: 1, Text: "teh cat"}}}
|
transcript := &schema.Transcript{Segments: []schema.Segment{{ID: 1, Speaker: "A", Start: 0, End: 1, Text: "teh cat"}}}
|
||||||
r := New(fakeFactory{modules: map[string]contracts.TranscriptModule{
|
r := New(fakeFactory{modules: map[string]contracts.TranscriptModule{
|
||||||
@@ -149,4 +162,148 @@ func TestRunnerRepeatedModuleInstanceNames(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func TestRunnerValidatorApprovedProposalApplied(t *testing.T) {
|
||||||
|
transcript := &schema.Transcript{Segments: []schema.Segment{{ID: 1, Text: "teh cat"}}}
|
||||||
|
allowAll := fakeValidator{name: "allow", validateF: func(req contracts.ValidationRequest) (validators.Result, error) {
|
||||||
|
decisions := make([]validators.Decision, len(req.CandidateProposal))
|
||||||
|
for i, p := range req.CandidateProposal {
|
||||||
|
decisions[i] = validators.Decision{ProposalIndex: p.ProposalIndex, Approved: true, ReasonCode: validators.ReasonApproved, Message: "approved"}
|
||||||
|
}
|
||||||
|
return validators.Result{ValidatorName: "allow", Decisions: decisions}, nil
|
||||||
|
}}
|
||||||
|
r := New(fakeFactory{modules: map[string]contracts.TranscriptModule{
|
||||||
|
"m": fakeModule{
|
||||||
|
key: "m",
|
||||||
|
policy: proposals.ReplacementPolicyRequireUnique,
|
||||||
|
validators: []contracts.Validator{allowAll},
|
||||||
|
proposeF: func(req contracts.ProposalRequest) ([]proposals.CorrectionProposal, error) {
|
||||||
|
return []proposals.CorrectionProposal{{TargetSegmentID: 1, OriginalText: "teh", CorrectedText: "the", Confidence: 1}}, nil
|
||||||
|
},
|
||||||
|
},
|
||||||
|
}})
|
||||||
|
|
||||||
|
out, err := r.Run(context.Background(), RunInput{Transcript: transcript, ModuleSpecs: []contracts.ModuleRunSpec{{ModuleKey: "m", InstanceName: "m"}}})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("Run error: %v", err)
|
||||||
|
}
|
||||||
|
if out.FinalTranscript.Segments[0].Text != "the cat" {
|
||||||
|
t.Fatalf("expected proposal applied, got %q", out.FinalTranscript.Segments[0].Text)
|
||||||
|
}
|
||||||
|
if len(out.ModuleResults[0].ValidatorDecisions) != 1 {
|
||||||
|
t.Fatalf("expected validator decisions recorded")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestRunnerValidatorRejectedProposalNotApplied(t *testing.T) {
|
||||||
|
transcript := &schema.Transcript{Segments: []schema.Segment{{ID: 1, Text: "teh cat"}}}
|
||||||
|
rejectAll := fakeValidator{name: "reject", validateF: func(req contracts.ValidationRequest) (validators.Result, error) {
|
||||||
|
decisions := make([]validators.Decision, len(req.CandidateProposal))
|
||||||
|
for i, p := range req.CandidateProposal {
|
||||||
|
decisions[i] = validators.Decision{ProposalIndex: p.ProposalIndex, Approved: false, ReasonCode: validators.ReasonNoEffect, Message: "rejected"}
|
||||||
|
}
|
||||||
|
return validators.Result{ValidatorName: "reject", Decisions: decisions}, nil
|
||||||
|
}}
|
||||||
|
r := New(fakeFactory{modules: map[string]contracts.TranscriptModule{
|
||||||
|
"m": fakeModule{key: "m", policy: proposals.ReplacementPolicyRequireUnique, validators: []contracts.Validator{rejectAll}, proposeF: func(req contracts.ProposalRequest) ([]proposals.CorrectionProposal, error) {
|
||||||
|
return []proposals.CorrectionProposal{{TargetSegmentID: 1, OriginalText: "teh", CorrectedText: "the", Confidence: 1}}, nil
|
||||||
|
}},
|
||||||
|
}})
|
||||||
|
out, err := r.Run(context.Background(), RunInput{Transcript: transcript, ModuleSpecs: []contracts.ModuleRunSpec{{ModuleKey: "m", InstanceName: "m"}}})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("Run error: %v", err)
|
||||||
|
}
|
||||||
|
if out.FinalTranscript.Segments[0].Text != "teh cat" {
|
||||||
|
t.Fatalf("expected rejected proposal not applied, got %q", out.FinalTranscript.Segments[0].Text)
|
||||||
|
}
|
||||||
|
if len(out.ModuleResults[0].ValidatorRejected) != 1 {
|
||||||
|
t.Fatalf("expected validator rejection recorded, got %+v", out.ModuleResults[0].ValidatorRejected)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestRunnerMultipleValidatorsRunInOrderAndFilterSurvivors(t *testing.T) {
|
||||||
|
transcript := &schema.Transcript{Segments: []schema.Segment{{ID: 1, Text: "one two"}}}
|
||||||
|
first := fakeValidator{name: "first", validateF: func(req contracts.ValidationRequest) (validators.Result, error) {
|
||||||
|
if len(req.CandidateProposal) != 2 {
|
||||||
|
t.Fatalf("expected first validator to see 2 candidates, got %d", len(req.CandidateProposal))
|
||||||
|
}
|
||||||
|
return validators.Result{
|
||||||
|
ValidatorName: "first",
|
||||||
|
Decisions: []validators.Decision{
|
||||||
|
{ProposalIndex: req.CandidateProposal[0].ProposalIndex, Approved: true, ReasonCode: validators.ReasonApproved, Message: "ok"},
|
||||||
|
{ProposalIndex: req.CandidateProposal[1].ProposalIndex, Approved: false, ReasonCode: validators.ReasonNoEffect, Message: "reject"},
|
||||||
|
},
|
||||||
|
}, nil
|
||||||
|
}}
|
||||||
|
second := fakeValidator{name: "second", validateF: func(req contracts.ValidationRequest) (validators.Result, error) {
|
||||||
|
if len(req.CandidateProposal) != 1 {
|
||||||
|
t.Fatalf("expected second validator to see only survivors, got %d", len(req.CandidateProposal))
|
||||||
|
}
|
||||||
|
return validators.Result{ValidatorName: "second", Decisions: []validators.Decision{{ProposalIndex: req.CandidateProposal[0].ProposalIndex, Approved: true, ReasonCode: validators.ReasonApproved, Message: "ok"}}}, nil
|
||||||
|
}}
|
||||||
|
r := New(fakeFactory{modules: map[string]contracts.TranscriptModule{
|
||||||
|
"m": fakeModule{key: "m", policy: proposals.ReplacementPolicyRequireUnique, validators: []contracts.Validator{first, second}, proposeF: func(req contracts.ProposalRequest) ([]proposals.CorrectionProposal, error) {
|
||||||
|
return []proposals.CorrectionProposal{
|
||||||
|
{TargetSegmentID: 1, OriginalText: "one", CorrectedText: "ONE", Confidence: 1},
|
||||||
|
{TargetSegmentID: 1, OriginalText: "two", CorrectedText: "TWO", Confidence: 1},
|
||||||
|
}, nil
|
||||||
|
}},
|
||||||
|
}})
|
||||||
|
|
||||||
|
out, err := r.Run(context.Background(), RunInput{Transcript: transcript, ModuleSpecs: []contracts.ModuleRunSpec{{ModuleKey: "m", InstanceName: "m"}}})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("Run error: %v", err)
|
||||||
|
}
|
||||||
|
if out.FinalTranscript.Segments[0].Text != "ONE two" {
|
||||||
|
t.Fatalf("expected only survivor applied, got %q", out.FinalTranscript.Segments[0].Text)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestRunnerValidatorCardinalityErrorStopsPipelineWithPartialProgress(t *testing.T) {
|
||||||
|
transcript := &schema.Transcript{Segments: []schema.Segment{{ID: 1, Text: "teh cat"}}}
|
||||||
|
good := fakeModule{key: "m1", policy: proposals.ReplacementPolicyRequireUnique, proposeF: func(req contracts.ProposalRequest) ([]proposals.CorrectionProposal, error) {
|
||||||
|
return []proposals.CorrectionProposal{{TargetSegmentID: 1, OriginalText: "teh", CorrectedText: "the", Confidence: 1}}, nil
|
||||||
|
}}
|
||||||
|
badValidator := fakeValidator{name: "bad", validateF: func(req contracts.ValidationRequest) (validators.Result, error) {
|
||||||
|
// Missing one decision triggers cardinality error.
|
||||||
|
return validators.Result{ValidatorName: "bad", Decisions: nil}, nil
|
||||||
|
}}
|
||||||
|
bad := fakeModule{key: "m2", policy: proposals.ReplacementPolicyRequireUnique, validators: []contracts.Validator{badValidator}, proposeF: func(req contracts.ProposalRequest) ([]proposals.CorrectionProposal, error) {
|
||||||
|
return []proposals.CorrectionProposal{{TargetSegmentID: 1, OriginalText: "cat", CorrectedText: "dog", Confidence: 1}}, nil
|
||||||
|
}}
|
||||||
|
r := New(fakeFactory{modules: map[string]contracts.TranscriptModule{"m1": good, "m2": bad}})
|
||||||
|
|
||||||
|
out, err := r.Run(context.Background(), RunInput{Transcript: transcript, ModuleSpecs: []contracts.ModuleRunSpec{{ModuleKey: "m1", InstanceName: "m1"}, {ModuleKey: "m2", InstanceName: "m2"}}})
|
||||||
|
if err == nil {
|
||||||
|
t.Fatal("expected cardinality error")
|
||||||
|
}
|
||||||
|
if out.FinalTranscript.Segments[0].Text != "the cat" {
|
||||||
|
t.Fatalf("expected partial progress preserved, got %q", out.FinalTranscript.Segments[0].Text)
|
||||||
|
}
|
||||||
|
if len(out.ModuleResults) != 2 || out.ModuleResults[1].Status != ModuleStatusFailed {
|
||||||
|
t.Fatalf("expected second module failed")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestRunnerApplicationSkipAfterValidatorApprovalReported(t *testing.T) {
|
||||||
|
transcript := &schema.Transcript{Segments: []schema.Segment{{ID: 1, Text: "word word"}}}
|
||||||
|
allow := fakeValidator{name: "allow", validateF: func(req contracts.ValidationRequest) (validators.Result, error) {
|
||||||
|
return validators.Result{ValidatorName: "allow", Decisions: []validators.Decision{{ProposalIndex: req.CandidateProposal[0].ProposalIndex, Approved: true, ReasonCode: validators.ReasonApproved, Message: "ok"}}}, nil
|
||||||
|
}}
|
||||||
|
r := New(fakeFactory{modules: map[string]contracts.TranscriptModule{
|
||||||
|
"m": fakeModule{key: "m", policy: proposals.ReplacementPolicyRequireUnique, validators: []contracts.Validator{allow}, proposeF: func(req contracts.ProposalRequest) ([]proposals.CorrectionProposal, error) {
|
||||||
|
return []proposals.CorrectionProposal{{TargetSegmentID: 1, OriginalText: "word", CorrectedText: "term", Confidence: 1}}, nil
|
||||||
|
}},
|
||||||
|
}})
|
||||||
|
out, err := r.Run(context.Background(), RunInput{Transcript: transcript, ModuleSpecs: []contracts.ModuleRunSpec{{ModuleKey: "m", InstanceName: "m"}}})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("Run error: %v", err)
|
||||||
|
}
|
||||||
|
if len(out.ModuleResults[0].SkippedChanges) != 1 {
|
||||||
|
t.Fatalf("expected application skip recorded")
|
||||||
|
}
|
||||||
|
if len(out.ModuleResults[0].ValidatorRejected) != 0 {
|
||||||
|
t.Fatalf("expected no validator rejection")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
func ptrConfig(c config.Config) *config.Config { return &c }
|
func ptrConfig(c config.Config) *config.Config { return &c }
|
||||||
|
|||||||
161
internal/framework/validators/deterministic.go
Normal file
161
internal/framework/validators/deterministic.go
Normal file
@@ -0,0 +1,161 @@
|
|||||||
|
package validators
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"fmt"
|
||||||
|
"strings"
|
||||||
|
|
||||||
|
"gitea.maximumdirect.net/eric/audita/internal/framework/proposals"
|
||||||
|
)
|
||||||
|
|
||||||
|
type ConfidenceThresholdValidator struct{}
|
||||||
|
|
||||||
|
func (v ConfidenceThresholdValidator) Name() string { return "confidence_threshold" }
|
||||||
|
|
||||||
|
func (v ConfidenceThresholdValidator) Validate(_ context.Context, req Request) (Result, error) {
|
||||||
|
threshold := confidenceThresholdForModule(req.ModuleKey, req.Config)
|
||||||
|
decisions := make([]Decision, 0, len(req.CandidateProposal))
|
||||||
|
for _, c := range req.CandidateProposal {
|
||||||
|
if c.Confidence < threshold {
|
||||||
|
decisions = append(decisions, rejection(c.ProposalIndex, ReasonLowConfidence, fmt.Sprintf("confidence %.4f below threshold %.4f", c.Confidence, threshold)))
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
decisions = append(decisions, approval(c.ProposalIndex))
|
||||||
|
}
|
||||||
|
if err := EnforceDecisionCardinality(req.CandidateProposal, decisions); err != nil {
|
||||||
|
return Result{}, err
|
||||||
|
}
|
||||||
|
return Result{ValidatorName: v.Name(), Decisions: decisions}, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
type OriginalTextPresenceValidator struct{}
|
||||||
|
|
||||||
|
func (v OriginalTextPresenceValidator) Name() string { return "original_text_presence" }
|
||||||
|
|
||||||
|
func (v OriginalTextPresenceValidator) Validate(_ context.Context, req Request) (Result, error) {
|
||||||
|
byID := make(map[int]string)
|
||||||
|
if req.WorkingTranscript != nil {
|
||||||
|
for _, seg := range req.WorkingTranscript.Segments {
|
||||||
|
byID[seg.ID] = seg.Text
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
decisions := make([]Decision, 0, len(req.CandidateProposal))
|
||||||
|
for _, c := range req.CandidateProposal {
|
||||||
|
text, ok := byID[c.TargetSegmentID]
|
||||||
|
if !ok {
|
||||||
|
decisions = append(decisions, rejection(c.ProposalIndex, ReasonMissingTargetSegment, "target segment was not found"))
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
if !strings.Contains(text, c.OriginalText) {
|
||||||
|
decisions = append(decisions, rejection(c.ProposalIndex, ReasonMissingOriginalText, "original_text was not found in current segment text"))
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
decisions = append(decisions, approval(c.ProposalIndex))
|
||||||
|
}
|
||||||
|
if err := EnforceDecisionCardinality(req.CandidateProposal, decisions); err != nil {
|
||||||
|
return Result{}, err
|
||||||
|
}
|
||||||
|
return Result{ValidatorName: v.Name(), Decisions: decisions}, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
type NonEmptyCorrectionValidator struct{}
|
||||||
|
|
||||||
|
func (v NonEmptyCorrectionValidator) Name() string { return "non_empty_correction" }
|
||||||
|
|
||||||
|
func (v NonEmptyCorrectionValidator) Validate(_ context.Context, req Request) (Result, error) {
|
||||||
|
decisions := make([]Decision, 0, len(req.CandidateProposal))
|
||||||
|
for _, c := range req.CandidateProposal {
|
||||||
|
if strings.TrimSpace(c.CorrectedText) == "" {
|
||||||
|
decisions = append(decisions, rejection(c.ProposalIndex, ReasonEmptyCorrectedText, "corrected_text must not be empty"))
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
decisions = append(decisions, approval(c.ProposalIndex))
|
||||||
|
}
|
||||||
|
if err := EnforceDecisionCardinality(req.CandidateProposal, decisions); err != nil {
|
||||||
|
return Result{}, err
|
||||||
|
}
|
||||||
|
return Result{ValidatorName: v.Name(), Decisions: decisions}, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
type NoEffectValidator struct{}
|
||||||
|
|
||||||
|
func (v NoEffectValidator) Name() string { return "no_effect" }
|
||||||
|
|
||||||
|
func (v NoEffectValidator) Validate(_ context.Context, req Request) (Result, error) {
|
||||||
|
decisions := make([]Decision, 0, len(req.CandidateProposal))
|
||||||
|
for _, c := range req.CandidateProposal {
|
||||||
|
if c.OriginalText == c.CorrectedText {
|
||||||
|
decisions = append(decisions, rejection(c.ProposalIndex, ReasonNoEffect, "original_text and corrected_text are identical"))
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
decisions = append(decisions, approval(c.ProposalIndex))
|
||||||
|
}
|
||||||
|
if err := EnforceDecisionCardinality(req.CandidateProposal, decisions); err != nil {
|
||||||
|
return Result{}, err
|
||||||
|
}
|
||||||
|
return Result{ValidatorName: v.Name(), Decisions: decisions}, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
type ProtectedGlossaryTermValidator struct{}
|
||||||
|
|
||||||
|
func (v ProtectedGlossaryTermValidator) Name() string { return "protected_glossary_terms" }
|
||||||
|
|
||||||
|
func (v ProtectedGlossaryTermValidator) Validate(_ context.Context, req Request) (Result, error) {
|
||||||
|
if req.ModuleKey == "glossary" {
|
||||||
|
decisions := make([]Decision, 0, len(req.CandidateProposal))
|
||||||
|
for _, c := range req.CandidateProposal {
|
||||||
|
decisions = append(decisions, approval(c.ProposalIndex))
|
||||||
|
}
|
||||||
|
return Result{ValidatorName: v.Name(), Decisions: decisions}, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
terms := glossaryTerms(req)
|
||||||
|
decisions := make([]Decision, 0, len(req.CandidateProposal))
|
||||||
|
for _, c := range req.CandidateProposal {
|
||||||
|
if altersProtectedTerm(c.CorrectionProposal, terms) {
|
||||||
|
decisions = append(decisions, rejection(c.ProposalIndex, ReasonProtectedGlossaryTerm, "proposal may alter protected glossary terminology"))
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
decisions = append(decisions, approval(c.ProposalIndex))
|
||||||
|
}
|
||||||
|
if err := EnforceDecisionCardinality(req.CandidateProposal, decisions); err != nil {
|
||||||
|
return Result{}, err
|
||||||
|
}
|
||||||
|
return Result{ValidatorName: v.Name(), Decisions: decisions}, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func glossaryTerms(req Request) []string {
|
||||||
|
if req.Glossary == nil {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
out := make([]string, 0)
|
||||||
|
for _, e := range req.Glossary.Entries {
|
||||||
|
if t := strings.TrimSpace(strings.ToLower(e.Name)); t != "" {
|
||||||
|
out = append(out, t)
|
||||||
|
}
|
||||||
|
for _, a := range e.Aliases {
|
||||||
|
if t := strings.TrimSpace(strings.ToLower(a)); t != "" {
|
||||||
|
out = append(out, t)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if t := strings.TrimSpace(strings.ToLower(e.Plural)); t != "" {
|
||||||
|
out = append(out, t)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return out
|
||||||
|
}
|
||||||
|
|
||||||
|
func altersProtectedTerm(p proposals.CorrectionProposal, terms []string) bool {
|
||||||
|
if len(terms) == 0 {
|
||||||
|
return false
|
||||||
|
}
|
||||||
|
orig := strings.ToLower(p.OriginalText)
|
||||||
|
corr := strings.ToLower(p.CorrectedText)
|
||||||
|
for _, t := range terms {
|
||||||
|
if strings.Contains(orig, t) && !strings.Contains(corr, t) {
|
||||||
|
return true
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return false
|
||||||
|
}
|
||||||
105
internal/framework/validators/models.go
Normal file
105
internal/framework/validators/models.go
Normal file
@@ -0,0 +1,105 @@
|
|||||||
|
package validators
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"fmt"
|
||||||
|
"strings"
|
||||||
|
|
||||||
|
"gitea.maximumdirect.net/eric/audita/internal/core/config"
|
||||||
|
"gitea.maximumdirect.net/eric/audita/internal/core/schema"
|
||||||
|
"gitea.maximumdirect.net/eric/audita/internal/framework/proposals"
|
||||||
|
)
|
||||||
|
|
||||||
|
const (
|
||||||
|
ReasonApproved = "approved"
|
||||||
|
ReasonLowConfidence = "low_confidence"
|
||||||
|
ReasonMissingOriginalText = "missing_original_text"
|
||||||
|
ReasonMissingTargetSegment = "missing_target_segment"
|
||||||
|
ReasonEmptyCorrectedText = "empty_corrected_text"
|
||||||
|
ReasonNoEffect = "no_effect"
|
||||||
|
ReasonProtectedGlossaryTerm = "protected_glossary_term"
|
||||||
|
)
|
||||||
|
|
||||||
|
// Request is the runtime input shared by deterministic validators.
|
||||||
|
type Request struct {
|
||||||
|
WorkingTranscript *schema.Transcript `json:"-"`
|
||||||
|
CandidateProposal []proposals.EnrichedCorrectionProposal `json:"candidate_proposals"`
|
||||||
|
ModuleKey string `json:"module_key"`
|
||||||
|
ModuleInstance string `json:"module_instance"`
|
||||||
|
ReplacementPolicy proposals.ReplacementPolicy `json:"replacement_policy"`
|
||||||
|
Glossary *schema.Glossary `json:"-"`
|
||||||
|
Config *config.Config `json:"-"`
|
||||||
|
}
|
||||||
|
|
||||||
|
// Decision is one validator decision for one proposal index.
|
||||||
|
type Decision struct {
|
||||||
|
ProposalIndex int `json:"proposal_index"`
|
||||||
|
Approved bool `json:"approved"`
|
||||||
|
ReasonCode string `json:"reason_code"`
|
||||||
|
Message string `json:"message"`
|
||||||
|
}
|
||||||
|
|
||||||
|
// Result is one validator output containing exactly one decision per proposal index.
|
||||||
|
type Result struct {
|
||||||
|
ValidatorName string `json:"validator_name"`
|
||||||
|
Decisions []Decision `json:"decisions"`
|
||||||
|
}
|
||||||
|
|
||||||
|
// Validator is the deterministic runtime validator interface.
|
||||||
|
type Validator interface {
|
||||||
|
Name() string
|
||||||
|
Validate(ctx context.Context, req Request) (Result, error)
|
||||||
|
}
|
||||||
|
|
||||||
|
// EnforceDecisionCardinality verifies every candidate proposal index receives exactly one decision.
|
||||||
|
func EnforceDecisionCardinality(candidate []proposals.EnrichedCorrectionProposal, decisions []Decision) error {
|
||||||
|
expected := make(map[int]struct{}, len(candidate))
|
||||||
|
for _, c := range candidate {
|
||||||
|
expected[c.ProposalIndex] = struct{}{}
|
||||||
|
}
|
||||||
|
|
||||||
|
seen := make(map[int]int, len(decisions))
|
||||||
|
for _, d := range decisions {
|
||||||
|
if _, ok := expected[d.ProposalIndex]; !ok {
|
||||||
|
return fmt.Errorf("unknown decision proposal index %d", d.ProposalIndex)
|
||||||
|
}
|
||||||
|
seen[d.ProposalIndex]++
|
||||||
|
if seen[d.ProposalIndex] > 1 {
|
||||||
|
return fmt.Errorf("duplicate decision proposal index %d", d.ProposalIndex)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
for idx := range expected {
|
||||||
|
if seen[idx] == 0 {
|
||||||
|
return fmt.Errorf("missing decision proposal index %d", idx)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func approval(index int) Decision {
|
||||||
|
return Decision{ProposalIndex: index, Approved: true, ReasonCode: ReasonApproved, Message: "approved"}
|
||||||
|
}
|
||||||
|
|
||||||
|
func rejection(index int, reasonCode string, msg string) Decision {
|
||||||
|
return Decision{ProposalIndex: index, Approved: false, ReasonCode: reasonCode, Message: strings.TrimSpace(msg)}
|
||||||
|
}
|
||||||
|
|
||||||
|
func confidenceThresholdForModule(moduleKey string, cfg *config.Config) float64 {
|
||||||
|
if cfg == nil {
|
||||||
|
return 0.0
|
||||||
|
}
|
||||||
|
switch moduleKey {
|
||||||
|
case "glossary":
|
||||||
|
return cfg.Thresholds.Glossary
|
||||||
|
case "grammar":
|
||||||
|
return cfg.Thresholds.Grammar
|
||||||
|
case "homophones":
|
||||||
|
return cfg.Thresholds.Homophones
|
||||||
|
case "spoken_word":
|
||||||
|
return cfg.Thresholds.SpokenWord
|
||||||
|
default:
|
||||||
|
return 0.0
|
||||||
|
}
|
||||||
|
}
|
||||||
181
internal/framework/validators/validators_test.go
Normal file
181
internal/framework/validators/validators_test.go
Normal file
@@ -0,0 +1,181 @@
|
|||||||
|
package validators
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"strings"
|
||||||
|
"testing"
|
||||||
|
|
||||||
|
"gitea.maximumdirect.net/eric/audita/internal/core/config"
|
||||||
|
"gitea.maximumdirect.net/eric/audita/internal/core/schema"
|
||||||
|
"gitea.maximumdirect.net/eric/audita/internal/framework/proposals"
|
||||||
|
)
|
||||||
|
|
||||||
|
func mkCandidate(index int, segID int, orig, corr string, conf float64) proposals.EnrichedCorrectionProposal {
|
||||||
|
return proposals.EnrichedCorrectionProposal{
|
||||||
|
CorrectionProposal: proposals.CorrectionProposal{TargetSegmentID: segID, OriginalText: orig, CorrectedText: corr, Confidence: conf},
|
||||||
|
ProposalMetadata: proposals.ProposalMetadata{ProposalIndex: index, ModuleKey: "grammar", ModuleInstance: "grammar_1"},
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestEnforceDecisionCardinalitySuccess(t *testing.T) {
|
||||||
|
candidates := []proposals.EnrichedCorrectionProposal{mkCandidate(0, 1, "teh", "the", 0.9), mkCandidate(1, 1, "recieve", "receive", 0.9)}
|
||||||
|
decisions := []Decision{{ProposalIndex: 0, Approved: true}, {ProposalIndex: 1, Approved: false}}
|
||||||
|
if err := EnforceDecisionCardinality(candidates, decisions); err != nil {
|
||||||
|
t.Fatalf("unexpected error: %v", err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestEnforceDecisionCardinalityMissingDecision(t *testing.T) {
|
||||||
|
candidates := []proposals.EnrichedCorrectionProposal{mkCandidate(0, 1, "teh", "the", 0.9), mkCandidate(1, 1, "recieve", "receive", 0.9)}
|
||||||
|
err := EnforceDecisionCardinality(candidates, []Decision{{ProposalIndex: 0, Approved: true}})
|
||||||
|
if err == nil || !strings.Contains(err.Error(), "missing decision") {
|
||||||
|
t.Fatalf("expected missing decision error, got %v", err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestEnforceDecisionCardinalityDuplicateDecision(t *testing.T) {
|
||||||
|
candidates := []proposals.EnrichedCorrectionProposal{mkCandidate(0, 1, "teh", "the", 0.9)}
|
||||||
|
err := EnforceDecisionCardinality(candidates, []Decision{{ProposalIndex: 0, Approved: true}, {ProposalIndex: 0, Approved: false}})
|
||||||
|
if err == nil || !strings.Contains(err.Error(), "duplicate decision") {
|
||||||
|
t.Fatalf("expected duplicate decision error, got %v", err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestEnforceDecisionCardinalityUnknownDecision(t *testing.T) {
|
||||||
|
candidates := []proposals.EnrichedCorrectionProposal{mkCandidate(0, 1, "teh", "the", 0.9)}
|
||||||
|
err := EnforceDecisionCardinality(candidates, []Decision{{ProposalIndex: 99, Approved: true}})
|
||||||
|
if err == nil || !strings.Contains(err.Error(), "unknown decision") {
|
||||||
|
t.Fatalf("expected unknown decision error, got %v", err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestConfidenceThresholdValidator(t *testing.T) {
|
||||||
|
cfg := config.Default()
|
||||||
|
cfg.Thresholds.Grammar = 0.8
|
||||||
|
req := Request{ModuleKey: "grammar", Config: &cfg, CandidateProposal: []proposals.EnrichedCorrectionProposal{
|
||||||
|
mkCandidate(0, 1, "teh", "the", 0.9),
|
||||||
|
mkCandidate(1, 1, "recieve", "receive", 0.7),
|
||||||
|
}}
|
||||||
|
res, err := (ConfidenceThresholdValidator{}).Validate(context.Background(), req)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("Validate error: %v", err)
|
||||||
|
}
|
||||||
|
if !res.Decisions[0].Approved || res.Decisions[0].ReasonCode != ReasonApproved {
|
||||||
|
t.Fatalf("expected first decision approved, got %+v", res.Decisions[0])
|
||||||
|
}
|
||||||
|
if res.Decisions[1].Approved || res.Decisions[1].ReasonCode != ReasonLowConfidence {
|
||||||
|
t.Fatalf("expected second decision low confidence reject, got %+v", res.Decisions[1])
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestOriginalTextPresenceValidator(t *testing.T) {
|
||||||
|
req := Request{WorkingTranscript: &schema.Transcript{Segments: []schema.Segment{{ID: 1, Text: "hello world"}}}, CandidateProposal: []proposals.EnrichedCorrectionProposal{
|
||||||
|
mkCandidate(0, 1, "hello", "hi", 0.9),
|
||||||
|
mkCandidate(1, 1, "missing", "x", 0.9),
|
||||||
|
mkCandidate(2, 5, "hello", "hi", 0.9),
|
||||||
|
}}
|
||||||
|
res, err := (OriginalTextPresenceValidator{}).Validate(context.Background(), req)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("Validate error: %v", err)
|
||||||
|
}
|
||||||
|
if !res.Decisions[0].Approved {
|
||||||
|
t.Fatalf("expected proposal 0 approved")
|
||||||
|
}
|
||||||
|
if res.Decisions[1].ReasonCode != ReasonMissingOriginalText {
|
||||||
|
t.Fatalf("expected missing_original_text, got %+v", res.Decisions[1])
|
||||||
|
}
|
||||||
|
if res.Decisions[2].ReasonCode != ReasonMissingTargetSegment {
|
||||||
|
t.Fatalf("expected missing_target_segment, got %+v", res.Decisions[2])
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestNonEmptyCorrectionValidator(t *testing.T) {
|
||||||
|
req := Request{CandidateProposal: []proposals.EnrichedCorrectionProposal{
|
||||||
|
mkCandidate(0, 1, "hello", "hi", 0.9),
|
||||||
|
mkCandidate(1, 1, "hello", " ", 0.9),
|
||||||
|
}}
|
||||||
|
res, err := (NonEmptyCorrectionValidator{}).Validate(context.Background(), req)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("Validate error: %v", err)
|
||||||
|
}
|
||||||
|
if !res.Decisions[0].Approved {
|
||||||
|
t.Fatalf("expected proposal 0 approved")
|
||||||
|
}
|
||||||
|
if res.Decisions[1].ReasonCode != ReasonEmptyCorrectedText {
|
||||||
|
t.Fatalf("expected empty_corrected_text, got %+v", res.Decisions[1])
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestNoEffectValidator(t *testing.T) {
|
||||||
|
req := Request{CandidateProposal: []proposals.EnrichedCorrectionProposal{
|
||||||
|
mkCandidate(0, 1, "hello", "hello", 0.9),
|
||||||
|
mkCandidate(1, 1, "hello", "hi", 0.9),
|
||||||
|
}}
|
||||||
|
res, err := (NoEffectValidator{}).Validate(context.Background(), req)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("Validate error: %v", err)
|
||||||
|
}
|
||||||
|
if res.Decisions[0].ReasonCode != ReasonNoEffect || res.Decisions[0].Approved {
|
||||||
|
t.Fatalf("expected no_effect rejection, got %+v", res.Decisions[0])
|
||||||
|
}
|
||||||
|
if !res.Decisions[1].Approved {
|
||||||
|
t.Fatalf("expected proposal 1 approved")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestProtectedGlossaryTermValidator(t *testing.T) {
|
||||||
|
glossary := &schema.Glossary{Entries: []schema.GlossaryEntry{{Name: "OpenAI", Aliases: []string{"Open AI"}, Plural: "OpenAIs", Category: "brand", Summary: "brand"}}}
|
||||||
|
req := Request{
|
||||||
|
Glossary: glossary,
|
||||||
|
ModuleKey: "grammar",
|
||||||
|
CandidateProposal: []proposals.EnrichedCorrectionProposal{
|
||||||
|
mkCandidate(0, 1, "OpenAI", "Open A I", 0.9),
|
||||||
|
mkCandidate(1, 1, "teh", "the", 0.9),
|
||||||
|
},
|
||||||
|
}
|
||||||
|
res, err := (ProtectedGlossaryTermValidator{}).Validate(context.Background(), req)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("Validate error: %v", err)
|
||||||
|
}
|
||||||
|
if res.Decisions[0].ReasonCode != ReasonProtectedGlossaryTerm || res.Decisions[0].Approved {
|
||||||
|
t.Fatalf("expected protected glossary rejection, got %+v", res.Decisions[0])
|
||||||
|
}
|
||||||
|
if !res.Decisions[1].Approved {
|
||||||
|
t.Fatalf("expected non-glossary proposal approved")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestProtectedGlossaryTermValidatorAllowsGlossaryModule(t *testing.T) {
|
||||||
|
glossary := &schema.Glossary{Entries: []schema.GlossaryEntry{{Name: "OpenAI", Category: "brand", Summary: "brand"}}}
|
||||||
|
req := Request{
|
||||||
|
Glossary: glossary,
|
||||||
|
ModuleKey: "glossary",
|
||||||
|
CandidateProposal: []proposals.EnrichedCorrectionProposal{
|
||||||
|
mkCandidate(0, 1, "OpenAI", "Open A I", 0.9),
|
||||||
|
},
|
||||||
|
}
|
||||||
|
res, err := (ProtectedGlossaryTermValidator{}).Validate(context.Background(), req)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("Validate error: %v", err)
|
||||||
|
}
|
||||||
|
if !res.Decisions[0].Approved {
|
||||||
|
t.Fatalf("expected glossary module approval, got %+v", res.Decisions[0])
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestStableReasonCodes(t *testing.T) {
|
||||||
|
codes := []string{
|
||||||
|
ReasonApproved,
|
||||||
|
ReasonLowConfidence,
|
||||||
|
ReasonMissingOriginalText,
|
||||||
|
ReasonMissingTargetSegment,
|
||||||
|
ReasonEmptyCorrectedText,
|
||||||
|
ReasonNoEffect,
|
||||||
|
ReasonProtectedGlossaryTerm,
|
||||||
|
}
|
||||||
|
for _, code := range codes {
|
||||||
|
if strings.TrimSpace(code) == "" {
|
||||||
|
t.Fatalf("reason code must not be empty")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
Reference in New Issue
Block a user