Complete Phase 8 deterministic validators

This commit is contained in:
2026-05-12 00:26:27 +00:00
parent 28fe899aa1
commit aeb31f1c0d
13 changed files with 940 additions and 115 deletions

View File

@@ -19,14 +19,21 @@ Implemented today:
- Framework foundation packages for contracts and proposal application. - Framework foundation packages for contracts and proposal application.
- Production runner orchestration package with deterministic sequential module execution. - Production runner orchestration package with deterministic sequential module execution.
- Module-level report structures with applied/skipped change records. - Module-level report structures with applied/skipped change records.
- Runtime validator models and deterministic validators.
- Deterministic validator-chain execution in the runner with cardinality enforcement.
- Module-level validator decision/rejection reporting.
Not implemented in CLI runtime path today: Not implemented in CLI runtime path today:
- Real module execution pipeline (`glossary`, `homophones`, `spoken_word`, `grammar`). - Real module execution pipeline (`glossary`, `homophones`, `spoken_word`, `grammar`).
- Structured LLM proposal generation. - Structured LLM proposal generation.
- Validator chain execution. - LLM-backed validators.
- Production LLM scheduler behavior. - Production LLM scheduler behavior.
- End-to-end transcript polishing with real module behavior. - End-to-end transcript polishing with real module behavior.
Phase sequencing note:
- structured LLM client and scheduler infrastructure remain Phase 9 work;
- LLM-backed validators remain Phase 10 work.
## Actual Go package layout ## Actual Go package layout
```text ```text
@@ -77,6 +84,10 @@ internal/framework/proposals/
internal/framework/runner/ internal/framework/runner/
runner.go runner.go
internal/framework/validators/
models.go
deterministic.go
``` ```
## Current CLI behavior ## Current CLI behavior
@@ -100,7 +111,7 @@ Current runtime flow (`internal/cli/run.go`):
11. Write chunking summary artifact. 11. Write chunking summary artifact.
12. Optionally execute runner modules sequentially when an injected module registry/factory is available (used by deterministic tests today). 12. Optionally execute runner modules sequentially when an injected module registry/factory is available (used by deterministic tests today).
13. Output working transcript to `--output` file or stdout. 13. Output working transcript to `--output` file or stdout.
14. Build process report (`phase` currently set to `phase7-runner`). 14. Build process report (`phase` currently set to `phase8-validators`).
15. Optionally write `--report-json`; always write run-dir `report.json`. 15. Optionally write `--report-json`; always write run-dir `report.json`.
16. Apply work-dir retention. 16. Apply work-dir retention.
@@ -194,6 +205,24 @@ Current behavior details:
These primitives are wired into the production runner and report model. Real module implementations are still pending. These primitives are wired into the production runner and report model. Real module implementations are still pending.
## Implemented validator runtime infrastructure
`internal/framework/validators` provides deterministic validator infrastructure:
- runtime validation request/result models;
- stable validator reason codes;
- cardinality enforcement for validator decisions:
- missing proposal indexes fail
- duplicate proposal indexes fail
- unknown proposal indexes fail
- deterministic validators:
- confidence threshold by module key/config threshold
- original-text presence against current working transcript
- non-empty corrected text
- identical/no-effect rejection
- conservative protected glossary-term guard for non-glossary modules
`internal/framework/runner` executes validator chains in order for each module and applies only validator-approved proposals.
Validator rejections are reported distinctly from proposal-application skips.
## Reports and diagnostics (implemented) ## Reports and diagnostics (implemented)
Current per-run artifacts include: Current per-run artifacts include:
- `source-transcript.json` - `source-transcript.json`
@@ -222,6 +251,7 @@ Current process reports include diagnostics metadata references for:
Current process reports also include: Current process reports also include:
- module-level results (when runner modules execute), including applied/skipped proposal changes; - module-level results (when runner modules execute), including applied/skipped proposal changes;
- run-level module summary totals and failed module instance metadata. - run-level module summary totals and failed module instance metadata.
- module-level validator decisions and validator rejections.
Retention modes implemented in `ApplyRetention`: Retention modes implemented in `ApplyRetention`:
- `always`: keep all run directories. - `always`: keep all run directories.
@@ -247,8 +277,9 @@ Implemented tests currently cover:
- contracts/foundation composition tests (`internal/framework/contracts/*_test.go`) - contracts/foundation composition tests (`internal/framework/contracts/*_test.go`)
- runner sequencing and failure behavior with deterministic fake modules (`internal/framework/runner/*_test.go`) - runner sequencing and failure behavior with deterministic fake modules (`internal/framework/runner/*_test.go`)
- CLI runner integration through injected fake module factories (`internal/cli/run_test.go`) - CLI runner integration through injected fake module factories (`internal/cli/run_test.go`)
- validator models, cardinality enforcement, and deterministic validators (`internal/framework/validators/*_test.go`)
Not covered yet (because not implemented): validator runtime flow with approvals/rejections and real LLM integration. Not covered yet (because not implemented): real LLM validator/runtime integration and production module behavior.
## Intended final architecture (not yet implemented) ## Intended final architecture (not yet implemented)
The intended end-state still matches the rewrite plan: The intended end-state still matches the rewrite plan:

View File

@@ -47,12 +47,16 @@ Implemented:
- Production runner orchestration over a mutable working transcript. - Production runner orchestration over a mutable working transcript.
- Module-level report structures and run-level module summaries. - Module-level report structures and run-level module summaries.
- CLI runner integration point via injectable module factory/registry (used by deterministic tests). - CLI runner integration point via injectable module factory/registry (used by deterministic tests).
- Runtime validator models and deterministic validator implementations.
- Validator cardinality enforcement (missing/duplicate/unknown proposal index errors).
- Deterministic validator-chain execution in the production runner.
- Module reports including validator decisions and validator rejections.
- Broad deterministic and CLI/subprocess test coverage for implemented phases through `go test ./...`. - Broad deterministic and CLI/subprocess test coverage for implemented phases through `go test ./...`.
Not yet implemented in runtime pipeline: Not yet implemented in runtime pipeline:
- Real correction modules. - Real correction modules.
- Validator-chain execution.
- Structured LLM client integration. - Structured LLM client integration.
- LLM-backed validators.
- Prompt/response diagnostics for LLM calls. - Prompt/response diagnostics for LLM calls.
- End-to-end transcript polishing behavior. - End-to-end transcript polishing behavior.
@@ -161,21 +165,12 @@ Not implemented in Phase 7 (by design):
Current runtime behavior note: Current runtime behavior note:
- Default user-facing CLI behavior remains deterministic normalization/chunking output unless test-only module injection is used during tests. - Default user-facing CLI behavior remains deterministic normalization/chunking output unless test-only module injection is used during tests.
## Remaining work plan
Next recommended phase: **Phase 8 (runtime validator framework and deterministic validators)**.
## Phase 8: Runtime validator framework and deterministic validators ## Phase 8: Runtime validator framework and deterministic validators
Completed.
### Purpose Implemented:
- Runtime validator request/result models in `internal/framework/validators`.
Wire validator-chain execution into the runner using deterministic validators first. This phase establishes the safety model before any real LLM proposal generation is introduced. - Deterministic validator reason codes for stable reporting.
### Scope
Implement:
- Runtime validator interfaces if the existing contracts need refinement.
- Validation request/result models.
- Validator cardinality enforcement: - Validator cardinality enforcement:
- one decision per candidate proposal index - one decision per candidate proposal index
- missing indexes are errors - missing indexes are errors
@@ -183,33 +178,25 @@ Implement:
- unknown indexes are errors - unknown indexes are errors
- Deterministic validators: - Deterministic validators:
- confidence threshold - confidence threshold
- original-text presence - original-text presence against working transcript
- non-empty correction - non-empty correction
- identical text/no-effect rejection - identical/no-effect rejection
- protected glossary term logic, if it can be implemented deterministically from current glossary schema - conservative protected glossary-term guard
- Validator ordering. - Ordered validator-chain execution in the production runner.
- Runner integration so candidate proposals pass through validators before application. - Runner behavior where only validator-approved proposals proceed to proposal application.
- Module report fields for validator approvals/rejections. - Module-level reporting of validator decisions and validator rejections, distinct from application-level skips.
- Deterministic fake-module tests covering approvals, rejections, validator order/filtering, and cardinality failure pipeline-stop behavior.
Do not implement: Not implemented in Phase 8 (by design):
- LLM-backed validators. - LLM-backed validators (Phase 10).
- Real modules. - Structured LLM client implementation or scheduler behavior (Phase 9).
- Real LLM proposal generation. - Real correction modules.
- Prompt/response diagnostics. - Prompt/response diagnostics.
- End-to-end transcript polishing.
### Expected behavior at end of phase ## Remaining work plan
Fake modules can generate deterministic proposals, those proposals can be filtered by deterministic validators, and only approved proposals are applied. Next recommended phase: **Phase 9 (structured LLM client and scheduler infrastructure)**.
### Definition of done
- Validator chains run in the production runner.
- Deterministic validators are implemented and tested.
- Validator cardinality enforcement is tested.
- Rejected proposals appear in module reports with stable reasons.
- Approved proposals are applied through existing proposal application semantics.
- No real LLM calls occur.
- `go test ./...` passes.
## Phase 9: Structured LLM client and scheduler infrastructure ## Phase 9: Structured LLM client and scheduler infrastructure

View File

@@ -361,7 +361,7 @@ func runProcess(args []string, stdout, stderr io.Writer) int {
hasSkippedCorrections := false hasSkippedCorrections := false
if runOutput != nil { if runOutput != nil {
for _, mr := range runOutput.ModuleResults { for _, mr := range runOutput.ModuleResults {
if len(mr.SkippedChanges) > 0 { if len(mr.SkippedChanges) > 0 || len(mr.ValidatorRejected) > 0 {
hasSkippedCorrections = true hasSkippedCorrections = true
break break
} }
@@ -395,7 +395,7 @@ func extractErrorPhase(err error) (phase string, message string) {
func buildProcessReport(status string, inv processInvocation, runDir *diagnostics.RunDirectory, startedAt, completedAt time.Time, errorMessage string, errorPhase string, normalizationSummary *normalization.NormalizationSummary, chunkingSummary *chunking.Summary, runOutput *runner.RunOutput) reporting.ProcessReport { func buildProcessReport(status string, inv processInvocation, runDir *diagnostics.RunDirectory, startedAt, completedAt time.Time, errorMessage string, errorPhase string, normalizationSummary *normalization.NormalizationSummary, chunkingSummary *chunking.Summary, runOutput *runner.RunOutput) reporting.ProcessReport {
report := reporting.ProcessReport{ report := reporting.ProcessReport{
Phase: "phase7-runner", Phase: "phase8-validators",
Status: status, Status: status,
Operation: "process", Operation: "process",
TranscriptPath: inv.TranscriptPath, TranscriptPath: inv.TranscriptPath,
@@ -466,6 +466,8 @@ func buildModuleReporting(runOutput *runner.RunOutput) (*reporting.ModulesSummar
ReplacementPolicy: string(r.ReplacementPolicy), ReplacementPolicy: string(r.ReplacementPolicy),
Status: r.Status, Status: r.Status,
ProposalCount: r.ProposalCount, ProposalCount: r.ProposalCount,
ValidatorDecisions: mapValidatorDecisions(r.ValidatorDecisions),
ValidatorRejected: mapValidatorRejected(r.ValidatorRejected),
AppliedChanges: r.AppliedChanges, AppliedChanges: r.AppliedChanges,
SkippedChanges: r.SkippedChanges, SkippedChanges: r.SkippedChanges,
ErrorMessage: r.ErrorMessage, ErrorMessage: r.ErrorMessage,
@@ -473,7 +475,7 @@ func buildModuleReporting(runOutput *runner.RunOutput) (*reporting.ModulesSummar
CompletedAt: &completedAt, CompletedAt: &completedAt,
}) })
summary.TotalAppliedChanges += len(r.AppliedChanges) summary.TotalAppliedChanges += len(r.AppliedChanges)
summary.TotalSkippedChanges += len(r.SkippedChanges) summary.TotalSkippedChanges += len(r.SkippedChanges) + len(r.ValidatorRejected)
if r.Status == runner.ModuleStatusFailed && summary.FailedModuleInstance == "" { if r.Status == runner.ModuleStatusFailed && summary.FailedModuleInstance == "" {
summary.FailedModuleInstance = r.ModuleInstance summary.FailedModuleInstance = r.ModuleInstance
} }
@@ -482,6 +484,44 @@ func buildModuleReporting(runOutput *runner.RunOutput) (*reporting.ModulesSummar
return summary, moduleReports return summary, moduleReports
} }
func mapValidatorDecisions(in []runner.ValidatorDecisionRecord) []reporting.ValidatorDecisionReport {
if len(in) == 0 {
return nil
}
out := make([]reporting.ValidatorDecisionReport, len(in))
for i, d := range in {
out[i] = reporting.ValidatorDecisionReport{
ValidatorName: d.ValidatorName,
ProposalIndex: d.ProposalIndex,
Approved: d.Approved,
ReasonCode: d.ReasonCode,
Message: d.Message,
}
}
return out
}
func mapValidatorRejected(in []runner.ValidatorRejectedChange) []reporting.ValidatorRejectedReport {
if len(in) == 0 {
return nil
}
out := make([]reporting.ValidatorRejectedReport, len(in))
for i, d := range in {
out[i] = reporting.ValidatorRejectedReport{
ValidatorName: d.ValidatorName,
ProposalIndex: d.ProposalIndex,
ModuleKey: d.ModuleKey,
ModuleInstance: d.ModuleInstance,
TargetSegmentID: d.TargetSegmentID,
OriginalText: d.OriginalText,
CorrectedText: d.CorrectedText,
ReasonCode: d.ReasonCode,
Message: d.Message,
}
}
return out
}
type processFlags struct { type processFlags struct {
glossaryPath *string glossaryPath *string
outputPath *string outputPath *string

View File

@@ -15,6 +15,7 @@ import (
"gitea.maximumdirect.net/eric/audita/internal/core/schema" "gitea.maximumdirect.net/eric/audita/internal/core/schema"
"gitea.maximumdirect.net/eric/audita/internal/framework/contracts" "gitea.maximumdirect.net/eric/audita/internal/framework/contracts"
"gitea.maximumdirect.net/eric/audita/internal/framework/proposals" "gitea.maximumdirect.net/eric/audita/internal/framework/proposals"
"gitea.maximumdirect.net/eric/audita/internal/framework/validators"
) )
func TestRunRootHelp(t *testing.T) { func TestRunRootHelp(t *testing.T) {
@@ -613,8 +614,8 @@ func TestRunProcessReportJSONIncludesChunkingSummary(t *testing.T) {
if report.Chunking.MaxSectionTokens == 0 { if report.Chunking.MaxSectionTokens == 0 {
t.Errorf("expected max_section_tokens in report") t.Errorf("expected max_section_tokens in report")
} }
if report.Phase != "phase7-runner" { if report.Phase != "phase8-validators" {
t.Errorf("expected phase 'phase7-runner', got %q", report.Phase) t.Errorf("expected phase 'phase8-validators', got %q", report.Phase)
} }
} }
@@ -633,12 +634,13 @@ func (f fakeModuleFactory) ModuleForSpec(spec contracts.ModuleRunSpec) (contract
type fakeModule struct { type fakeModule struct {
key string key string
policy proposals.ReplacementPolicy policy proposals.ReplacementPolicy
validators []contracts.Validator
proposeF func(req contracts.ProposalRequest) ([]proposals.CorrectionProposal, error) proposeF func(req contracts.ProposalRequest) ([]proposals.CorrectionProposal, error)
} }
func (m fakeModule) Key() string { return m.key } func (m fakeModule) Key() string { return m.key }
func (m fakeModule) ReplacementPolicy() proposals.ReplacementPolicy { return m.policy } func (m fakeModule) ReplacementPolicy() proposals.ReplacementPolicy { return m.policy }
func (m fakeModule) Validators() []contracts.Validator { return nil } func (m fakeModule) Validators() []contracts.Validator { return m.validators }
func (m fakeModule) Propose(ctx context.Context, req contracts.ProposalRequest) ([]proposals.CorrectionProposal, error) { func (m fakeModule) Propose(ctx context.Context, req contracts.ProposalRequest) ([]proposals.CorrectionProposal, error) {
if m.proposeF == nil { if m.proposeF == nil {
return nil, nil return nil, nil
@@ -646,14 +648,32 @@ func (m fakeModule) Propose(ctx context.Context, req contracts.ProposalRequest)
return m.proposeF(req) return m.proposeF(req)
} }
type fakeValidator struct {
name string
validateF func(req contracts.ValidationRequest) (validators.Result, error)
}
func (v fakeValidator) Name() string { return v.name }
func (v fakeValidator) Validate(ctx context.Context, req contracts.ValidationRequest) (validators.Result, error) {
_ = ctx
return v.validateF(req)
}
func TestRunProcessInjectedFactoryExecutesRunnerAndReportsModules(t *testing.T) { func TestRunProcessInjectedFactoryExecutesRunnerAndReportsModules(t *testing.T) {
allow := fakeValidator{name: "allow", validateF: func(req contracts.ValidationRequest) (validators.Result, error) {
decisions := make([]validators.Decision, len(req.CandidateProposal))
for i, p := range req.CandidateProposal {
decisions[i] = validators.Decision{ProposalIndex: p.ProposalIndex, Approved: true, ReasonCode: validators.ReasonApproved, Message: "approved"}
}
return validators.Result{ValidatorName: "allow", Decisions: decisions}, nil
}}
processModuleFactory = fakeModuleFactory{modules: map[string]contracts.TranscriptModule{ processModuleFactory = fakeModuleFactory{modules: map[string]contracts.TranscriptModule{
"m1": fakeModule{key: "m1", policy: proposals.ReplacementPolicyRequireUnique, proposeF: func(req contracts.ProposalRequest) ([]proposals.CorrectionProposal, error) { "m1": fakeModule{key: "m1", policy: proposals.ReplacementPolicyRequireUnique, validators: []contracts.Validator{allow}, proposeF: func(req contracts.ProposalRequest) ([]proposals.CorrectionProposal, error) {
return []proposals.CorrectionProposal{ return []proposals.CorrectionProposal{
{TargetSegmentID: 1, OriginalText: "Hello", CorrectedText: "Hi", Confidence: 1}, {TargetSegmentID: 1, OriginalText: "Hello", CorrectedText: "Hi", Confidence: 1},
}, nil }, nil
}}, }},
"m2": fakeModule{key: "m2", policy: proposals.ReplacementPolicyRequireUnique, proposeF: func(req contracts.ProposalRequest) ([]proposals.CorrectionProposal, error) { "m2": fakeModule{key: "m2", policy: proposals.ReplacementPolicyRequireUnique, validators: []contracts.Validator{allow}, proposeF: func(req contracts.ProposalRequest) ([]proposals.CorrectionProposal, error) {
if req.WorkingTranscript.Segments[0].Text != "Hi world" { if req.WorkingTranscript.Segments[0].Text != "Hi world" {
t.Fatalf("expected module 2 to see module 1 changes, got %q", req.WorkingTranscript.Segments[0].Text) t.Fatalf("expected module 2 to see module 1 changes, got %q", req.WorkingTranscript.Segments[0].Text)
} }
@@ -711,11 +731,17 @@ func TestRunProcessInjectedFactoryExecutesRunnerAndReportsModules(t *testing.T)
if len(report.ModuleResults[1].SkippedChanges) != 1 { if len(report.ModuleResults[1].SkippedChanges) != 1 {
t.Fatalf("expected skipped change for module 2, got %+v", report.ModuleResults[1].SkippedChanges) t.Fatalf("expected skipped change for module 2, got %+v", report.ModuleResults[1].SkippedChanges)
} }
if len(report.ModuleResults[0].ValidatorDecisions) == 0 {
t.Fatalf("expected validator decisions in module report")
}
runDirReport := readProcessReport(t, filepath.Join(onlyRunDir(t, workDir), "report.json")) runDirReport := readProcessReport(t, filepath.Join(onlyRunDir(t, workDir), "report.json"))
if len(runDirReport.ModuleResults) != 2 { if len(runDirReport.ModuleResults) != 2 {
t.Fatalf("expected module results in run-dir report") t.Fatalf("expected module results in run-dir report")
} }
if len(runDirReport.ModuleResults[1].ValidatorDecisions) == 0 {
t.Fatalf("expected validator decisions in run-dir report")
}
} }
func TestRunProcessInjectedFactorySkippedKeepsAutoRetention(t *testing.T) { func TestRunProcessInjectedFactorySkippedKeepsAutoRetention(t *testing.T) {

View File

@@ -38,6 +38,8 @@ type ModuleReport struct {
ReplacementPolicy string `json:"replacement_policy,omitempty"` ReplacementPolicy string `json:"replacement_policy,omitempty"`
Status string `json:"status"` Status string `json:"status"`
ProposalCount int `json:"proposal_count"` ProposalCount int `json:"proposal_count"`
ValidatorDecisions []ValidatorDecisionReport `json:"validator_decisions,omitempty"`
ValidatorRejected []ValidatorRejectedReport `json:"validator_rejected,omitempty"`
AppliedChanges []proposals.AppliedChange `json:"applied_changes,omitempty"` AppliedChanges []proposals.AppliedChange `json:"applied_changes,omitempty"`
SkippedChanges []proposals.SkippedChange `json:"skipped_changes,omitempty"` SkippedChanges []proposals.SkippedChange `json:"skipped_changes,omitempty"`
ErrorMessage string `json:"error_message,omitempty"` ErrorMessage string `json:"error_message,omitempty"`
@@ -45,6 +47,26 @@ type ModuleReport struct {
CompletedAt *time.Time `json:"completed_at,omitempty"` CompletedAt *time.Time `json:"completed_at,omitempty"`
} }
type ValidatorDecisionReport struct {
ValidatorName string `json:"validator_name"`
ProposalIndex int `json:"proposal_index"`
Approved bool `json:"approved"`
ReasonCode string `json:"reason_code"`
Message string `json:"message"`
}
type ValidatorRejectedReport struct {
ValidatorName string `json:"validator_name"`
ProposalIndex int `json:"proposal_index"`
ModuleKey string `json:"module_key"`
ModuleInstance string `json:"module_instance"`
TargetSegmentID int `json:"target_segment_id"`
OriginalText string `json:"original_text"`
CorrectedText string `json:"corrected_text"`
ReasonCode string `json:"reason_code"`
Message string `json:"message"`
}
type ModulesSummary struct { type ModulesSummary struct {
ModuleCount int `json:"module_count"` ModuleCount int `json:"module_count"`
TotalAppliedChanges int `json:"total_applied_changes"` TotalAppliedChanges int `json:"total_applied_changes"`

View File

@@ -11,7 +11,7 @@ import (
func TestProcessReportModuleResultsJSONSuccessAndSkipped(t *testing.T) { func TestProcessReportModuleResultsJSONSuccessAndSkipped(t *testing.T) {
now := time.Now().UTC() now := time.Now().UTC()
report := ProcessReport{ report := ProcessReport{
Phase: "phase7-runner", Phase: "phase8-validators",
Status: "success", Status: "success",
ModuleResults: []ModuleReport{ ModuleResults: []ModuleReport{
{ {
@@ -20,6 +20,24 @@ func TestProcessReportModuleResultsJSONSuccessAndSkipped(t *testing.T) {
ReplacementPolicy: "require_unique", ReplacementPolicy: "require_unique",
Status: "success", Status: "success",
ProposalCount: 2, ProposalCount: 2,
ValidatorDecisions: []ValidatorDecisionReport{{
ValidatorName: "confidence_threshold",
ProposalIndex: 0,
Approved: true,
ReasonCode: "approved",
Message: "approved",
}},
ValidatorRejected: []ValidatorRejectedReport{{
ValidatorName: "confidence_threshold",
ProposalIndex: 1,
ModuleKey: "grammar",
ModuleInstance: "grammar_1",
TargetSegmentID: 1,
OriginalText: "bad",
CorrectedText: "worse",
ReasonCode: "low_confidence",
Message: "rejected",
}},
AppliedChanges: []proposals.AppliedChange{{ AppliedChanges: []proposals.AppliedChange{{
ProposalIndex: 0, ProposalIndex: 0,
ModuleKey: "grammar", ModuleKey: "grammar",
@@ -51,6 +69,9 @@ func TestProcessReportModuleResultsJSONSuccessAndSkipped(t *testing.T) {
if len(parsed.ModuleResults) != 1 { if len(parsed.ModuleResults) != 1 {
t.Fatalf("expected one module result, got %d", len(parsed.ModuleResults)) t.Fatalf("expected one module result, got %d", len(parsed.ModuleResults))
} }
if len(parsed.ModuleResults[0].ValidatorDecisions) != 1 || len(parsed.ModuleResults[0].ValidatorRejected) != 1 {
t.Fatalf("expected validator result payloads in module report")
}
if parsed.ModulesSummary == nil || parsed.ModulesSummary.TotalSkippedChanges != 1 { if parsed.ModulesSummary == nil || parsed.ModulesSummary.TotalSkippedChanges != 1 {
t.Fatalf("unexpected module summary: %+v", parsed.ModulesSummary) t.Fatalf("unexpected module summary: %+v", parsed.ModulesSummary)
} }
@@ -59,7 +80,7 @@ func TestProcessReportModuleResultsJSONSuccessAndSkipped(t *testing.T) {
func TestProcessReportModuleResultsJSONFailedModule(t *testing.T) { func TestProcessReportModuleResultsJSONFailedModule(t *testing.T) {
now := time.Now().UTC() now := time.Now().UTC()
report := ProcessReport{ report := ProcessReport{
Phase: "phase7-runner", Phase: "phase8-validators",
Status: "failed", Status: "failed",
ModuleResults: []ModuleReport{ ModuleResults: []ModuleReport{
{ {

View File

@@ -9,6 +9,7 @@ import (
"gitea.maximumdirect.net/eric/audita/internal/core/config" "gitea.maximumdirect.net/eric/audita/internal/core/config"
"gitea.maximumdirect.net/eric/audita/internal/core/schema" "gitea.maximumdirect.net/eric/audita/internal/core/schema"
"gitea.maximumdirect.net/eric/audita/internal/framework/proposals" "gitea.maximumdirect.net/eric/audita/internal/framework/proposals"
"gitea.maximumdirect.net/eric/audita/internal/framework/validators"
) )
// StructuredLLMClient provides provider-agnostic structured completion. // StructuredLLMClient provides provider-agnostic structured completion.
@@ -27,7 +28,7 @@ type TranscriptModule interface {
// Validator evaluates candidate proposals and returns one decision per proposal index. // Validator evaluates candidate proposals and returns one decision per proposal index.
type Validator interface { type Validator interface {
Name() string Name() string
Validate(ctx context.Context, req ValidationRequest) ([]ValidationDecision, error) Validate(ctx context.Context, req ValidationRequest) (validators.Result, error)
} }
// StructuredCompletionRequest is a transport-neutral structured completion request. // StructuredCompletionRequest is a transport-neutral structured completion request.
@@ -90,19 +91,7 @@ type ProposalRequest struct {
} }
// ValidationRequest is the input to validator execution. // ValidationRequest is the input to validator execution.
type ValidationRequest struct { type ValidationRequest = validators.Request
ExecutionContext
RunSpec ModuleRunSpec `json:"run_spec"`
CandidateProposals []proposals.EnrichedCorrectionProposal `json:"candidate_proposals"`
}
// ValidationDecision is one validator decision for one proposal index.
type ValidationDecision struct {
ProposalIndex int `json:"proposal_index"`
Approved bool `json:"approved"`
Confidence *float64 `json:"confidence,omitempty"`
Reason string `json:"reason,omitempty"`
}
// ResolveModuleRunSpecs deterministically resolves instance names from logical keys. // ResolveModuleRunSpecs deterministically resolves instance names from logical keys.
// Repeated keys are suffixed with _<n> (1-based), while singleton keys keep their raw key. // Repeated keys are suffixed with _<n> (1-based), while singleton keys keep their raw key.

View File

@@ -8,6 +8,7 @@ import (
"gitea.maximumdirect.net/eric/audita/internal/core/chunking" "gitea.maximumdirect.net/eric/audita/internal/core/chunking"
"gitea.maximumdirect.net/eric/audita/internal/framework/proposals" "gitea.maximumdirect.net/eric/audita/internal/framework/proposals"
"gitea.maximumdirect.net/eric/audita/internal/framework/validators"
) )
type fakeLLMClient struct{} type fakeLLMClient struct{}
@@ -22,13 +23,13 @@ type fakeValidator struct{}
func (f *fakeValidator) Name() string { return "fake-validator" } func (f *fakeValidator) Name() string { return "fake-validator" }
func (f *fakeValidator) Validate(ctx context.Context, req ValidationRequest) ([]ValidationDecision, error) { func (f *fakeValidator) Validate(ctx context.Context, req ValidationRequest) (validators.Result, error) {
_ = ctx _ = ctx
decisions := make([]ValidationDecision, len(req.CandidateProposals)) decisions := make([]validators.Decision, len(req.CandidateProposal))
for i, proposal := range req.CandidateProposals { for i, proposal := range req.CandidateProposal {
decisions[i] = ValidationDecision{ProposalIndex: proposal.ProposalIndex, Approved: true} decisions[i] = validators.Decision{ProposalIndex: proposal.ProposalIndex, Approved: true, ReasonCode: validators.ReasonApproved, Message: "approved"}
} }
return decisions, nil return validators.Result{ValidatorName: f.Name(), Decisions: decisions}, nil
} }
type fakeModule struct{} type fakeModule struct{}

View File

@@ -9,6 +9,7 @@ import (
"gitea.maximumdirect.net/eric/audita/internal/core/schema" "gitea.maximumdirect.net/eric/audita/internal/core/schema"
"gitea.maximumdirect.net/eric/audita/internal/framework/contracts" "gitea.maximumdirect.net/eric/audita/internal/framework/contracts"
"gitea.maximumdirect.net/eric/audita/internal/framework/proposals" "gitea.maximumdirect.net/eric/audita/internal/framework/proposals"
"gitea.maximumdirect.net/eric/audita/internal/framework/validators"
) )
const ( const (
@@ -33,6 +34,8 @@ type ModuleResult struct {
ReplacementPolicy proposals.ReplacementPolicy `json:"replacement_policy"` ReplacementPolicy proposals.ReplacementPolicy `json:"replacement_policy"`
Status string `json:"status"` Status string `json:"status"`
ProposalCount int `json:"proposal_count"` ProposalCount int `json:"proposal_count"`
ValidatorDecisions []ValidatorDecisionRecord `json:"validator_decisions,omitempty"`
ValidatorRejected []ValidatorRejectedChange `json:"validator_rejected,omitempty"`
AppliedChanges []proposals.AppliedChange `json:"applied_changes,omitempty"` AppliedChanges []proposals.AppliedChange `json:"applied_changes,omitempty"`
SkippedChanges []proposals.SkippedChange `json:"skipped_changes,omitempty"` SkippedChanges []proposals.SkippedChange `json:"skipped_changes,omitempty"`
ErrorMessage string `json:"error_message,omitempty"` ErrorMessage string `json:"error_message,omitempty"`
@@ -40,6 +43,26 @@ type ModuleResult struct {
CompletedAt time.Time `json:"completed_at"` CompletedAt time.Time `json:"completed_at"`
} }
type ValidatorDecisionRecord struct {
ValidatorName string `json:"validator_name"`
ProposalIndex int `json:"proposal_index"`
Approved bool `json:"approved"`
ReasonCode string `json:"reason_code"`
Message string `json:"message"`
}
type ValidatorRejectedChange struct {
ValidatorName string `json:"validator_name"`
ProposalIndex int `json:"proposal_index"`
ModuleKey string `json:"module_key"`
ModuleInstance string `json:"module_instance"`
TargetSegmentID int `json:"target_segment_id"`
OriginalText string `json:"original_text"`
CorrectedText string `json:"corrected_text"`
ReasonCode string `json:"reason_code"`
Message string `json:"message"`
}
// RunInput is the deterministic runner input. // RunInput is the deterministic runner input.
type RunInput struct { type RunInput struct {
Config *config.Config Config *config.Config
@@ -121,7 +144,86 @@ func (r *Runner) Run(ctx context.Context, input RunInput) (RunOutput, error) {
}) })
} }
applyResult := proposals.ApplyProposals(working, enriched, policy) validatorDecisions := make([]ValidatorDecisionRecord, 0)
validatorRejected := make([]ValidatorRejectedChange, 0)
eligible := enriched
for _, validator := range module.Validators() {
vResult, vErr := validator.Validate(ctx, contracts.ValidationRequest{
WorkingTranscript: working,
CandidateProposal: eligible,
ModuleKey: spec.ModuleKey,
ModuleInstance: spec.InstanceName,
ReplacementPolicy: policy,
Glossary: input.Glossary,
Config: input.Config,
})
if vErr != nil {
failed := ModuleResult{
ModuleKey: spec.ModuleKey,
ModuleInstance: spec.InstanceName,
ReplacementPolicy: policy,
Status: ModuleStatusFailed,
ProposalCount: len(enriched),
ValidatorDecisions: validatorDecisions,
ValidatorRejected: validatorRejected,
ErrorMessage: vErr.Error(),
StartedAt: startedAt,
CompletedAt: time.Now().UTC(),
}
results = append(results, failed)
return RunOutput{FinalTranscript: working, ModuleResults: results}, fmt.Errorf("module %q validator %q failed: %w", spec.InstanceName, validator.Name(), vErr)
}
if err := validators.EnforceDecisionCardinality(eligible, vResult.Decisions); err != nil {
failed := ModuleResult{
ModuleKey: spec.ModuleKey,
ModuleInstance: spec.InstanceName,
ReplacementPolicy: policy,
Status: ModuleStatusFailed,
ProposalCount: len(enriched),
ValidatorDecisions: validatorDecisions,
ValidatorRejected: validatorRejected,
ErrorMessage: err.Error(),
StartedAt: startedAt,
CompletedAt: time.Now().UTC(),
}
results = append(results, failed)
return RunOutput{FinalTranscript: working, ModuleResults: results}, fmt.Errorf("module %q validator %q cardinality failed: %w", spec.InstanceName, validator.Name(), err)
}
nextEligible := make([]proposals.EnrichedCorrectionProposal, 0, len(eligible))
byIndex := make(map[int]proposals.EnrichedCorrectionProposal, len(eligible))
for _, p := range eligible {
byIndex[p.ProposalIndex] = p
}
for _, d := range vResult.Decisions {
validatorDecisions = append(validatorDecisions, ValidatorDecisionRecord{
ValidatorName: validator.Name(),
ProposalIndex: d.ProposalIndex,
Approved: d.Approved,
ReasonCode: d.ReasonCode,
Message: d.Message,
})
if d.Approved {
nextEligible = append(nextEligible, byIndex[d.ProposalIndex])
continue
}
p := byIndex[d.ProposalIndex]
validatorRejected = append(validatorRejected, ValidatorRejectedChange{
ValidatorName: validator.Name(),
ProposalIndex: p.ProposalIndex,
ModuleKey: p.ModuleKey,
ModuleInstance: p.ModuleInstance,
TargetSegmentID: p.TargetSegmentID,
OriginalText: p.OriginalText,
CorrectedText: p.CorrectedText,
ReasonCode: d.ReasonCode,
Message: d.Message,
})
}
eligible = nextEligible
}
applyResult := proposals.ApplyProposals(working, eligible, policy)
working = applyResult.Transcript working = applyResult.Transcript
results = append(results, ModuleResult{ results = append(results, ModuleResult{
@@ -130,6 +232,8 @@ func (r *Runner) Run(ctx context.Context, input RunInput) (RunOutput, error) {
ReplacementPolicy: policy, ReplacementPolicy: policy,
Status: ModuleStatusSuccess, Status: ModuleStatusSuccess,
ProposalCount: len(enriched), ProposalCount: len(enriched),
ValidatorDecisions: validatorDecisions,
ValidatorRejected: validatorRejected,
AppliedChanges: applyResult.Applied, AppliedChanges: applyResult.Applied,
SkippedChanges: applyResult.Skipped, SkippedChanges: applyResult.Skipped,
StartedAt: startedAt, StartedAt: startedAt,

View File

@@ -9,6 +9,7 @@ import (
"gitea.maximumdirect.net/eric/audita/internal/core/schema" "gitea.maximumdirect.net/eric/audita/internal/core/schema"
"gitea.maximumdirect.net/eric/audita/internal/framework/contracts" "gitea.maximumdirect.net/eric/audita/internal/framework/contracts"
"gitea.maximumdirect.net/eric/audita/internal/framework/proposals" "gitea.maximumdirect.net/eric/audita/internal/framework/proposals"
"gitea.maximumdirect.net/eric/audita/internal/framework/validators"
) )
type fakeFactory struct { type fakeFactory struct {
@@ -26,12 +27,13 @@ func (f fakeFactory) ModuleForSpec(spec contracts.ModuleRunSpec) (contracts.Tran
type fakeModule struct { type fakeModule struct {
key string key string
policy proposals.ReplacementPolicy policy proposals.ReplacementPolicy
validators []contracts.Validator
proposeF func(req contracts.ProposalRequest) ([]proposals.CorrectionProposal, error) proposeF func(req contracts.ProposalRequest) ([]proposals.CorrectionProposal, error)
} }
func (m fakeModule) Key() string { return m.key } func (m fakeModule) Key() string { return m.key }
func (m fakeModule) ReplacementPolicy() proposals.ReplacementPolicy { return m.policy } func (m fakeModule) ReplacementPolicy() proposals.ReplacementPolicy { return m.policy }
func (m fakeModule) Validators() []contracts.Validator { return nil } func (m fakeModule) Validators() []contracts.Validator { return m.validators }
func (m fakeModule) Propose(ctx context.Context, req contracts.ProposalRequest) ([]proposals.CorrectionProposal, error) { func (m fakeModule) Propose(ctx context.Context, req contracts.ProposalRequest) ([]proposals.CorrectionProposal, error) {
if m.proposeF == nil { if m.proposeF == nil {
return nil, nil return nil, nil
@@ -39,6 +41,17 @@ func (m fakeModule) Propose(ctx context.Context, req contracts.ProposalRequest)
return m.proposeF(req) return m.proposeF(req)
} }
type fakeValidator struct {
name string
validateF func(req contracts.ValidationRequest) (validators.Result, error)
}
func (v fakeValidator) Name() string { return v.name }
func (v fakeValidator) Validate(ctx context.Context, req contracts.ValidationRequest) (validators.Result, error) {
_ = ctx
return v.validateF(req)
}
func TestRunnerOneModuleAppliesProposal(t *testing.T) { func TestRunnerOneModuleAppliesProposal(t *testing.T) {
transcript := &schema.Transcript{Segments: []schema.Segment{{ID: 1, Speaker: "A", Start: 0, End: 1, Text: "teh cat"}}} transcript := &schema.Transcript{Segments: []schema.Segment{{ID: 1, Speaker: "A", Start: 0, End: 1, Text: "teh cat"}}}
r := New(fakeFactory{modules: map[string]contracts.TranscriptModule{ r := New(fakeFactory{modules: map[string]contracts.TranscriptModule{
@@ -149,4 +162,148 @@ func TestRunnerRepeatedModuleInstanceNames(t *testing.T) {
} }
} }
func TestRunnerValidatorApprovedProposalApplied(t *testing.T) {
transcript := &schema.Transcript{Segments: []schema.Segment{{ID: 1, Text: "teh cat"}}}
allowAll := fakeValidator{name: "allow", validateF: func(req contracts.ValidationRequest) (validators.Result, error) {
decisions := make([]validators.Decision, len(req.CandidateProposal))
for i, p := range req.CandidateProposal {
decisions[i] = validators.Decision{ProposalIndex: p.ProposalIndex, Approved: true, ReasonCode: validators.ReasonApproved, Message: "approved"}
}
return validators.Result{ValidatorName: "allow", Decisions: decisions}, nil
}}
r := New(fakeFactory{modules: map[string]contracts.TranscriptModule{
"m": fakeModule{
key: "m",
policy: proposals.ReplacementPolicyRequireUnique,
validators: []contracts.Validator{allowAll},
proposeF: func(req contracts.ProposalRequest) ([]proposals.CorrectionProposal, error) {
return []proposals.CorrectionProposal{{TargetSegmentID: 1, OriginalText: "teh", CorrectedText: "the", Confidence: 1}}, nil
},
},
}})
out, err := r.Run(context.Background(), RunInput{Transcript: transcript, ModuleSpecs: []contracts.ModuleRunSpec{{ModuleKey: "m", InstanceName: "m"}}})
if err != nil {
t.Fatalf("Run error: %v", err)
}
if out.FinalTranscript.Segments[0].Text != "the cat" {
t.Fatalf("expected proposal applied, got %q", out.FinalTranscript.Segments[0].Text)
}
if len(out.ModuleResults[0].ValidatorDecisions) != 1 {
t.Fatalf("expected validator decisions recorded")
}
}
func TestRunnerValidatorRejectedProposalNotApplied(t *testing.T) {
transcript := &schema.Transcript{Segments: []schema.Segment{{ID: 1, Text: "teh cat"}}}
rejectAll := fakeValidator{name: "reject", validateF: func(req contracts.ValidationRequest) (validators.Result, error) {
decisions := make([]validators.Decision, len(req.CandidateProposal))
for i, p := range req.CandidateProposal {
decisions[i] = validators.Decision{ProposalIndex: p.ProposalIndex, Approved: false, ReasonCode: validators.ReasonNoEffect, Message: "rejected"}
}
return validators.Result{ValidatorName: "reject", Decisions: decisions}, nil
}}
r := New(fakeFactory{modules: map[string]contracts.TranscriptModule{
"m": fakeModule{key: "m", policy: proposals.ReplacementPolicyRequireUnique, validators: []contracts.Validator{rejectAll}, proposeF: func(req contracts.ProposalRequest) ([]proposals.CorrectionProposal, error) {
return []proposals.CorrectionProposal{{TargetSegmentID: 1, OriginalText: "teh", CorrectedText: "the", Confidence: 1}}, nil
}},
}})
out, err := r.Run(context.Background(), RunInput{Transcript: transcript, ModuleSpecs: []contracts.ModuleRunSpec{{ModuleKey: "m", InstanceName: "m"}}})
if err != nil {
t.Fatalf("Run error: %v", err)
}
if out.FinalTranscript.Segments[0].Text != "teh cat" {
t.Fatalf("expected rejected proposal not applied, got %q", out.FinalTranscript.Segments[0].Text)
}
if len(out.ModuleResults[0].ValidatorRejected) != 1 {
t.Fatalf("expected validator rejection recorded, got %+v", out.ModuleResults[0].ValidatorRejected)
}
}
func TestRunnerMultipleValidatorsRunInOrderAndFilterSurvivors(t *testing.T) {
transcript := &schema.Transcript{Segments: []schema.Segment{{ID: 1, Text: "one two"}}}
first := fakeValidator{name: "first", validateF: func(req contracts.ValidationRequest) (validators.Result, error) {
if len(req.CandidateProposal) != 2 {
t.Fatalf("expected first validator to see 2 candidates, got %d", len(req.CandidateProposal))
}
return validators.Result{
ValidatorName: "first",
Decisions: []validators.Decision{
{ProposalIndex: req.CandidateProposal[0].ProposalIndex, Approved: true, ReasonCode: validators.ReasonApproved, Message: "ok"},
{ProposalIndex: req.CandidateProposal[1].ProposalIndex, Approved: false, ReasonCode: validators.ReasonNoEffect, Message: "reject"},
},
}, nil
}}
second := fakeValidator{name: "second", validateF: func(req contracts.ValidationRequest) (validators.Result, error) {
if len(req.CandidateProposal) != 1 {
t.Fatalf("expected second validator to see only survivors, got %d", len(req.CandidateProposal))
}
return validators.Result{ValidatorName: "second", Decisions: []validators.Decision{{ProposalIndex: req.CandidateProposal[0].ProposalIndex, Approved: true, ReasonCode: validators.ReasonApproved, Message: "ok"}}}, nil
}}
r := New(fakeFactory{modules: map[string]contracts.TranscriptModule{
"m": fakeModule{key: "m", policy: proposals.ReplacementPolicyRequireUnique, validators: []contracts.Validator{first, second}, proposeF: func(req contracts.ProposalRequest) ([]proposals.CorrectionProposal, error) {
return []proposals.CorrectionProposal{
{TargetSegmentID: 1, OriginalText: "one", CorrectedText: "ONE", Confidence: 1},
{TargetSegmentID: 1, OriginalText: "two", CorrectedText: "TWO", Confidence: 1},
}, nil
}},
}})
out, err := r.Run(context.Background(), RunInput{Transcript: transcript, ModuleSpecs: []contracts.ModuleRunSpec{{ModuleKey: "m", InstanceName: "m"}}})
if err != nil {
t.Fatalf("Run error: %v", err)
}
if out.FinalTranscript.Segments[0].Text != "ONE two" {
t.Fatalf("expected only survivor applied, got %q", out.FinalTranscript.Segments[0].Text)
}
}
func TestRunnerValidatorCardinalityErrorStopsPipelineWithPartialProgress(t *testing.T) {
transcript := &schema.Transcript{Segments: []schema.Segment{{ID: 1, Text: "teh cat"}}}
good := fakeModule{key: "m1", policy: proposals.ReplacementPolicyRequireUnique, proposeF: func(req contracts.ProposalRequest) ([]proposals.CorrectionProposal, error) {
return []proposals.CorrectionProposal{{TargetSegmentID: 1, OriginalText: "teh", CorrectedText: "the", Confidence: 1}}, nil
}}
badValidator := fakeValidator{name: "bad", validateF: func(req contracts.ValidationRequest) (validators.Result, error) {
// Missing one decision triggers cardinality error.
return validators.Result{ValidatorName: "bad", Decisions: nil}, nil
}}
bad := fakeModule{key: "m2", policy: proposals.ReplacementPolicyRequireUnique, validators: []contracts.Validator{badValidator}, proposeF: func(req contracts.ProposalRequest) ([]proposals.CorrectionProposal, error) {
return []proposals.CorrectionProposal{{TargetSegmentID: 1, OriginalText: "cat", CorrectedText: "dog", Confidence: 1}}, nil
}}
r := New(fakeFactory{modules: map[string]contracts.TranscriptModule{"m1": good, "m2": bad}})
out, err := r.Run(context.Background(), RunInput{Transcript: transcript, ModuleSpecs: []contracts.ModuleRunSpec{{ModuleKey: "m1", InstanceName: "m1"}, {ModuleKey: "m2", InstanceName: "m2"}}})
if err == nil {
t.Fatal("expected cardinality error")
}
if out.FinalTranscript.Segments[0].Text != "the cat" {
t.Fatalf("expected partial progress preserved, got %q", out.FinalTranscript.Segments[0].Text)
}
if len(out.ModuleResults) != 2 || out.ModuleResults[1].Status != ModuleStatusFailed {
t.Fatalf("expected second module failed")
}
}
func TestRunnerApplicationSkipAfterValidatorApprovalReported(t *testing.T) {
transcript := &schema.Transcript{Segments: []schema.Segment{{ID: 1, Text: "word word"}}}
allow := fakeValidator{name: "allow", validateF: func(req contracts.ValidationRequest) (validators.Result, error) {
return validators.Result{ValidatorName: "allow", Decisions: []validators.Decision{{ProposalIndex: req.CandidateProposal[0].ProposalIndex, Approved: true, ReasonCode: validators.ReasonApproved, Message: "ok"}}}, nil
}}
r := New(fakeFactory{modules: map[string]contracts.TranscriptModule{
"m": fakeModule{key: "m", policy: proposals.ReplacementPolicyRequireUnique, validators: []contracts.Validator{allow}, proposeF: func(req contracts.ProposalRequest) ([]proposals.CorrectionProposal, error) {
return []proposals.CorrectionProposal{{TargetSegmentID: 1, OriginalText: "word", CorrectedText: "term", Confidence: 1}}, nil
}},
}})
out, err := r.Run(context.Background(), RunInput{Transcript: transcript, ModuleSpecs: []contracts.ModuleRunSpec{{ModuleKey: "m", InstanceName: "m"}}})
if err != nil {
t.Fatalf("Run error: %v", err)
}
if len(out.ModuleResults[0].SkippedChanges) != 1 {
t.Fatalf("expected application skip recorded")
}
if len(out.ModuleResults[0].ValidatorRejected) != 0 {
t.Fatalf("expected no validator rejection")
}
}
func ptrConfig(c config.Config) *config.Config { return &c } func ptrConfig(c config.Config) *config.Config { return &c }

View File

@@ -0,0 +1,161 @@
package validators
import (
"context"
"fmt"
"strings"
"gitea.maximumdirect.net/eric/audita/internal/framework/proposals"
)
type ConfidenceThresholdValidator struct{}
func (v ConfidenceThresholdValidator) Name() string { return "confidence_threshold" }
func (v ConfidenceThresholdValidator) Validate(_ context.Context, req Request) (Result, error) {
threshold := confidenceThresholdForModule(req.ModuleKey, req.Config)
decisions := make([]Decision, 0, len(req.CandidateProposal))
for _, c := range req.CandidateProposal {
if c.Confidence < threshold {
decisions = append(decisions, rejection(c.ProposalIndex, ReasonLowConfidence, fmt.Sprintf("confidence %.4f below threshold %.4f", c.Confidence, threshold)))
continue
}
decisions = append(decisions, approval(c.ProposalIndex))
}
if err := EnforceDecisionCardinality(req.CandidateProposal, decisions); err != nil {
return Result{}, err
}
return Result{ValidatorName: v.Name(), Decisions: decisions}, nil
}
type OriginalTextPresenceValidator struct{}
func (v OriginalTextPresenceValidator) Name() string { return "original_text_presence" }
func (v OriginalTextPresenceValidator) Validate(_ context.Context, req Request) (Result, error) {
byID := make(map[int]string)
if req.WorkingTranscript != nil {
for _, seg := range req.WorkingTranscript.Segments {
byID[seg.ID] = seg.Text
}
}
decisions := make([]Decision, 0, len(req.CandidateProposal))
for _, c := range req.CandidateProposal {
text, ok := byID[c.TargetSegmentID]
if !ok {
decisions = append(decisions, rejection(c.ProposalIndex, ReasonMissingTargetSegment, "target segment was not found"))
continue
}
if !strings.Contains(text, c.OriginalText) {
decisions = append(decisions, rejection(c.ProposalIndex, ReasonMissingOriginalText, "original_text was not found in current segment text"))
continue
}
decisions = append(decisions, approval(c.ProposalIndex))
}
if err := EnforceDecisionCardinality(req.CandidateProposal, decisions); err != nil {
return Result{}, err
}
return Result{ValidatorName: v.Name(), Decisions: decisions}, nil
}
type NonEmptyCorrectionValidator struct{}
func (v NonEmptyCorrectionValidator) Name() string { return "non_empty_correction" }
func (v NonEmptyCorrectionValidator) Validate(_ context.Context, req Request) (Result, error) {
decisions := make([]Decision, 0, len(req.CandidateProposal))
for _, c := range req.CandidateProposal {
if strings.TrimSpace(c.CorrectedText) == "" {
decisions = append(decisions, rejection(c.ProposalIndex, ReasonEmptyCorrectedText, "corrected_text must not be empty"))
continue
}
decisions = append(decisions, approval(c.ProposalIndex))
}
if err := EnforceDecisionCardinality(req.CandidateProposal, decisions); err != nil {
return Result{}, err
}
return Result{ValidatorName: v.Name(), Decisions: decisions}, nil
}
type NoEffectValidator struct{}
func (v NoEffectValidator) Name() string { return "no_effect" }
func (v NoEffectValidator) Validate(_ context.Context, req Request) (Result, error) {
decisions := make([]Decision, 0, len(req.CandidateProposal))
for _, c := range req.CandidateProposal {
if c.OriginalText == c.CorrectedText {
decisions = append(decisions, rejection(c.ProposalIndex, ReasonNoEffect, "original_text and corrected_text are identical"))
continue
}
decisions = append(decisions, approval(c.ProposalIndex))
}
if err := EnforceDecisionCardinality(req.CandidateProposal, decisions); err != nil {
return Result{}, err
}
return Result{ValidatorName: v.Name(), Decisions: decisions}, nil
}
type ProtectedGlossaryTermValidator struct{}
func (v ProtectedGlossaryTermValidator) Name() string { return "protected_glossary_terms" }
func (v ProtectedGlossaryTermValidator) Validate(_ context.Context, req Request) (Result, error) {
if req.ModuleKey == "glossary" {
decisions := make([]Decision, 0, len(req.CandidateProposal))
for _, c := range req.CandidateProposal {
decisions = append(decisions, approval(c.ProposalIndex))
}
return Result{ValidatorName: v.Name(), Decisions: decisions}, nil
}
terms := glossaryTerms(req)
decisions := make([]Decision, 0, len(req.CandidateProposal))
for _, c := range req.CandidateProposal {
if altersProtectedTerm(c.CorrectionProposal, terms) {
decisions = append(decisions, rejection(c.ProposalIndex, ReasonProtectedGlossaryTerm, "proposal may alter protected glossary terminology"))
continue
}
decisions = append(decisions, approval(c.ProposalIndex))
}
if err := EnforceDecisionCardinality(req.CandidateProposal, decisions); err != nil {
return Result{}, err
}
return Result{ValidatorName: v.Name(), Decisions: decisions}, nil
}
func glossaryTerms(req Request) []string {
if req.Glossary == nil {
return nil
}
out := make([]string, 0)
for _, e := range req.Glossary.Entries {
if t := strings.TrimSpace(strings.ToLower(e.Name)); t != "" {
out = append(out, t)
}
for _, a := range e.Aliases {
if t := strings.TrimSpace(strings.ToLower(a)); t != "" {
out = append(out, t)
}
}
if t := strings.TrimSpace(strings.ToLower(e.Plural)); t != "" {
out = append(out, t)
}
}
return out
}
func altersProtectedTerm(p proposals.CorrectionProposal, terms []string) bool {
if len(terms) == 0 {
return false
}
orig := strings.ToLower(p.OriginalText)
corr := strings.ToLower(p.CorrectedText)
for _, t := range terms {
if strings.Contains(orig, t) && !strings.Contains(corr, t) {
return true
}
}
return false
}

View File

@@ -0,0 +1,105 @@
package validators
import (
"context"
"fmt"
"strings"
"gitea.maximumdirect.net/eric/audita/internal/core/config"
"gitea.maximumdirect.net/eric/audita/internal/core/schema"
"gitea.maximumdirect.net/eric/audita/internal/framework/proposals"
)
const (
ReasonApproved = "approved"
ReasonLowConfidence = "low_confidence"
ReasonMissingOriginalText = "missing_original_text"
ReasonMissingTargetSegment = "missing_target_segment"
ReasonEmptyCorrectedText = "empty_corrected_text"
ReasonNoEffect = "no_effect"
ReasonProtectedGlossaryTerm = "protected_glossary_term"
)
// Request is the runtime input shared by deterministic validators.
type Request struct {
WorkingTranscript *schema.Transcript `json:"-"`
CandidateProposal []proposals.EnrichedCorrectionProposal `json:"candidate_proposals"`
ModuleKey string `json:"module_key"`
ModuleInstance string `json:"module_instance"`
ReplacementPolicy proposals.ReplacementPolicy `json:"replacement_policy"`
Glossary *schema.Glossary `json:"-"`
Config *config.Config `json:"-"`
}
// Decision is one validator decision for one proposal index.
type Decision struct {
ProposalIndex int `json:"proposal_index"`
Approved bool `json:"approved"`
ReasonCode string `json:"reason_code"`
Message string `json:"message"`
}
// Result is one validator output containing exactly one decision per proposal index.
type Result struct {
ValidatorName string `json:"validator_name"`
Decisions []Decision `json:"decisions"`
}
// Validator is the deterministic runtime validator interface.
type Validator interface {
Name() string
Validate(ctx context.Context, req Request) (Result, error)
}
// EnforceDecisionCardinality verifies every candidate proposal index receives exactly one decision.
func EnforceDecisionCardinality(candidate []proposals.EnrichedCorrectionProposal, decisions []Decision) error {
expected := make(map[int]struct{}, len(candidate))
for _, c := range candidate {
expected[c.ProposalIndex] = struct{}{}
}
seen := make(map[int]int, len(decisions))
for _, d := range decisions {
if _, ok := expected[d.ProposalIndex]; !ok {
return fmt.Errorf("unknown decision proposal index %d", d.ProposalIndex)
}
seen[d.ProposalIndex]++
if seen[d.ProposalIndex] > 1 {
return fmt.Errorf("duplicate decision proposal index %d", d.ProposalIndex)
}
}
for idx := range expected {
if seen[idx] == 0 {
return fmt.Errorf("missing decision proposal index %d", idx)
}
}
return nil
}
func approval(index int) Decision {
return Decision{ProposalIndex: index, Approved: true, ReasonCode: ReasonApproved, Message: "approved"}
}
func rejection(index int, reasonCode string, msg string) Decision {
return Decision{ProposalIndex: index, Approved: false, ReasonCode: reasonCode, Message: strings.TrimSpace(msg)}
}
func confidenceThresholdForModule(moduleKey string, cfg *config.Config) float64 {
if cfg == nil {
return 0.0
}
switch moduleKey {
case "glossary":
return cfg.Thresholds.Glossary
case "grammar":
return cfg.Thresholds.Grammar
case "homophones":
return cfg.Thresholds.Homophones
case "spoken_word":
return cfg.Thresholds.SpokenWord
default:
return 0.0
}
}

View File

@@ -0,0 +1,181 @@
package validators
import (
"context"
"strings"
"testing"
"gitea.maximumdirect.net/eric/audita/internal/core/config"
"gitea.maximumdirect.net/eric/audita/internal/core/schema"
"gitea.maximumdirect.net/eric/audita/internal/framework/proposals"
)
func mkCandidate(index int, segID int, orig, corr string, conf float64) proposals.EnrichedCorrectionProposal {
return proposals.EnrichedCorrectionProposal{
CorrectionProposal: proposals.CorrectionProposal{TargetSegmentID: segID, OriginalText: orig, CorrectedText: corr, Confidence: conf},
ProposalMetadata: proposals.ProposalMetadata{ProposalIndex: index, ModuleKey: "grammar", ModuleInstance: "grammar_1"},
}
}
func TestEnforceDecisionCardinalitySuccess(t *testing.T) {
candidates := []proposals.EnrichedCorrectionProposal{mkCandidate(0, 1, "teh", "the", 0.9), mkCandidate(1, 1, "recieve", "receive", 0.9)}
decisions := []Decision{{ProposalIndex: 0, Approved: true}, {ProposalIndex: 1, Approved: false}}
if err := EnforceDecisionCardinality(candidates, decisions); err != nil {
t.Fatalf("unexpected error: %v", err)
}
}
func TestEnforceDecisionCardinalityMissingDecision(t *testing.T) {
candidates := []proposals.EnrichedCorrectionProposal{mkCandidate(0, 1, "teh", "the", 0.9), mkCandidate(1, 1, "recieve", "receive", 0.9)}
err := EnforceDecisionCardinality(candidates, []Decision{{ProposalIndex: 0, Approved: true}})
if err == nil || !strings.Contains(err.Error(), "missing decision") {
t.Fatalf("expected missing decision error, got %v", err)
}
}
func TestEnforceDecisionCardinalityDuplicateDecision(t *testing.T) {
candidates := []proposals.EnrichedCorrectionProposal{mkCandidate(0, 1, "teh", "the", 0.9)}
err := EnforceDecisionCardinality(candidates, []Decision{{ProposalIndex: 0, Approved: true}, {ProposalIndex: 0, Approved: false}})
if err == nil || !strings.Contains(err.Error(), "duplicate decision") {
t.Fatalf("expected duplicate decision error, got %v", err)
}
}
func TestEnforceDecisionCardinalityUnknownDecision(t *testing.T) {
candidates := []proposals.EnrichedCorrectionProposal{mkCandidate(0, 1, "teh", "the", 0.9)}
err := EnforceDecisionCardinality(candidates, []Decision{{ProposalIndex: 99, Approved: true}})
if err == nil || !strings.Contains(err.Error(), "unknown decision") {
t.Fatalf("expected unknown decision error, got %v", err)
}
}
func TestConfidenceThresholdValidator(t *testing.T) {
cfg := config.Default()
cfg.Thresholds.Grammar = 0.8
req := Request{ModuleKey: "grammar", Config: &cfg, CandidateProposal: []proposals.EnrichedCorrectionProposal{
mkCandidate(0, 1, "teh", "the", 0.9),
mkCandidate(1, 1, "recieve", "receive", 0.7),
}}
res, err := (ConfidenceThresholdValidator{}).Validate(context.Background(), req)
if err != nil {
t.Fatalf("Validate error: %v", err)
}
if !res.Decisions[0].Approved || res.Decisions[0].ReasonCode != ReasonApproved {
t.Fatalf("expected first decision approved, got %+v", res.Decisions[0])
}
if res.Decisions[1].Approved || res.Decisions[1].ReasonCode != ReasonLowConfidence {
t.Fatalf("expected second decision low confidence reject, got %+v", res.Decisions[1])
}
}
func TestOriginalTextPresenceValidator(t *testing.T) {
req := Request{WorkingTranscript: &schema.Transcript{Segments: []schema.Segment{{ID: 1, Text: "hello world"}}}, CandidateProposal: []proposals.EnrichedCorrectionProposal{
mkCandidate(0, 1, "hello", "hi", 0.9),
mkCandidate(1, 1, "missing", "x", 0.9),
mkCandidate(2, 5, "hello", "hi", 0.9),
}}
res, err := (OriginalTextPresenceValidator{}).Validate(context.Background(), req)
if err != nil {
t.Fatalf("Validate error: %v", err)
}
if !res.Decisions[0].Approved {
t.Fatalf("expected proposal 0 approved")
}
if res.Decisions[1].ReasonCode != ReasonMissingOriginalText {
t.Fatalf("expected missing_original_text, got %+v", res.Decisions[1])
}
if res.Decisions[2].ReasonCode != ReasonMissingTargetSegment {
t.Fatalf("expected missing_target_segment, got %+v", res.Decisions[2])
}
}
func TestNonEmptyCorrectionValidator(t *testing.T) {
req := Request{CandidateProposal: []proposals.EnrichedCorrectionProposal{
mkCandidate(0, 1, "hello", "hi", 0.9),
mkCandidate(1, 1, "hello", " ", 0.9),
}}
res, err := (NonEmptyCorrectionValidator{}).Validate(context.Background(), req)
if err != nil {
t.Fatalf("Validate error: %v", err)
}
if !res.Decisions[0].Approved {
t.Fatalf("expected proposal 0 approved")
}
if res.Decisions[1].ReasonCode != ReasonEmptyCorrectedText {
t.Fatalf("expected empty_corrected_text, got %+v", res.Decisions[1])
}
}
func TestNoEffectValidator(t *testing.T) {
req := Request{CandidateProposal: []proposals.EnrichedCorrectionProposal{
mkCandidate(0, 1, "hello", "hello", 0.9),
mkCandidate(1, 1, "hello", "hi", 0.9),
}}
res, err := (NoEffectValidator{}).Validate(context.Background(), req)
if err != nil {
t.Fatalf("Validate error: %v", err)
}
if res.Decisions[0].ReasonCode != ReasonNoEffect || res.Decisions[0].Approved {
t.Fatalf("expected no_effect rejection, got %+v", res.Decisions[0])
}
if !res.Decisions[1].Approved {
t.Fatalf("expected proposal 1 approved")
}
}
func TestProtectedGlossaryTermValidator(t *testing.T) {
glossary := &schema.Glossary{Entries: []schema.GlossaryEntry{{Name: "OpenAI", Aliases: []string{"Open AI"}, Plural: "OpenAIs", Category: "brand", Summary: "brand"}}}
req := Request{
Glossary: glossary,
ModuleKey: "grammar",
CandidateProposal: []proposals.EnrichedCorrectionProposal{
mkCandidate(0, 1, "OpenAI", "Open A I", 0.9),
mkCandidate(1, 1, "teh", "the", 0.9),
},
}
res, err := (ProtectedGlossaryTermValidator{}).Validate(context.Background(), req)
if err != nil {
t.Fatalf("Validate error: %v", err)
}
if res.Decisions[0].ReasonCode != ReasonProtectedGlossaryTerm || res.Decisions[0].Approved {
t.Fatalf("expected protected glossary rejection, got %+v", res.Decisions[0])
}
if !res.Decisions[1].Approved {
t.Fatalf("expected non-glossary proposal approved")
}
}
func TestProtectedGlossaryTermValidatorAllowsGlossaryModule(t *testing.T) {
glossary := &schema.Glossary{Entries: []schema.GlossaryEntry{{Name: "OpenAI", Category: "brand", Summary: "brand"}}}
req := Request{
Glossary: glossary,
ModuleKey: "glossary",
CandidateProposal: []proposals.EnrichedCorrectionProposal{
mkCandidate(0, 1, "OpenAI", "Open A I", 0.9),
},
}
res, err := (ProtectedGlossaryTermValidator{}).Validate(context.Background(), req)
if err != nil {
t.Fatalf("Validate error: %v", err)
}
if !res.Decisions[0].Approved {
t.Fatalf("expected glossary module approval, got %+v", res.Decisions[0])
}
}
func TestStableReasonCodes(t *testing.T) {
codes := []string{
ReasonApproved,
ReasonLowConfidence,
ReasonMissingOriginalText,
ReasonMissingTargetSegment,
ReasonEmptyCorrectedText,
ReasonNoEffect,
ReasonProtectedGlossaryTerm,
}
for _, code := range codes {
if strings.TrimSpace(code) == "" {
t.Fatalf("reason code must not be empty")
}
}
}