Refactor validators into package-owned components
This commit is contained in:
@@ -134,8 +134,30 @@ internal/framework/validators/
|
|||||||
llm_validators.go
|
llm_validators.go
|
||||||
|
|
||||||
internal/validators/
|
internal/validators/
|
||||||
|
metadata/
|
||||||
|
metadata.go
|
||||||
registry.go
|
registry.go
|
||||||
chains.go
|
chains.go
|
||||||
|
confidence_threshold/
|
||||||
|
validator.go
|
||||||
|
original_text_presence/
|
||||||
|
validator.go
|
||||||
|
non_empty_corrected_text/
|
||||||
|
validator.go
|
||||||
|
no_effect/
|
||||||
|
validator.go
|
||||||
|
protected_terms/
|
||||||
|
validator.go
|
||||||
|
spoken_form_plausibility/
|
||||||
|
validator.go
|
||||||
|
meaning_reversal_review/
|
||||||
|
validator.go
|
||||||
|
editorial_review/
|
||||||
|
validator.go
|
||||||
|
grammar_review/
|
||||||
|
validator.go
|
||||||
|
spoken_word_review/
|
||||||
|
validator.go
|
||||||
|
|
||||||
internal/prompts/
|
internal/prompts/
|
||||||
registry.go
|
registry.go
|
||||||
@@ -195,7 +217,7 @@ Current runtime flow (`internal/cli/run.go`):
|
|||||||
- test/injected module factory path remains available for deterministic runtime tests.
|
- test/injected module factory path remains available for deterministic runtime tests.
|
||||||
- each module recomputes chunks from the current working transcript, runs chunk proposal work concurrently, aggregates deterministically, validates, and applies approved proposals once.
|
- each module recomputes chunks from the current working transcript, runs chunk proposal work concurrently, aggregates deterministically, validates, and applies approved proposals once.
|
||||||
13. Output working transcript to `--output` file or stdout.
|
13. Output working transcript to `--output` file or stdout.
|
||||||
14. Build process report (`phase` currently set to `default_pipeline`).
|
14. Build process report metadata.
|
||||||
15. Optionally write `--report-json`; always write run-dir `report.json`.
|
15. Optionally write `--report-json`; always write run-dir `report.json`.
|
||||||
16. Apply work-dir retention.
|
16. Apply work-dir retention.
|
||||||
|
|
||||||
@@ -495,6 +517,26 @@ Validator composition is now explicit and registry-backed through `internal/vali
|
|||||||
- built-in chain definitions per production module key;
|
- built-in chain definitions per production module key;
|
||||||
- production modules resolve validator chains from those built-in definitions.
|
- production modules resolve validator chains from those built-in definitions.
|
||||||
|
|
||||||
|
Package ownership boundary:
|
||||||
|
- `internal/validators/<validator_key>` owns built-in validator construction and stable key identity.
|
||||||
|
- `internal/framework/validators` remains shared runtime machinery:
|
||||||
|
- request/result models;
|
||||||
|
- decision cardinality enforcement;
|
||||||
|
- protected-vocabulary helpers;
|
||||||
|
- generic LLM-backed validator runtime, batching, and diagnostics glue.
|
||||||
|
|
||||||
|
Validator execution classification metadata:
|
||||||
|
- `internal/validators/metadata` defines execution class markers:
|
||||||
|
- `deterministic`
|
||||||
|
- `llm_backed`
|
||||||
|
- runner ordering uses this metadata interface rather than concrete framework validator type assertions.
|
||||||
|
- validators without classification metadata default to deterministic ordering.
|
||||||
|
|
||||||
|
`protected_terms` construction ownership:
|
||||||
|
- `internal/validators/protected_terms.New()` builds the general (non-glossary-stage) variant.
|
||||||
|
- `internal/validators/protected_terms.NewGlossaryStage()` builds the glossary-stage variant used by glossary chains.
|
||||||
|
- both variants preserve the stable key `protected_terms`.
|
||||||
|
|
||||||
Stable built-in validator keys:
|
Stable built-in validator keys:
|
||||||
- deterministic:
|
- deterministic:
|
||||||
- `confidence_threshold`
|
- `confidence_threshold`
|
||||||
|
|||||||
@@ -4,6 +4,36 @@ This document describes Audita's built-in validator registry and module validato
|
|||||||
|
|
||||||
For LLM-backed validator prompt asset details, see [`docs/prompts.md`](prompts.md).
|
For LLM-backed validator prompt asset details, see [`docs/prompts.md`](prompts.md).
|
||||||
|
|
||||||
|
## Package ownership
|
||||||
|
|
||||||
|
Built-in validator construction is package-owned under `internal/validators/<validator_key>`:
|
||||||
|
- `internal/validators/confidence_threshold`
|
||||||
|
- `internal/validators/original_text_presence`
|
||||||
|
- `internal/validators/non_empty_corrected_text`
|
||||||
|
- `internal/validators/no_effect`
|
||||||
|
- `internal/validators/protected_terms`
|
||||||
|
- `internal/validators/spoken_form_plausibility`
|
||||||
|
- `internal/validators/meaning_reversal_review`
|
||||||
|
- `internal/validators/editorial_review`
|
||||||
|
- `internal/validators/grammar_review`
|
||||||
|
- `internal/validators/spoken_word_review`
|
||||||
|
|
||||||
|
Registry and chain wiring stay in:
|
||||||
|
- `internal/validators/registry.go`
|
||||||
|
- `internal/validators/chains.go`
|
||||||
|
|
||||||
|
Shared validator runtime mechanics stay in `internal/framework/validators`:
|
||||||
|
- request/result/decision models
|
||||||
|
- decision cardinality helpers
|
||||||
|
- protected vocabulary helpers
|
||||||
|
- shared LLM validator runtime, batching, and diagnostics helpers
|
||||||
|
|
||||||
|
Execution classification metadata is defined in `internal/validators/metadata`:
|
||||||
|
- `deterministic`
|
||||||
|
- `llm_backed`
|
||||||
|
|
||||||
|
Runner ordering uses this metadata so deterministic validators run before LLM-backed validators without concrete framework type assertions.
|
||||||
|
|
||||||
## Scope
|
## Scope
|
||||||
|
|
||||||
Validator chains are built-in runtime behavior.
|
Validator chains are built-in runtime behavior.
|
||||||
@@ -82,6 +112,14 @@ Current built-in chains resolved from `internal/validators/chains.go`:
|
|||||||
- `grammar_review`
|
- `grammar_review`
|
||||||
- `meaning_reversal_review`
|
- `meaning_reversal_review`
|
||||||
|
|
||||||
|
## Protected terms construction
|
||||||
|
|
||||||
|
`protected_terms` has explicit constructors:
|
||||||
|
- general constructor used by non-glossary modules through the built-in registry
|
||||||
|
- glossary-stage constructor used by glossary chain resolution
|
||||||
|
|
||||||
|
Both variants preserve existing behavior and report the stable key `protected_terms`.
|
||||||
|
|
||||||
## Execution semantics
|
## Execution semantics
|
||||||
|
|
||||||
- modules execute serially;
|
- modules execute serially;
|
||||||
@@ -105,6 +143,8 @@ These are separate outcomes and are reported separately.
|
|||||||
- validator LLM diagnostics include validator identity in interaction metadata and structured response schema metadata.
|
- validator LLM diagnostics include validator identity in interaction metadata and structured response schema metadata.
|
||||||
- correction ledger entries include deterministic and LLM validator decision snapshots keyed by the same stable validator keys, and keep validator rejection distinct from application-level skip.
|
- correction ledger entries include deterministic and LLM validator decision snapshots keyed by the same stable validator keys, and keep validator rejection distinct from application-level skip.
|
||||||
|
|
||||||
|
Prompt assets are unchanged by the validator package-ownership refactor and remain built-in under `internal/prompts`.
|
||||||
|
|
||||||
## Configurable knobs that remain supported
|
## Configurable knobs that remain supported
|
||||||
|
|
||||||
- per-module confidence thresholds (`thresholds.*` / equivalent env+CLI overrides)
|
- per-module confidence thresholds (`thresholds.*` / equivalent env+CLI overrides)
|
||||||
|
|||||||
@@ -43,7 +43,7 @@ func (c noOpStructuredLLMClient) CompleteStructured(ctx context.Context, req con
|
|||||||
}
|
}
|
||||||
|
|
||||||
func shouldUseNoOpLLMClientForTests() bool {
|
func shouldUseNoOpLLMClientForTests() bool {
|
||||||
return strings.HasSuffix(filepath.Base(os.Args[0]), ".test")
|
return strings.HasSuffix(filepath.Base(os.Args[0]), ".test") || os.Getenv("GO_WANT_HELPER_PROCESS") == "1"
|
||||||
}
|
}
|
||||||
|
|
||||||
type processInvocation struct {
|
type processInvocation struct {
|
||||||
|
|||||||
@@ -28,9 +28,6 @@ func ConfigureSubprocessTestHooksFromEnv() {
|
|||||||
if mode == "" && timeoutMSRaw == "" {
|
if mode == "" && timeoutMSRaw == "" {
|
||||||
return
|
return
|
||||||
}
|
}
|
||||||
if !shouldUseNoOpLLMClientForTests() {
|
|
||||||
return
|
|
||||||
}
|
|
||||||
|
|
||||||
if mode != "" {
|
if mode != "" {
|
||||||
client := &subprocessTestLLMClient{mode: mode}
|
client := &subprocessTestLLMClient{mode: mode}
|
||||||
|
|||||||
@@ -8,7 +8,7 @@ import (
|
|||||||
"time"
|
"time"
|
||||||
|
|
||||||
"gitea.maximumdirect.net/eric/audita/internal/framework/contracts"
|
"gitea.maximumdirect.net/eric/audita/internal/framework/contracts"
|
||||||
frameworkvalidators "gitea.maximumdirect.net/eric/audita/internal/framework/validators"
|
validatormetadata "gitea.maximumdirect.net/eric/audita/internal/validators/metadata"
|
||||||
)
|
)
|
||||||
|
|
||||||
type UtilizationDiagnostics struct {
|
type UtilizationDiagnostics struct {
|
||||||
@@ -337,11 +337,7 @@ func withModuleInstanceContext(ctx context.Context, moduleInstance string) conte
|
|||||||
}
|
}
|
||||||
|
|
||||||
func isLLMBackedValidator(v contracts.Validator) bool {
|
func isLLMBackedValidator(v contracts.Validator) bool {
|
||||||
if v == nil {
|
return validatormetadata.ClassOf(v) == validatormetadata.ExecutionClassLLMBacked
|
||||||
return false
|
|
||||||
}
|
|
||||||
_, ok := v.(*frameworkvalidators.LLMBackedValidator)
|
|
||||||
return ok
|
|
||||||
}
|
}
|
||||||
|
|
||||||
func sortValidatorSummaries(in []ValidatorTimingSummary) {
|
func sortValidatorSummaries(in []ValidatorTimingSummary) {
|
||||||
|
|||||||
@@ -15,6 +15,7 @@ import (
|
|||||||
"gitea.maximumdirect.net/eric/audita/internal/framework/llm"
|
"gitea.maximumdirect.net/eric/audita/internal/framework/llm"
|
||||||
"gitea.maximumdirect.net/eric/audita/internal/framework/proposals"
|
"gitea.maximumdirect.net/eric/audita/internal/framework/proposals"
|
||||||
"gitea.maximumdirect.net/eric/audita/internal/framework/validators"
|
"gitea.maximumdirect.net/eric/audita/internal/framework/validators"
|
||||||
|
validatormetadata "gitea.maximumdirect.net/eric/audita/internal/validators/metadata"
|
||||||
)
|
)
|
||||||
|
|
||||||
const (
|
const (
|
||||||
@@ -568,7 +569,7 @@ func reorderValidatorsForPipeline(in []contracts.Validator) ([]contracts.Validat
|
|||||||
deterministic := make([]contracts.Validator, 0, len(in))
|
deterministic := make([]contracts.Validator, 0, len(in))
|
||||||
llmBacked := make([]contracts.Validator, 0, len(in))
|
llmBacked := make([]contracts.Validator, 0, len(in))
|
||||||
for _, validator := range in {
|
for _, validator := range in {
|
||||||
if _, ok := validator.(*validators.LLMBackedValidator); ok {
|
if validatormetadata.ClassOf(validator) == validatormetadata.ExecutionClassLLMBacked {
|
||||||
llmBacked = append(llmBacked, validator)
|
llmBacked = append(llmBacked, validator)
|
||||||
continue
|
continue
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -22,6 +22,7 @@ import (
|
|||||||
"gitea.maximumdirect.net/eric/audita/internal/framework/proposals"
|
"gitea.maximumdirect.net/eric/audita/internal/framework/proposals"
|
||||||
"gitea.maximumdirect.net/eric/audita/internal/framework/responseschema"
|
"gitea.maximumdirect.net/eric/audita/internal/framework/responseschema"
|
||||||
"gitea.maximumdirect.net/eric/audita/internal/framework/validators"
|
"gitea.maximumdirect.net/eric/audita/internal/framework/validators"
|
||||||
|
validatormetadata "gitea.maximumdirect.net/eric/audita/internal/validators/metadata"
|
||||||
)
|
)
|
||||||
|
|
||||||
type fakeFactory struct {
|
type fakeFactory struct {
|
||||||
@@ -64,6 +65,58 @@ func (v fakeValidator) Validate(ctx context.Context, req contracts.ValidationReq
|
|||||||
return v.validateF(req)
|
return v.validateF(req)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
type classifiedFakeValidator struct {
|
||||||
|
fakeValidator
|
||||||
|
class validatormetadata.ExecutionClass
|
||||||
|
}
|
||||||
|
|
||||||
|
func (v classifiedFakeValidator) ExecutionClass() validatormetadata.ExecutionClass {
|
||||||
|
return v.class
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestReorderValidatorsDeterministicBeforeLLMBacked(t *testing.T) {
|
||||||
|
llm := classifiedFakeValidator{
|
||||||
|
fakeValidator: fakeValidator{name: "llm", validateF: func(req contracts.ValidationRequest) (validators.Result, error) {
|
||||||
|
return validators.Result{ValidatorName: "llm"}, nil
|
||||||
|
}},
|
||||||
|
class: validatormetadata.ExecutionClassLLMBacked,
|
||||||
|
}
|
||||||
|
deterministic := classifiedFakeValidator{
|
||||||
|
fakeValidator: fakeValidator{name: "deterministic", validateF: func(req contracts.ValidationRequest) (validators.Result, error) {
|
||||||
|
return validators.Result{ValidatorName: "deterministic"}, nil
|
||||||
|
}},
|
||||||
|
class: validatormetadata.ExecutionClassDeterministic,
|
||||||
|
}
|
||||||
|
|
||||||
|
ordered, _ := reorderValidatorsForPipeline([]contracts.Validator{llm, deterministic})
|
||||||
|
if len(ordered) != 2 {
|
||||||
|
t.Fatalf("expected 2 validators, got %d", len(ordered))
|
||||||
|
}
|
||||||
|
if ordered[0].Name() != "deterministic" || ordered[1].Name() != "llm" {
|
||||||
|
t.Fatalf("unexpected validator order: %s, %s", ordered[0].Name(), ordered[1].Name())
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestReorderValidatorsDefaultsUnclassifiedToDeterministic(t *testing.T) {
|
||||||
|
unclassified := fakeValidator{name: "plain", validateF: func(req contracts.ValidationRequest) (validators.Result, error) {
|
||||||
|
return validators.Result{ValidatorName: "plain"}, nil
|
||||||
|
}}
|
||||||
|
llm := classifiedFakeValidator{
|
||||||
|
fakeValidator: fakeValidator{name: "llm", validateF: func(req contracts.ValidationRequest) (validators.Result, error) {
|
||||||
|
return validators.Result{ValidatorName: "llm"}, nil
|
||||||
|
}},
|
||||||
|
class: validatormetadata.ExecutionClassLLMBacked,
|
||||||
|
}
|
||||||
|
|
||||||
|
ordered, _ := reorderValidatorsForPipeline([]contracts.Validator{llm, unclassified})
|
||||||
|
if len(ordered) != 2 {
|
||||||
|
t.Fatalf("expected 2 validators, got %d", len(ordered))
|
||||||
|
}
|
||||||
|
if ordered[0].Name() != "plain" || ordered[1].Name() != "llm" {
|
||||||
|
t.Fatalf("unexpected validator order with unclassified validator: %s, %s", ordered[0].Name(), ordered[1].Name())
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
func TestRunnerOneModuleAppliesProposal(t *testing.T) {
|
func TestRunnerOneModuleAppliesProposal(t *testing.T) {
|
||||||
transcript := &schema.Transcript{Segments: []schema.Segment{{ID: 1, Speaker: "A", Start: 0, End: 1, Text: "teh cat"}}}
|
transcript := &schema.Transcript{Segments: []schema.Segment{{ID: 1, Speaker: "A", Start: 0, End: 1, Text: "teh cat"}}}
|
||||||
r := New(fakeFactory{modules: map[string]contracts.TranscriptModule{
|
r := New(fakeFactory{modules: map[string]contracts.TranscriptModule{
|
||||||
|
|||||||
@@ -95,23 +95,16 @@ func (v NoEffectValidator) Validate(_ context.Context, req Request) (Result, err
|
|||||||
return Result{ValidatorName: v.Name(), Decisions: decisions}, nil
|
return Result{ValidatorName: v.Name(), Decisions: decisions}, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
type ProtectedGlossaryTermValidator struct{}
|
func ValidateProtectedTerms(req Request, glossaryStage bool) (Result, error) {
|
||||||
|
|
||||||
func (v ProtectedGlossaryTermValidator) Name() string { return "protected_terms" }
|
|
||||||
|
|
||||||
func (v ProtectedGlossaryTermValidator) Validate(_ context.Context, req Request) (Result, error) {
|
|
||||||
if req.ModuleKey == "glossary" {
|
|
||||||
decisions := make([]Decision, 0, len(req.CandidateProposal))
|
|
||||||
for _, c := range req.CandidateProposal {
|
|
||||||
decisions = append(decisions, approval(c.ProposalIndex))
|
|
||||||
}
|
|
||||||
return Result{ValidatorName: v.Name(), Decisions: decisions}, nil
|
|
||||||
}
|
|
||||||
|
|
||||||
vocab := NewProtectedVocabulary(req.Glossary)
|
vocab := NewProtectedVocabulary(req.Glossary)
|
||||||
decisions := make([]Decision, 0, len(req.CandidateProposal))
|
decisions := make([]Decision, 0, len(req.CandidateProposal))
|
||||||
for _, c := range req.CandidateProposal {
|
for _, c := range req.CandidateProposal {
|
||||||
reason := vocab.violationReason(c.OriginalText, c.CorrectedText)
|
var reason string
|
||||||
|
if glossaryStage {
|
||||||
|
reason = vocab.glossaryStageViolationReason(c.OriginalText, c.CorrectedText)
|
||||||
|
} else {
|
||||||
|
reason = vocab.violationReason(c.OriginalText, c.CorrectedText)
|
||||||
|
}
|
||||||
if reason != "" {
|
if reason != "" {
|
||||||
decisions = append(decisions, rejection(c.ProposalIndex, ReasonProtectedGlossaryTerm, reason))
|
decisions = append(decisions, rejection(c.ProposalIndex, ReasonProtectedGlossaryTerm, reason))
|
||||||
continue
|
continue
|
||||||
@@ -121,28 +114,5 @@ func (v ProtectedGlossaryTermValidator) Validate(_ context.Context, req Request)
|
|||||||
if err := EnforceDecisionCardinality(req.CandidateProposal, decisions); err != nil {
|
if err := EnforceDecisionCardinality(req.CandidateProposal, decisions); err != nil {
|
||||||
return Result{}, err
|
return Result{}, err
|
||||||
}
|
}
|
||||||
return Result{ValidatorName: v.Name(), Decisions: decisions}, nil
|
return Result{ValidatorName: "protected_terms", Decisions: decisions}, nil
|
||||||
}
|
|
||||||
|
|
||||||
type GlossaryStageProtectedGlossaryTermValidator struct{}
|
|
||||||
|
|
||||||
func (v GlossaryStageProtectedGlossaryTermValidator) Name() string {
|
|
||||||
return "protected_terms"
|
|
||||||
}
|
|
||||||
|
|
||||||
func (v GlossaryStageProtectedGlossaryTermValidator) Validate(_ context.Context, req Request) (Result, error) {
|
|
||||||
vocab := NewProtectedVocabulary(req.Glossary)
|
|
||||||
decisions := make([]Decision, 0, len(req.CandidateProposal))
|
|
||||||
for _, c := range req.CandidateProposal {
|
|
||||||
reason := vocab.glossaryStageViolationReason(c.OriginalText, c.CorrectedText)
|
|
||||||
if reason != "" {
|
|
||||||
decisions = append(decisions, rejection(c.ProposalIndex, ReasonProtectedGlossaryTerm, reason))
|
|
||||||
continue
|
|
||||||
}
|
|
||||||
decisions = append(decisions, approval(c.ProposalIndex))
|
|
||||||
}
|
|
||||||
if err := EnforceDecisionCardinality(req.CandidateProposal, decisions); err != nil {
|
|
||||||
return Result{}, err
|
|
||||||
}
|
|
||||||
return Result{ValidatorName: v.Name(), Decisions: decisions}, nil
|
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -1,7 +1,6 @@
|
|||||||
package validators
|
package validators
|
||||||
|
|
||||||
import (
|
import (
|
||||||
"context"
|
|
||||||
"reflect"
|
"reflect"
|
||||||
"testing"
|
"testing"
|
||||||
|
|
||||||
@@ -31,7 +30,7 @@ func TestExtractProtectedTermsEmptyGlossary(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
func TestProtectedGlossaryTermValidatorAppliesToNonGlossaryModule(t *testing.T) {
|
func TestValidateProtectedTermsAppliesToNonGlossaryModule(t *testing.T) {
|
||||||
req := Request{
|
req := Request{
|
||||||
ModuleKey: "grammar",
|
ModuleKey: "grammar",
|
||||||
Glossary: &schema.Glossary{
|
Glossary: &schema.Glossary{
|
||||||
@@ -49,7 +48,7 @@ func TestProtectedGlossaryTermValidatorAppliesToNonGlossaryModule(t *testing.T)
|
|||||||
},
|
},
|
||||||
},
|
},
|
||||||
}
|
}
|
||||||
res, err := (ProtectedGlossaryTermValidator{}).Validate(context.Background(), req)
|
res, err := ValidateProtectedTerms(req, false)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
t.Fatalf("validator error: %v", err)
|
t.Fatalf("validator error: %v", err)
|
||||||
}
|
}
|
||||||
@@ -58,7 +57,7 @@ func TestProtectedGlossaryTermValidatorAppliesToNonGlossaryModule(t *testing.T)
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
func TestGlossaryStageProtectedGlossaryTermValidatorAllowsProtectedTermSwap(t *testing.T) {
|
func TestValidateProtectedTermsGlossaryStageAllowsProtectedTermSwap(t *testing.T) {
|
||||||
req := Request{
|
req := Request{
|
||||||
ModuleKey: "glossary",
|
ModuleKey: "glossary",
|
||||||
Glossary: &schema.Glossary{
|
Glossary: &schema.Glossary{
|
||||||
@@ -76,7 +75,7 @@ func TestGlossaryStageProtectedGlossaryTermValidatorAllowsProtectedTermSwap(t *t
|
|||||||
},
|
},
|
||||||
},
|
},
|
||||||
}
|
}
|
||||||
res, err := (GlossaryStageProtectedGlossaryTermValidator{}).Validate(context.Background(), req)
|
res, err := ValidateProtectedTerms(req, true)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
t.Fatalf("validator error: %v", err)
|
t.Fatalf("validator error: %v", err)
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -123,7 +123,7 @@ func TestNoEffectValidator(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
func TestProtectedGlossaryTermValidator(t *testing.T) {
|
func TestValidateProtectedTerms(t *testing.T) {
|
||||||
glossary := &schema.Glossary{Entries: []schema.GlossaryEntry{{Name: "OpenAI", Aliases: []string{"Open AI"}, Plural: "OpenAIs", Category: "brand", Summary: "brand"}}}
|
glossary := &schema.Glossary{Entries: []schema.GlossaryEntry{{Name: "OpenAI", Aliases: []string{"Open AI"}, Plural: "OpenAIs", Category: "brand", Summary: "brand"}}}
|
||||||
req := Request{
|
req := Request{
|
||||||
Glossary: glossary,
|
Glossary: glossary,
|
||||||
@@ -133,7 +133,7 @@ func TestProtectedGlossaryTermValidator(t *testing.T) {
|
|||||||
mkCandidate(1, 1, "teh", "the", 0.9),
|
mkCandidate(1, 1, "teh", "the", 0.9),
|
||||||
},
|
},
|
||||||
}
|
}
|
||||||
res, err := (ProtectedGlossaryTermValidator{}).Validate(context.Background(), req)
|
res, err := ValidateProtectedTerms(req, false)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
t.Fatalf("Validate error: %v", err)
|
t.Fatalf("Validate error: %v", err)
|
||||||
}
|
}
|
||||||
@@ -145,24 +145,6 @@ func TestProtectedGlossaryTermValidator(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
func TestProtectedGlossaryTermValidatorAllowsGlossaryModule(t *testing.T) {
|
|
||||||
glossary := &schema.Glossary{Entries: []schema.GlossaryEntry{{Name: "OpenAI", Category: "brand", Summary: "brand"}}}
|
|
||||||
req := Request{
|
|
||||||
Glossary: glossary,
|
|
||||||
ModuleKey: "glossary",
|
|
||||||
CandidateProposal: []proposals.EnrichedCorrectionProposal{
|
|
||||||
mkCandidate(0, 1, "OpenAI", "Open A I", 0.9),
|
|
||||||
},
|
|
||||||
}
|
|
||||||
res, err := (ProtectedGlossaryTermValidator{}).Validate(context.Background(), req)
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("Validate error: %v", err)
|
|
||||||
}
|
|
||||||
if !res.Decisions[0].Approved {
|
|
||||||
t.Fatalf("expected glossary module approval, got %+v", res.Decisions[0])
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestStableReasonCodes(t *testing.T) {
|
func TestStableReasonCodes(t *testing.T) {
|
||||||
codes := []string{
|
codes := []string{
|
||||||
ReasonApproved,
|
ReasonApproved,
|
||||||
|
|||||||
@@ -4,7 +4,7 @@ import (
|
|||||||
"fmt"
|
"fmt"
|
||||||
|
|
||||||
"gitea.maximumdirect.net/eric/audita/internal/framework/contracts"
|
"gitea.maximumdirect.net/eric/audita/internal/framework/contracts"
|
||||||
frameworkvalidators "gitea.maximumdirect.net/eric/audita/internal/framework/validators"
|
"gitea.maximumdirect.net/eric/audita/internal/validators/protected_terms"
|
||||||
)
|
)
|
||||||
|
|
||||||
var builtInChains = map[string][]string{
|
var builtInChains = map[string][]string{
|
||||||
@@ -70,7 +70,10 @@ func ResolveBuiltInChain(moduleKey string, registry *Registry) ([]contracts.Vali
|
|||||||
if moduleKey == "glossary" && key == KeyProtectedTerms {
|
if moduleKey == "glossary" && key == KeyProtectedTerms {
|
||||||
// Glossary stages preserve current stricter protection semantics while
|
// Glossary stages preserve current stricter protection semantics while
|
||||||
// reporting the stable protected_terms key.
|
// reporting the stable protected_terms key.
|
||||||
v := frameworkvalidators.GlossaryStageProtectedGlossaryTermValidator{}
|
v, buildErr := protected_terms.NewGlossaryStage()
|
||||||
|
if buildErr != nil {
|
||||||
|
return nil, buildErr
|
||||||
|
}
|
||||||
out = append(out, v)
|
out = append(out, v)
|
||||||
continue
|
continue
|
||||||
}
|
}
|
||||||
|
|||||||
13
internal/validators/confidence_threshold/validator.go
Normal file
13
internal/validators/confidence_threshold/validator.go
Normal file
@@ -0,0 +1,13 @@
|
|||||||
|
package confidence_threshold
|
||||||
|
|
||||||
|
import (
|
||||||
|
"gitea.maximumdirect.net/eric/audita/internal/framework/contracts"
|
||||||
|
frameworkvalidators "gitea.maximumdirect.net/eric/audita/internal/framework/validators"
|
||||||
|
validatormetadata "gitea.maximumdirect.net/eric/audita/internal/validators/metadata"
|
||||||
|
)
|
||||||
|
|
||||||
|
const Key = "confidence_threshold"
|
||||||
|
|
||||||
|
func New() (contracts.Validator, error) {
|
||||||
|
return validatormetadata.Wrap(frameworkvalidators.ConfidenceThresholdValidator{}, validatormetadata.ExecutionClassDeterministic), nil
|
||||||
|
}
|
||||||
17
internal/validators/editorial_review/validator.go
Normal file
17
internal/validators/editorial_review/validator.go
Normal file
@@ -0,0 +1,17 @@
|
|||||||
|
package editorial_review
|
||||||
|
|
||||||
|
import (
|
||||||
|
"gitea.maximumdirect.net/eric/audita/internal/framework/contracts"
|
||||||
|
frameworkvalidators "gitea.maximumdirect.net/eric/audita/internal/framework/validators"
|
||||||
|
validatormetadata "gitea.maximumdirect.net/eric/audita/internal/validators/metadata"
|
||||||
|
)
|
||||||
|
|
||||||
|
const Key = "editorial_review"
|
||||||
|
|
||||||
|
func New() (contracts.Validator, error) {
|
||||||
|
v, err := frameworkvalidators.NewLLMBackedValidator(Key, frameworkvalidators.LLMValidatorTypeEditorialReview, "")
|
||||||
|
if err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
return validatormetadata.Wrap(v, validatormetadata.ExecutionClassLLMBacked), nil
|
||||||
|
}
|
||||||
17
internal/validators/grammar_review/validator.go
Normal file
17
internal/validators/grammar_review/validator.go
Normal file
@@ -0,0 +1,17 @@
|
|||||||
|
package grammar_review
|
||||||
|
|
||||||
|
import (
|
||||||
|
"gitea.maximumdirect.net/eric/audita/internal/framework/contracts"
|
||||||
|
frameworkvalidators "gitea.maximumdirect.net/eric/audita/internal/framework/validators"
|
||||||
|
validatormetadata "gitea.maximumdirect.net/eric/audita/internal/validators/metadata"
|
||||||
|
)
|
||||||
|
|
||||||
|
const Key = "grammar_review"
|
||||||
|
|
||||||
|
func New() (contracts.Validator, error) {
|
||||||
|
v, err := frameworkvalidators.NewLLMBackedValidator(Key, frameworkvalidators.LLMValidatorTypeGrammarReview, "")
|
||||||
|
if err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
return validatormetadata.Wrap(v, validatormetadata.ExecutionClassLLMBacked), nil
|
||||||
|
}
|
||||||
17
internal/validators/meaning_reversal_review/validator.go
Normal file
17
internal/validators/meaning_reversal_review/validator.go
Normal file
@@ -0,0 +1,17 @@
|
|||||||
|
package meaning_reversal_review
|
||||||
|
|
||||||
|
import (
|
||||||
|
"gitea.maximumdirect.net/eric/audita/internal/framework/contracts"
|
||||||
|
frameworkvalidators "gitea.maximumdirect.net/eric/audita/internal/framework/validators"
|
||||||
|
validatormetadata "gitea.maximumdirect.net/eric/audita/internal/validators/metadata"
|
||||||
|
)
|
||||||
|
|
||||||
|
const Key = "meaning_reversal_review"
|
||||||
|
|
||||||
|
func New() (contracts.Validator, error) {
|
||||||
|
v, err := frameworkvalidators.NewLLMBackedValidator(Key, frameworkvalidators.LLMValidatorTypeMeaningReversal, "")
|
||||||
|
if err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
return validatormetadata.Wrap(v, validatormetadata.ExecutionClassLLMBacked), nil
|
||||||
|
}
|
||||||
55
internal/validators/metadata/metadata.go
Normal file
55
internal/validators/metadata/metadata.go
Normal file
@@ -0,0 +1,55 @@
|
|||||||
|
package metadata
|
||||||
|
|
||||||
|
import "gitea.maximumdirect.net/eric/audita/internal/framework/contracts"
|
||||||
|
|
||||||
|
type ExecutionClass string
|
||||||
|
|
||||||
|
const (
|
||||||
|
ExecutionClassDeterministic ExecutionClass = "deterministic"
|
||||||
|
ExecutionClassLLMBacked ExecutionClass = "llm_backed"
|
||||||
|
)
|
||||||
|
|
||||||
|
type ClassifiedValidator interface {
|
||||||
|
contracts.Validator
|
||||||
|
ExecutionClass() ExecutionClass
|
||||||
|
}
|
||||||
|
|
||||||
|
func ClassOf(v contracts.Validator) ExecutionClass {
|
||||||
|
if v == nil {
|
||||||
|
return ExecutionClassDeterministic
|
||||||
|
}
|
||||||
|
classified, ok := v.(ClassifiedValidator)
|
||||||
|
if !ok {
|
||||||
|
return ExecutionClassDeterministic
|
||||||
|
}
|
||||||
|
switch classified.ExecutionClass() {
|
||||||
|
case ExecutionClassLLMBacked:
|
||||||
|
return ExecutionClassLLMBacked
|
||||||
|
case ExecutionClassDeterministic:
|
||||||
|
return ExecutionClassDeterministic
|
||||||
|
default:
|
||||||
|
return ExecutionClassDeterministic
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func Wrap(v contracts.Validator, class ExecutionClass) contracts.Validator {
|
||||||
|
if v == nil {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
if class != ExecutionClassLLMBacked {
|
||||||
|
class = ExecutionClassDeterministic
|
||||||
|
}
|
||||||
|
return wrappedValidator{
|
||||||
|
Validator: v,
|
||||||
|
class: class,
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
type wrappedValidator struct {
|
||||||
|
contracts.Validator
|
||||||
|
class ExecutionClass
|
||||||
|
}
|
||||||
|
|
||||||
|
func (w wrappedValidator) ExecutionClass() ExecutionClass {
|
||||||
|
return w.class
|
||||||
|
}
|
||||||
30
internal/validators/metadata/metadata_test.go
Normal file
30
internal/validators/metadata/metadata_test.go
Normal file
@@ -0,0 +1,30 @@
|
|||||||
|
package metadata
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"testing"
|
||||||
|
|
||||||
|
"gitea.maximumdirect.net/eric/audita/internal/framework/contracts"
|
||||||
|
frameworkvalidators "gitea.maximumdirect.net/eric/audita/internal/framework/validators"
|
||||||
|
)
|
||||||
|
|
||||||
|
type unclassifiedValidator struct{}
|
||||||
|
|
||||||
|
func (u unclassifiedValidator) Name() string { return "unclassified" }
|
||||||
|
|
||||||
|
func (u unclassifiedValidator) Validate(_ context.Context, _ contracts.ValidationRequest) (frameworkvalidators.Result, error) {
|
||||||
|
return frameworkvalidators.Result{ValidatorName: u.Name(), Decisions: nil}, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestClassOfDefaultsToDeterministic(t *testing.T) {
|
||||||
|
if got := ClassOf(unclassifiedValidator{}); got != ExecutionClassDeterministic {
|
||||||
|
t.Fatalf("expected deterministic default class, got %q", got)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestWrapExposesExecutionClass(t *testing.T) {
|
||||||
|
wrapped := Wrap(unclassifiedValidator{}, ExecutionClassLLMBacked)
|
||||||
|
if got := ClassOf(wrapped); got != ExecutionClassLLMBacked {
|
||||||
|
t.Fatalf("expected llm_backed class, got %q", got)
|
||||||
|
}
|
||||||
|
}
|
||||||
13
internal/validators/no_effect/validator.go
Normal file
13
internal/validators/no_effect/validator.go
Normal file
@@ -0,0 +1,13 @@
|
|||||||
|
package no_effect
|
||||||
|
|
||||||
|
import (
|
||||||
|
"gitea.maximumdirect.net/eric/audita/internal/framework/contracts"
|
||||||
|
frameworkvalidators "gitea.maximumdirect.net/eric/audita/internal/framework/validators"
|
||||||
|
validatormetadata "gitea.maximumdirect.net/eric/audita/internal/validators/metadata"
|
||||||
|
)
|
||||||
|
|
||||||
|
const Key = "no_effect"
|
||||||
|
|
||||||
|
func New() (contracts.Validator, error) {
|
||||||
|
return validatormetadata.Wrap(frameworkvalidators.NoEffectValidator{}, validatormetadata.ExecutionClassDeterministic), nil
|
||||||
|
}
|
||||||
13
internal/validators/non_empty_corrected_text/validator.go
Normal file
13
internal/validators/non_empty_corrected_text/validator.go
Normal file
@@ -0,0 +1,13 @@
|
|||||||
|
package non_empty_corrected_text
|
||||||
|
|
||||||
|
import (
|
||||||
|
"gitea.maximumdirect.net/eric/audita/internal/framework/contracts"
|
||||||
|
frameworkvalidators "gitea.maximumdirect.net/eric/audita/internal/framework/validators"
|
||||||
|
validatormetadata "gitea.maximumdirect.net/eric/audita/internal/validators/metadata"
|
||||||
|
)
|
||||||
|
|
||||||
|
const Key = "non_empty_corrected_text"
|
||||||
|
|
||||||
|
func New() (contracts.Validator, error) {
|
||||||
|
return validatormetadata.Wrap(frameworkvalidators.NonEmptyCorrectionValidator{}, validatormetadata.ExecutionClassDeterministic), nil
|
||||||
|
}
|
||||||
13
internal/validators/original_text_presence/validator.go
Normal file
13
internal/validators/original_text_presence/validator.go
Normal file
@@ -0,0 +1,13 @@
|
|||||||
|
package original_text_presence
|
||||||
|
|
||||||
|
import (
|
||||||
|
"gitea.maximumdirect.net/eric/audita/internal/framework/contracts"
|
||||||
|
frameworkvalidators "gitea.maximumdirect.net/eric/audita/internal/framework/validators"
|
||||||
|
validatormetadata "gitea.maximumdirect.net/eric/audita/internal/validators/metadata"
|
||||||
|
)
|
||||||
|
|
||||||
|
const Key = "original_text_presence"
|
||||||
|
|
||||||
|
func New() (contracts.Validator, error) {
|
||||||
|
return validatormetadata.Wrap(frameworkvalidators.OriginalTextPresenceValidator{}, validatormetadata.ExecutionClassDeterministic), nil
|
||||||
|
}
|
||||||
44
internal/validators/protected_terms/validator.go
Normal file
44
internal/validators/protected_terms/validator.go
Normal file
@@ -0,0 +1,44 @@
|
|||||||
|
package protected_terms
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
|
||||||
|
"gitea.maximumdirect.net/eric/audita/internal/framework/contracts"
|
||||||
|
frameworkvalidators "gitea.maximumdirect.net/eric/audita/internal/framework/validators"
|
||||||
|
validatormetadata "gitea.maximumdirect.net/eric/audita/internal/validators/metadata"
|
||||||
|
)
|
||||||
|
|
||||||
|
const Key = "protected_terms"
|
||||||
|
|
||||||
|
func New() (contracts.Validator, error) {
|
||||||
|
return validatormetadata.Wrap(validator{}, validatormetadata.ExecutionClassDeterministic), nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func NewGlossaryStage() (contracts.Validator, error) {
|
||||||
|
return validatormetadata.Wrap(validator{glossaryStage: true}, validatormetadata.ExecutionClassDeterministic), nil
|
||||||
|
}
|
||||||
|
|
||||||
|
type validator struct {
|
||||||
|
glossaryStage bool
|
||||||
|
}
|
||||||
|
|
||||||
|
func (v validator) Name() string {
|
||||||
|
return Key
|
||||||
|
}
|
||||||
|
|
||||||
|
func (v validator) Validate(_ context.Context, req contracts.ValidationRequest) (frameworkvalidators.Result, error) {
|
||||||
|
if !v.glossaryStage && req.ModuleKey == "glossary" {
|
||||||
|
decisions := make([]frameworkvalidators.Decision, 0, len(req.CandidateProposal))
|
||||||
|
for _, c := range req.CandidateProposal {
|
||||||
|
decisions = append(decisions, frameworkvalidators.Decision{
|
||||||
|
ProposalIndex: c.ProposalIndex,
|
||||||
|
Approved: true,
|
||||||
|
ReasonCode: frameworkvalidators.ReasonApproved,
|
||||||
|
Message: "approved",
|
||||||
|
})
|
||||||
|
}
|
||||||
|
return frameworkvalidators.Result{ValidatorName: Key, Decisions: decisions}, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
return frameworkvalidators.ValidateProtectedTerms(req, v.glossaryStage)
|
||||||
|
}
|
||||||
164
internal/validators/protected_terms/validator_test.go
Normal file
164
internal/validators/protected_terms/validator_test.go
Normal file
@@ -0,0 +1,164 @@
|
|||||||
|
package protected_terms
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"testing"
|
||||||
|
|
||||||
|
"gitea.maximumdirect.net/eric/audita/internal/core/schema"
|
||||||
|
"gitea.maximumdirect.net/eric/audita/internal/framework/contracts"
|
||||||
|
"gitea.maximumdirect.net/eric/audita/internal/framework/proposals"
|
||||||
|
frameworkvalidators "gitea.maximumdirect.net/eric/audita/internal/framework/validators"
|
||||||
|
validatormetadata "gitea.maximumdirect.net/eric/audita/internal/validators/metadata"
|
||||||
|
)
|
||||||
|
|
||||||
|
func proposal(index int, moduleKey, original, corrected string) proposals.EnrichedCorrectionProposal {
|
||||||
|
return proposals.EnrichedCorrectionProposal{
|
||||||
|
CorrectionProposal: proposals.CorrectionProposal{
|
||||||
|
TargetSegmentID: 1,
|
||||||
|
OriginalText: original,
|
||||||
|
CorrectedText: corrected,
|
||||||
|
Confidence: 0.95,
|
||||||
|
},
|
||||||
|
ProposalMetadata: proposals.ProposalMetadata{
|
||||||
|
ProposalIndex: index,
|
||||||
|
ModuleKey: moduleKey,
|
||||||
|
ModuleInstance: moduleKey,
|
||||||
|
},
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func tinyGlossary() *schema.Glossary {
|
||||||
|
return &schema.Glossary{
|
||||||
|
Entries: []schema.GlossaryEntry{
|
||||||
|
{Name: "Jesters", Category: "faction", Summary: "Faction"},
|
||||||
|
},
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestNewReportsStableKeyAndDeterministicClass(t *testing.T) {
|
||||||
|
v, err := New()
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("New: %v", err)
|
||||||
|
}
|
||||||
|
if v.Name() != Key {
|
||||||
|
t.Fatalf("expected key %q, got %q", Key, v.Name())
|
||||||
|
}
|
||||||
|
if got := validatormetadata.ClassOf(v); got != validatormetadata.ExecutionClassDeterministic {
|
||||||
|
t.Fatalf("expected deterministic class, got %q", got)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestNewUsesGeneralBehaviorForGlossaryModule(t *testing.T) {
|
||||||
|
v, err := New()
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("New: %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
req := contracts.ValidationRequest{
|
||||||
|
ModuleKey: "glossary",
|
||||||
|
Glossary: tinyGlossary(),
|
||||||
|
CandidateProposal: []proposals.EnrichedCorrectionProposal{
|
||||||
|
proposal(0, "glossary", "Jesters", "Gestures"),
|
||||||
|
},
|
||||||
|
}
|
||||||
|
result, err := v.Validate(context.Background(), req)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("Validate: %v", err)
|
||||||
|
}
|
||||||
|
if len(result.Decisions) != 1 {
|
||||||
|
t.Fatalf("expected one decision, got %+v", result.Decisions)
|
||||||
|
}
|
||||||
|
if !result.Decisions[0].Approved {
|
||||||
|
t.Fatalf("expected glossary-module approval in general variant, got %+v", result.Decisions[0])
|
||||||
|
}
|
||||||
|
if result.Decisions[0].ReasonCode != frameworkvalidators.ReasonApproved {
|
||||||
|
t.Fatalf("expected reason %q, got %+v", frameworkvalidators.ReasonApproved, result.Decisions[0])
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestNewRejectsProtectedTermChangeOutsideGlossaryStage(t *testing.T) {
|
||||||
|
v, err := New()
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("New: %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
req := contracts.ValidationRequest{
|
||||||
|
ModuleKey: "homophones",
|
||||||
|
Glossary: tinyGlossary(),
|
||||||
|
CandidateProposal: []proposals.EnrichedCorrectionProposal{
|
||||||
|
proposal(0, "homophones", "Jesters", "Gestures"),
|
||||||
|
},
|
||||||
|
}
|
||||||
|
result, err := v.Validate(context.Background(), req)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("Validate: %v", err)
|
||||||
|
}
|
||||||
|
if len(result.Decisions) != 1 {
|
||||||
|
t.Fatalf("expected one decision, got %+v", result.Decisions)
|
||||||
|
}
|
||||||
|
if result.Decisions[0].Approved {
|
||||||
|
t.Fatalf("expected protected_terms rejection, got %+v", result.Decisions[0])
|
||||||
|
}
|
||||||
|
if result.Decisions[0].ReasonCode != frameworkvalidators.ReasonProtectedGlossaryTerm {
|
||||||
|
t.Fatalf("expected reason %q, got %+v", frameworkvalidators.ReasonProtectedGlossaryTerm, result.Decisions[0])
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestNewGlossaryStageRejectsProtectedTermChange(t *testing.T) {
|
||||||
|
v, err := NewGlossaryStage()
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("NewGlossaryStage: %v", err)
|
||||||
|
}
|
||||||
|
if v.Name() != Key {
|
||||||
|
t.Fatalf("expected key %q, got %q", Key, v.Name())
|
||||||
|
}
|
||||||
|
if got := validatormetadata.ClassOf(v); got != validatormetadata.ExecutionClassDeterministic {
|
||||||
|
t.Fatalf("expected deterministic class, got %q", got)
|
||||||
|
}
|
||||||
|
|
||||||
|
req := contracts.ValidationRequest{
|
||||||
|
ModuleKey: "glossary",
|
||||||
|
Glossary: tinyGlossary(),
|
||||||
|
CandidateProposal: []proposals.EnrichedCorrectionProposal{
|
||||||
|
proposal(0, "glossary", "Jesters", "Gestures"),
|
||||||
|
},
|
||||||
|
}
|
||||||
|
result, err := v.Validate(context.Background(), req)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("Validate: %v", err)
|
||||||
|
}
|
||||||
|
if len(result.Decisions) != 1 {
|
||||||
|
t.Fatalf("expected one decision, got %+v", result.Decisions)
|
||||||
|
}
|
||||||
|
if result.Decisions[0].Approved {
|
||||||
|
t.Fatalf("expected glossary-stage protected_terms rejection, got %+v", result.Decisions[0])
|
||||||
|
}
|
||||||
|
if result.Decisions[0].ReasonCode != frameworkvalidators.ReasonProtectedGlossaryTerm {
|
||||||
|
t.Fatalf("expected reason %q, got %+v", frameworkvalidators.ReasonProtectedGlossaryTerm, result.Decisions[0])
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestNewGlossaryStageAllowsProtectedTermInsertion(t *testing.T) {
|
||||||
|
v, err := NewGlossaryStage()
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("NewGlossaryStage: %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
req := contracts.ValidationRequest{
|
||||||
|
ModuleKey: "glossary",
|
||||||
|
Glossary: tinyGlossary(),
|
||||||
|
CandidateProposal: []proposals.EnrichedCorrectionProposal{
|
||||||
|
proposal(0, "glossary", "gestures", "Jesters"),
|
||||||
|
},
|
||||||
|
}
|
||||||
|
result, err := v.Validate(context.Background(), req)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("Validate: %v", err)
|
||||||
|
}
|
||||||
|
if len(result.Decisions) != 1 {
|
||||||
|
t.Fatalf("expected one decision, got %+v", result.Decisions)
|
||||||
|
}
|
||||||
|
if !result.Decisions[0].Approved {
|
||||||
|
t.Fatalf("expected glossary-stage approval for protected-term insertion, got %+v", result.Decisions[0])
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -5,7 +5,16 @@ import (
|
|||||||
"strings"
|
"strings"
|
||||||
|
|
||||||
"gitea.maximumdirect.net/eric/audita/internal/framework/contracts"
|
"gitea.maximumdirect.net/eric/audita/internal/framework/contracts"
|
||||||
frameworkvalidators "gitea.maximumdirect.net/eric/audita/internal/framework/validators"
|
"gitea.maximumdirect.net/eric/audita/internal/validators/confidence_threshold"
|
||||||
|
"gitea.maximumdirect.net/eric/audita/internal/validators/editorial_review"
|
||||||
|
"gitea.maximumdirect.net/eric/audita/internal/validators/grammar_review"
|
||||||
|
"gitea.maximumdirect.net/eric/audita/internal/validators/meaning_reversal_review"
|
||||||
|
"gitea.maximumdirect.net/eric/audita/internal/validators/no_effect"
|
||||||
|
"gitea.maximumdirect.net/eric/audita/internal/validators/non_empty_corrected_text"
|
||||||
|
"gitea.maximumdirect.net/eric/audita/internal/validators/original_text_presence"
|
||||||
|
"gitea.maximumdirect.net/eric/audita/internal/validators/protected_terms"
|
||||||
|
"gitea.maximumdirect.net/eric/audita/internal/validators/spoken_form_plausibility"
|
||||||
|
"gitea.maximumdirect.net/eric/audita/internal/validators/spoken_word_review"
|
||||||
)
|
)
|
||||||
|
|
||||||
const (
|
const (
|
||||||
@@ -34,26 +43,16 @@ type Registry struct {
|
|||||||
|
|
||||||
func NewBuiltInRegistry() *Registry {
|
func NewBuiltInRegistry() *Registry {
|
||||||
defs := []BuiltInValidatorDefinition{
|
defs := []BuiltInValidatorDefinition{
|
||||||
{Key: KeyConfidenceThreshold, Build: func() (contracts.Validator, error) { return frameworkvalidators.ConfidenceThresholdValidator{}, nil }},
|
{Key: KeyConfidenceThreshold, Build: confidence_threshold.New},
|
||||||
{Key: KeyOriginalTextPresence, Build: func() (contracts.Validator, error) { return frameworkvalidators.OriginalTextPresenceValidator{}, nil }},
|
{Key: KeyOriginalTextPresence, Build: original_text_presence.New},
|
||||||
{Key: KeyNonEmptyCorrectedText, Build: func() (contracts.Validator, error) { return frameworkvalidators.NonEmptyCorrectionValidator{}, nil }},
|
{Key: KeyNonEmptyCorrectedText, Build: non_empty_corrected_text.New},
|
||||||
{Key: KeyNoEffect, Build: func() (contracts.Validator, error) { return frameworkvalidators.NoEffectValidator{}, nil }},
|
{Key: KeyNoEffect, Build: no_effect.New},
|
||||||
{Key: KeyProtectedTerms, Build: func() (contracts.Validator, error) { return frameworkvalidators.ProtectedGlossaryTermValidator{}, nil }},
|
{Key: KeyProtectedTerms, Build: protected_terms.New},
|
||||||
{Key: KeySpokenFormPlausibility, LLMBacked: true, Build: func() (contracts.Validator, error) {
|
{Key: KeySpokenFormPlausibility, LLMBacked: true, Build: spoken_form_plausibility.New},
|
||||||
return frameworkvalidators.NewLLMBackedValidator(KeySpokenFormPlausibility, frameworkvalidators.LLMValidatorTypeSpokenFormPlausibility, "")
|
{Key: KeyMeaningReversalReview, LLMBacked: true, Build: meaning_reversal_review.New},
|
||||||
}},
|
{Key: KeyEditorialReview, LLMBacked: true, Build: editorial_review.New},
|
||||||
{Key: KeyMeaningReversalReview, LLMBacked: true, Build: func() (contracts.Validator, error) {
|
{Key: KeyGrammarReview, LLMBacked: true, Build: grammar_review.New},
|
||||||
return frameworkvalidators.NewLLMBackedValidator(KeyMeaningReversalReview, frameworkvalidators.LLMValidatorTypeMeaningReversal, "")
|
{Key: KeySpokenWordReview, LLMBacked: true, Build: spoken_word_review.New},
|
||||||
}},
|
|
||||||
{Key: KeyEditorialReview, LLMBacked: true, Build: func() (contracts.Validator, error) {
|
|
||||||
return frameworkvalidators.NewLLMBackedValidator(KeyEditorialReview, frameworkvalidators.LLMValidatorTypeEditorialReview, "")
|
|
||||||
}},
|
|
||||||
{Key: KeyGrammarReview, LLMBacked: true, Build: func() (contracts.Validator, error) {
|
|
||||||
return frameworkvalidators.NewLLMBackedValidator(KeyGrammarReview, frameworkvalidators.LLMValidatorTypeGrammarReview, "")
|
|
||||||
}},
|
|
||||||
{Key: KeySpokenWordReview, LLMBacked: true, Build: func() (contracts.Validator, error) {
|
|
||||||
return frameworkvalidators.NewLLMBackedValidator(KeySpokenWordReview, frameworkvalidators.LLMValidatorTypeSpokenWordReview, "")
|
|
||||||
}},
|
|
||||||
}
|
}
|
||||||
|
|
||||||
m := make(map[string]BuiltInValidatorDefinition, len(defs))
|
m := make(map[string]BuiltInValidatorDefinition, len(defs))
|
||||||
|
|||||||
@@ -1,7 +1,23 @@
|
|||||||
package validators
|
package validators
|
||||||
|
|
||||||
import (
|
import (
|
||||||
|
"context"
|
||||||
"testing"
|
"testing"
|
||||||
|
|
||||||
|
"gitea.maximumdirect.net/eric/audita/internal/core/schema"
|
||||||
|
"gitea.maximumdirect.net/eric/audita/internal/framework/contracts"
|
||||||
|
"gitea.maximumdirect.net/eric/audita/internal/framework/proposals"
|
||||||
|
"gitea.maximumdirect.net/eric/audita/internal/validators/confidence_threshold"
|
||||||
|
"gitea.maximumdirect.net/eric/audita/internal/validators/editorial_review"
|
||||||
|
"gitea.maximumdirect.net/eric/audita/internal/validators/grammar_review"
|
||||||
|
"gitea.maximumdirect.net/eric/audita/internal/validators/meaning_reversal_review"
|
||||||
|
validatormetadata "gitea.maximumdirect.net/eric/audita/internal/validators/metadata"
|
||||||
|
"gitea.maximumdirect.net/eric/audita/internal/validators/no_effect"
|
||||||
|
"gitea.maximumdirect.net/eric/audita/internal/validators/non_empty_corrected_text"
|
||||||
|
"gitea.maximumdirect.net/eric/audita/internal/validators/original_text_presence"
|
||||||
|
"gitea.maximumdirect.net/eric/audita/internal/validators/protected_terms"
|
||||||
|
"gitea.maximumdirect.net/eric/audita/internal/validators/spoken_form_plausibility"
|
||||||
|
"gitea.maximumdirect.net/eric/audita/internal/validators/spoken_word_review"
|
||||||
)
|
)
|
||||||
|
|
||||||
func TestBuiltInRegistryRegistersAllKeys(t *testing.T) {
|
func TestBuiltInRegistryRegistersAllKeys(t *testing.T) {
|
||||||
@@ -32,6 +48,155 @@ func TestBuiltInRegistryRegistersAllKeys(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func TestBuiltInValidatorPackagesConstruct(t *testing.T) {
|
||||||
|
type validatorCtor struct {
|
||||||
|
name string
|
||||||
|
key string
|
||||||
|
build func() (contracts.Validator, error)
|
||||||
|
wantClass validatormetadata.ExecutionClass
|
||||||
|
}
|
||||||
|
cases := []validatorCtor{
|
||||||
|
{name: "confidence_threshold", key: KeyConfidenceThreshold, build: confidence_threshold.New, wantClass: validatormetadata.ExecutionClassDeterministic},
|
||||||
|
{name: "original_text_presence", key: KeyOriginalTextPresence, build: original_text_presence.New, wantClass: validatormetadata.ExecutionClassDeterministic},
|
||||||
|
{name: "non_empty_corrected_text", key: KeyNonEmptyCorrectedText, build: non_empty_corrected_text.New, wantClass: validatormetadata.ExecutionClassDeterministic},
|
||||||
|
{name: "no_effect", key: KeyNoEffect, build: no_effect.New, wantClass: validatormetadata.ExecutionClassDeterministic},
|
||||||
|
{name: "protected_terms", key: KeyProtectedTerms, build: protected_terms.New, wantClass: validatormetadata.ExecutionClassDeterministic},
|
||||||
|
{name: "spoken_form_plausibility", key: KeySpokenFormPlausibility, build: spoken_form_plausibility.New, wantClass: validatormetadata.ExecutionClassLLMBacked},
|
||||||
|
{name: "meaning_reversal_review", key: KeyMeaningReversalReview, build: meaning_reversal_review.New, wantClass: validatormetadata.ExecutionClassLLMBacked},
|
||||||
|
{name: "editorial_review", key: KeyEditorialReview, build: editorial_review.New, wantClass: validatormetadata.ExecutionClassLLMBacked},
|
||||||
|
{name: "grammar_review", key: KeyGrammarReview, build: grammar_review.New, wantClass: validatormetadata.ExecutionClassLLMBacked},
|
||||||
|
{name: "spoken_word_review", key: KeySpokenWordReview, build: spoken_word_review.New, wantClass: validatormetadata.ExecutionClassLLMBacked},
|
||||||
|
}
|
||||||
|
|
||||||
|
for _, tc := range cases {
|
||||||
|
v, err := tc.build()
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("%s: build: %v", tc.name, err)
|
||||||
|
}
|
||||||
|
if v.Name() != tc.key {
|
||||||
|
t.Fatalf("%s: expected key %q, got %q", tc.name, tc.key, v.Name())
|
||||||
|
}
|
||||||
|
if got := validatormetadata.ClassOf(v); got != tc.wantClass {
|
||||||
|
t.Fatalf("%s: expected class %q, got %q", tc.name, tc.wantClass, got)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestRegistryBuildsClassifiedValidators(t *testing.T) {
|
||||||
|
r := NewBuiltInRegistry()
|
||||||
|
llmKeys := map[string]bool{
|
||||||
|
KeySpokenFormPlausibility: true,
|
||||||
|
KeyMeaningReversalReview: true,
|
||||||
|
KeyEditorialReview: true,
|
||||||
|
KeyGrammarReview: true,
|
||||||
|
KeySpokenWordReview: true,
|
||||||
|
}
|
||||||
|
for _, key := range r.RegisteredKeys() {
|
||||||
|
v, err := r.MustBuild(key)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("must build %q: %v", key, err)
|
||||||
|
}
|
||||||
|
if _, ok := v.(validatormetadata.ClassifiedValidator); !ok {
|
||||||
|
t.Fatalf("expected built validator %q to expose execution classification metadata", key)
|
||||||
|
}
|
||||||
|
want := validatormetadata.ExecutionClassDeterministic
|
||||||
|
if llmKeys[key] {
|
||||||
|
want = validatormetadata.ExecutionClassLLMBacked
|
||||||
|
}
|
||||||
|
if got := validatormetadata.ClassOf(v); got != want {
|
||||||
|
t.Fatalf("expected class %q for %q, got %q", want, key, got)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestRegistryProtectedTermsUsesNonGlossaryStageBehavior(t *testing.T) {
|
||||||
|
r := NewBuiltInRegistry()
|
||||||
|
v, err := r.MustBuild(KeyProtectedTerms)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("must build protected_terms: %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
req := contracts.ValidationRequest{
|
||||||
|
ModuleKey: "glossary",
|
||||||
|
Glossary: &schema.Glossary{
|
||||||
|
Entries: []schema.GlossaryEntry{{Name: "Jesters", Category: "faction", Summary: "Protected"}},
|
||||||
|
},
|
||||||
|
CandidateProposal: []proposals.EnrichedCorrectionProposal{
|
||||||
|
{
|
||||||
|
CorrectionProposal: proposals.CorrectionProposal{
|
||||||
|
TargetSegmentID: 1,
|
||||||
|
OriginalText: "Jesters",
|
||||||
|
CorrectedText: "Gestures",
|
||||||
|
Confidence: 1.0,
|
||||||
|
},
|
||||||
|
ProposalMetadata: proposals.ProposalMetadata{
|
||||||
|
ProposalIndex: 0,
|
||||||
|
ModuleKey: "glossary",
|
||||||
|
},
|
||||||
|
},
|
||||||
|
},
|
||||||
|
}
|
||||||
|
result, err := v.Validate(context.Background(), req)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("validate protected_terms: %v", err)
|
||||||
|
}
|
||||||
|
if len(result.Decisions) != 1 {
|
||||||
|
t.Fatalf("expected one decision, got %+v", result.Decisions)
|
||||||
|
}
|
||||||
|
if !result.Decisions[0].Approved {
|
||||||
|
t.Fatalf("expected glossary module proposal to be approved by non-glossary-stage protected_terms validator, got %+v", result.Decisions[0])
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestResolveBuiltInChainUsesGlossaryStageProtectedTermsBehavior(t *testing.T) {
|
||||||
|
chain, err := ResolveBuiltInChain("glossary", NewBuiltInRegistry())
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("resolve glossary chain: %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
var v contracts.Validator
|
||||||
|
for _, candidate := range chain {
|
||||||
|
if candidate.Name() == KeyProtectedTerms {
|
||||||
|
v = candidate
|
||||||
|
break
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if v == nil {
|
||||||
|
t.Fatalf("expected protected_terms validator in glossary chain")
|
||||||
|
}
|
||||||
|
|
||||||
|
req := contracts.ValidationRequest{
|
||||||
|
ModuleKey: "glossary",
|
||||||
|
Glossary: &schema.Glossary{
|
||||||
|
Entries: []schema.GlossaryEntry{{Name: "Jesters", Category: "faction", Summary: "Protected"}},
|
||||||
|
},
|
||||||
|
CandidateProposal: []proposals.EnrichedCorrectionProposal{
|
||||||
|
{
|
||||||
|
CorrectionProposal: proposals.CorrectionProposal{
|
||||||
|
TargetSegmentID: 1,
|
||||||
|
OriginalText: "Jesters",
|
||||||
|
CorrectedText: "Gestures",
|
||||||
|
Confidence: 1.0,
|
||||||
|
},
|
||||||
|
ProposalMetadata: proposals.ProposalMetadata{
|
||||||
|
ProposalIndex: 0,
|
||||||
|
ModuleKey: "glossary",
|
||||||
|
},
|
||||||
|
},
|
||||||
|
},
|
||||||
|
}
|
||||||
|
result, err := v.Validate(context.Background(), req)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("validate protected_terms: %v", err)
|
||||||
|
}
|
||||||
|
if len(result.Decisions) != 1 {
|
||||||
|
t.Fatalf("expected one decision, got %+v", result.Decisions)
|
||||||
|
}
|
||||||
|
if result.Decisions[0].Approved {
|
||||||
|
t.Fatalf("expected glossary-stage protected_terms rejection, got %+v", result.Decisions[0])
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
func TestBuiltInRegistryUnknownKeyFails(t *testing.T) {
|
func TestBuiltInRegistryUnknownKeyFails(t *testing.T) {
|
||||||
r := NewBuiltInRegistry()
|
r := NewBuiltInRegistry()
|
||||||
if _, err := r.MustBuild("missing"); err == nil {
|
if _, err := r.MustBuild("missing"); err == nil {
|
||||||
|
|||||||
17
internal/validators/spoken_form_plausibility/validator.go
Normal file
17
internal/validators/spoken_form_plausibility/validator.go
Normal file
@@ -0,0 +1,17 @@
|
|||||||
|
package spoken_form_plausibility
|
||||||
|
|
||||||
|
import (
|
||||||
|
"gitea.maximumdirect.net/eric/audita/internal/framework/contracts"
|
||||||
|
frameworkvalidators "gitea.maximumdirect.net/eric/audita/internal/framework/validators"
|
||||||
|
validatormetadata "gitea.maximumdirect.net/eric/audita/internal/validators/metadata"
|
||||||
|
)
|
||||||
|
|
||||||
|
const Key = "spoken_form_plausibility"
|
||||||
|
|
||||||
|
func New() (contracts.Validator, error) {
|
||||||
|
v, err := frameworkvalidators.NewLLMBackedValidator(Key, frameworkvalidators.LLMValidatorTypeSpokenFormPlausibility, "")
|
||||||
|
if err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
return validatormetadata.Wrap(v, validatormetadata.ExecutionClassLLMBacked), nil
|
||||||
|
}
|
||||||
17
internal/validators/spoken_word_review/validator.go
Normal file
17
internal/validators/spoken_word_review/validator.go
Normal file
@@ -0,0 +1,17 @@
|
|||||||
|
package spoken_word_review
|
||||||
|
|
||||||
|
import (
|
||||||
|
"gitea.maximumdirect.net/eric/audita/internal/framework/contracts"
|
||||||
|
frameworkvalidators "gitea.maximumdirect.net/eric/audita/internal/framework/validators"
|
||||||
|
validatormetadata "gitea.maximumdirect.net/eric/audita/internal/validators/metadata"
|
||||||
|
)
|
||||||
|
|
||||||
|
const Key = "spoken_word_review"
|
||||||
|
|
||||||
|
func New() (contracts.Validator, error) {
|
||||||
|
v, err := frameworkvalidators.NewLLMBackedValidator(Key, frameworkvalidators.LLMValidatorTypeSpokenWordReview, "")
|
||||||
|
if err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
return validatormetadata.Wrap(v, validatormetadata.ExecutionClassLLMBacked), nil
|
||||||
|
}
|
||||||
Reference in New Issue
Block a user