eric/audita
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 2 Wiki Activity

Refactor validators into package-owned components

Browse Source
This commit is contained in:
Eric Rakestraw
2026-05-14 00:28:28 +00:00
parent 3b160cf05b
commit 52ffe42e73
26 changed files with 775 additions and 98 deletions
Download Patch File Download Diff File Expand all files Collapse all files

44
docs/architecture.md
View File

@@ -134,8 +134,30 @@ internal/framework/validators/
llm_validators.go
internal/validators/
metadata/
metadata.go
registry.go
chains.go
confidence_threshold/
validator.go
original_text_presence/
validator.go
non_empty_corrected_text/
validator.go
no_effect/
validator.go
protected_terms/
validator.go
spoken_form_plausibility/
validator.go
meaning_reversal_review/
validator.go
editorial_review/
validator.go
grammar_review/
validator.go
spoken_word_review/
validator.go
internal/prompts/
registry.go
@@ -195,7 +217,7 @@ Current runtime flow (`internal/cli/run.go`):
- test/injected module factory path remains available for deterministic runtime tests.
- each module recomputes chunks from the current working transcript, runs chunk proposal work concurrently, aggregates deterministically, validates, and applies approved proposals once.
13. Output working transcript to `--output` file or stdout.
14. Build process report (`phase` currently set to `default_pipeline`).
14. Build process report metadata.
15. Optionally write `--report-json`; always write run-dir `report.json`.
16. Apply work-dir retention.
@@ -495,6 +517,26 @@ Validator composition is now explicit and registry-backed through `internal/vali
- built-in chain definitions per production module key;
- production modules resolve validator chains from those built-in definitions.
Package ownership boundary:
- `internal/validators/<validator_key>` owns built-in validator construction and stable key identity.
- `internal/framework/validators` remains shared runtime machinery:
- request/result models;
- decision cardinality enforcement;
- protected-vocabulary helpers;
- generic LLM-backed validator runtime, batching, and diagnostics glue.
Validator execution classification metadata:
- `internal/validators/metadata` defines execution class markers:
- `deterministic`
- `llm_backed`
- runner ordering uses this metadata interface rather than concrete framework validator type assertions.
- validators without classification metadata default to deterministic ordering.
`protected_terms` construction ownership:
- `internal/validators/protected_terms.New()` builds the general (non-glossary-stage) variant.
- `internal/validators/protected_terms.NewGlossaryStage()` builds the glossary-stage variant used by glossary chains.
- both variants preserve the stable key `protected_terms`.
Stable built-in validator keys:
- deterministic:
- `confidence_threshold`

40
docs/validators.md
View File

@@ -4,6 +4,36 @@ This document describes Audita's built-in validator registry and module validato
For LLM-backed validator prompt asset details, see [`docs/prompts.md`](prompts.md).
## Package ownership
Built-in validator construction is package-owned under `internal/validators/<validator_key>`:
- `internal/validators/confidence_threshold`
- `internal/validators/original_text_presence`
- `internal/validators/non_empty_corrected_text`
- `internal/validators/no_effect`
- `internal/validators/protected_terms`
- `internal/validators/spoken_form_plausibility`
- `internal/validators/meaning_reversal_review`
- `internal/validators/editorial_review`
- `internal/validators/grammar_review`
- `internal/validators/spoken_word_review`
Registry and chain wiring stay in:
- `internal/validators/registry.go`
- `internal/validators/chains.go`
Shared validator runtime mechanics stay in `internal/framework/validators`:
- request/result/decision models
- decision cardinality helpers
- protected vocabulary helpers
- shared LLM validator runtime, batching, and diagnostics helpers
Execution classification metadata is defined in `internal/validators/metadata`:
- `deterministic`
- `llm_backed`
Runner ordering uses this metadata so deterministic validators run before LLM-backed validators without concrete framework type assertions.
## Scope
Validator chains are built-in runtime behavior.
@@ -82,6 +112,14 @@ Current built-in chains resolved from `internal/validators/chains.go`:
- `grammar_review`
- `meaning_reversal_review`
## Protected terms construction
`protected_terms` has explicit constructors:
- general constructor used by non-glossary modules through the built-in registry
- glossary-stage constructor used by glossary chain resolution
Both variants preserve existing behavior and report the stable key `protected_terms`.
## Execution semantics
- modules execute serially;
@@ -105,6 +143,8 @@ These are separate outcomes and are reported separately.
- validator LLM diagnostics include validator identity in interaction metadata and structured response schema metadata.
- correction ledger entries include deterministic and LLM validator decision snapshots keyed by the same stable validator keys, and keep validator rejection distinct from application-level skip.
Prompt assets are unchanged by the validator package-ownership refactor and remain built-in under `internal/prompts`.
## Configurable knobs that remain supported
- per-module confidence thresholds (`thresholds.*` / equivalent env+CLI overrides)

2
internal/cli/run.go
View File

@@ -43,7 +43,7 @@ func (c noOpStructuredLLMClient) CompleteStructured(ctx context.Context, req con
}
func shouldUseNoOpLLMClientForTests() bool {
return strings.HasSuffix(filepath.Base(os.Args[0]), ".test")
return strings.HasSuffix(filepath.Base(os.Args[0]), ".test") || os.Getenv("GO_WANT_HELPER_PROCESS") == "1"
}
type processInvocation struct {

3
internal/cli/subprocess_test_hooks.go
View File

@@ -28,9 +28,6 @@ func ConfigureSubprocessTestHooksFromEnv() {
if mode == "" && timeoutMSRaw == "" {
return
}
if !shouldUseNoOpLLMClientForTests() {
return
}
if mode != "" {
client := &subprocessTestLLMClient{mode: mode}

8
internal/framework/runner/observability.go
View File

@@ -8,7 +8,7 @@ import (
"time"
"gitea.maximumdirect.net/eric/audita/internal/framework/contracts"
frameworkvalidators "gitea.maximumdirect.net/eric/audita/internal/framework/validators"
validatormetadata "gitea.maximumdirect.net/eric/audita/internal/validators/metadata"
)
type UtilizationDiagnostics struct {
@@ -337,11 +337,7 @@ func withModuleInstanceContext(ctx context.Context, moduleInstance string) conte
}
func isLLMBackedValidator(v contracts.Validator) bool {
if v == nil {
return false
}
_, ok := v.(*frameworkvalidators.LLMBackedValidator)
return ok
return validatormetadata.ClassOf(v) == validatormetadata.ExecutionClassLLMBacked
}
func sortValidatorSummaries(in []ValidatorTimingSummary) {

3
internal/framework/runner/runner.go
View File

@@ -15,6 +15,7 @@ import (
"gitea.maximumdirect.net/eric/audita/internal/framework/llm"
"gitea.maximumdirect.net/eric/audita/internal/framework/proposals"
"gitea.maximumdirect.net/eric/audita/internal/framework/validators"
validatormetadata "gitea.maximumdirect.net/eric/audita/internal/validators/metadata"
)
const (
@@ -568,7 +569,7 @@ func reorderValidatorsForPipeline(in []contracts.Validator) ([]contracts.Validat
deterministic := make([]contracts.Validator, 0, len(in))
llmBacked := make([]contracts.Validator, 0, len(in))
for _, validator := range in {
if _, ok := validator.(*validators.LLMBackedValidator); ok {
if validatormetadata.ClassOf(validator) == validatormetadata.ExecutionClassLLMBacked {
llmBacked = append(llmBacked, validator)
continue
}

53
internal/framework/runner/runner_test.go
View File

@@ -22,6 +22,7 @@ import (
"gitea.maximumdirect.net/eric/audita/internal/framework/proposals"
"gitea.maximumdirect.net/eric/audita/internal/framework/responseschema"
"gitea.maximumdirect.net/eric/audita/internal/framework/validators"
validatormetadata "gitea.maximumdirect.net/eric/audita/internal/validators/metadata"
)
type fakeFactory struct {
@@ -64,6 +65,58 @@ func (v fakeValidator) Validate(ctx context.Context, req contracts.ValidationReq
return v.validateF(req)
}
type classifiedFakeValidator struct {
fakeValidator
class validatormetadata.ExecutionClass
}
func (v classifiedFakeValidator) ExecutionClass() validatormetadata.ExecutionClass {
return v.class
}
func TestReorderValidatorsDeterministicBeforeLLMBacked(t *testing.T) {
llm := classifiedFakeValidator{
fakeValidator: fakeValidator{name: "llm", validateF: func(req contracts.ValidationRequest) (validators.Result, error) {
return validators.Result{ValidatorName: "llm"}, nil
}},
class: validatormetadata.ExecutionClassLLMBacked,
}
deterministic := classifiedFakeValidator{
fakeValidator: fakeValidator{name: "deterministic", validateF: func(req contracts.ValidationRequest) (validators.Result, error) {
return validators.Result{ValidatorName: "deterministic"}, nil
}},
class: validatormetadata.ExecutionClassDeterministic,
}
ordered, _ := reorderValidatorsForPipeline([]contracts.Validator{llm, deterministic})
if len(ordered) != 2 {
t.Fatalf("expected 2 validators, got %d", len(ordered))
}
if ordered[0].Name() != "deterministic" || ordered[1].Name() != "llm" {
t.Fatalf("unexpected validator order: %s, %s", ordered[0].Name(), ordered[1].Name())
}
}
func TestReorderValidatorsDefaultsUnclassifiedToDeterministic(t *testing.T) {
unclassified := fakeValidator{name: "plain", validateF: func(req contracts.ValidationRequest) (validators.Result, error) {
return validators.Result{ValidatorName: "plain"}, nil
}}
llm := classifiedFakeValidator{
fakeValidator: fakeValidator{name: "llm", validateF: func(req contracts.ValidationRequest) (validators.Result, error) {
return validators.Result{ValidatorName: "llm"}, nil
}},
class: validatormetadata.ExecutionClassLLMBacked,
}
ordered, _ := reorderValidatorsForPipeline([]contracts.Validator{llm, unclassified})
if len(ordered) != 2 {
t.Fatalf("expected 2 validators, got %d", len(ordered))
}
if ordered[0].Name() != "plain" || ordered[1].Name() != "llm" {
t.Fatalf("unexpected validator order with unclassified validator: %s, %s", ordered[0].Name(), ordered[1].Name())
}
}
func TestRunnerOneModuleAppliesProposal(t *testing.T) {
transcript := &schema.Transcript{Segments: []schema.Segment{{ID: 1, Speaker: "A", Start: 0, End: 1, Text: "teh cat"}}}
r := New(fakeFactory{modules: map[string]contracts.TranscriptModule{

46
internal/framework/validators/deterministic.go
View File

@@ -95,23 +95,16 @@ func (v NoEffectValidator) Validate(_ context.Context, req Request) (Result, err
return Result{ValidatorName: v.Name(), Decisions: decisions}, nil
}
type ProtectedGlossaryTermValidator struct{}
func (v ProtectedGlossaryTermValidator) Name() string { return "protected_terms" }
func (v ProtectedGlossaryTermValidator) Validate(_ context.Context, req Request) (Result, error) {
if req.ModuleKey == "glossary" {
decisions := make([]Decision, 0, len(req.CandidateProposal))
for _, c := range req.CandidateProposal {
decisions = append(decisions, approval(c.ProposalIndex))
}
return Result{ValidatorName: v.Name(), Decisions: decisions}, nil
}
func ValidateProtectedTerms(req Request, glossaryStage bool) (Result, error) {
vocab := NewProtectedVocabulary(req.Glossary)
decisions := make([]Decision, 0, len(req.CandidateProposal))
for _, c := range req.CandidateProposal {
reason := vocab.violationReason(c.OriginalText, c.CorrectedText)
var reason string
if glossaryStage {
reason = vocab.glossaryStageViolationReason(c.OriginalText, c.CorrectedText)
} else {
reason = vocab.violationReason(c.OriginalText, c.CorrectedText)
}
if reason != "" {
decisions = append(decisions, rejection(c.ProposalIndex, ReasonProtectedGlossaryTerm, reason))
continue
@@ -121,28 +114,5 @@ func (v ProtectedGlossaryTermValidator) Validate(_ context.Context, req Request)
if err := EnforceDecisionCardinality(req.CandidateProposal, decisions); err != nil {
return Result{}, err
}
return Result{ValidatorName: v.Name(), Decisions: decisions}, nil
}
type GlossaryStageProtectedGlossaryTermValidator struct{}
func (v GlossaryStageProtectedGlossaryTermValidator) Name() string {
return "protected_terms"
}
func (v GlossaryStageProtectedGlossaryTermValidator) Validate(_ context.Context, req Request) (Result, error) {
vocab := NewProtectedVocabulary(req.Glossary)
decisions := make([]Decision, 0, len(req.CandidateProposal))
for _, c := range req.CandidateProposal {
reason := vocab.glossaryStageViolationReason(c.OriginalText, c.CorrectedText)
if reason != "" {
decisions = append(decisions, rejection(c.ProposalIndex, ReasonProtectedGlossaryTerm, reason))
continue
}
decisions = append(decisions, approval(c.ProposalIndex))
}
if err := EnforceDecisionCardinality(req.CandidateProposal, decisions); err != nil {
return Result{}, err
}
return Result{ValidatorName: v.Name(), Decisions: decisions}, nil
return Result{ValidatorName: "protected_terms", Decisions: decisions}, nil
}

9
internal/framework/validators/protected_terms_test.go
View File

@@ -1,7 +1,6 @@
package validators
import (
"context"
"reflect"
"testing"
@@ -31,7 +30,7 @@ func TestExtractProtectedTermsEmptyGlossary(t *testing.T) {
}
}
func TestProtectedGlossaryTermValidatorAppliesToNonGlossaryModule(t *testing.T) {
func TestValidateProtectedTermsAppliesToNonGlossaryModule(t *testing.T) {
req := Request{
ModuleKey: "grammar",
Glossary: &schema.Glossary{
@@ -49,7 +48,7 @@ func TestProtectedGlossaryTermValidatorAppliesToNonGlossaryModule(t *testing.T)
},
},
}
res, err := (ProtectedGlossaryTermValidator{}).Validate(context.Background(), req)
res, err := ValidateProtectedTerms(req, false)
if err != nil {
t.Fatalf("validator error: %v", err)
}
@@ -58,7 +57,7 @@ func TestProtectedGlossaryTermValidatorAppliesToNonGlossaryModule(t *testing.T)
}
}
func TestGlossaryStageProtectedGlossaryTermValidatorAllowsProtectedTermSwap(t *testing.T) {
func TestValidateProtectedTermsGlossaryStageAllowsProtectedTermSwap(t *testing.T) {
req := Request{
ModuleKey: "glossary",
Glossary: &schema.Glossary{
@@ -76,7 +75,7 @@ func TestGlossaryStageProtectedGlossaryTermValidatorAllowsProtectedTermSwap(t *t
},
},
}
res, err := (GlossaryStageProtectedGlossaryTermValidator{}).Validate(context.Background(), req)
res, err := ValidateProtectedTerms(req, true)
if err != nil {
t.Fatalf("validator error: %v", err)
}

22
internal/framework/validators/validators_test.go
View File

@@ -123,7 +123,7 @@ func TestNoEffectValidator(t *testing.T) {
}
}
func TestProtectedGlossaryTermValidator(t *testing.T) {
func TestValidateProtectedTerms(t *testing.T) {
glossary := &schema.Glossary{Entries: []schema.GlossaryEntry{{Name: "OpenAI", Aliases: []string{"Open AI"}, Plural: "OpenAIs", Category: "brand", Summary: "brand"}}}
req := Request{
Glossary: glossary,
@@ -133,7 +133,7 @@ func TestProtectedGlossaryTermValidator(t *testing.T) {
mkCandidate(1, 1, "teh", "the", 0.9),
},
}
res, err := (ProtectedGlossaryTermValidator{}).Validate(context.Background(), req)
res, err := ValidateProtectedTerms(req, false)
if err != nil {
t.Fatalf("Validate error: %v", err)
}
@@ -145,24 +145,6 @@ func TestProtectedGlossaryTermValidator(t *testing.T) {
}
}
func TestProtectedGlossaryTermValidatorAllowsGlossaryModule(t *testing.T) {
glossary := &schema.Glossary{Entries: []schema.GlossaryEntry{{Name: "OpenAI", Category: "brand", Summary: "brand"}}}
req := Request{
Glossary: glossary,
ModuleKey: "glossary",
CandidateProposal: []proposals.EnrichedCorrectionProposal{
mkCandidate(0, 1, "OpenAI", "Open A I", 0.9),
},
}
res, err := (ProtectedGlossaryTermValidator{}).Validate(context.Background(), req)
if err != nil {
t.Fatalf("Validate error: %v", err)
}
if !res.Decisions[0].Approved {
t.Fatalf("expected glossary module approval, got %+v", res.Decisions[0])
}
}
func TestStableReasonCodes(t *testing.T) {
codes := []string{
ReasonApproved,

7
internal/validators/chains.go
View File

@@ -4,7 +4,7 @@ import (
"fmt"
"gitea.maximumdirect.net/eric/audita/internal/framework/contracts"
frameworkvalidators "gitea.maximumdirect.net/eric/audita/internal/framework/validators"
"gitea.maximumdirect.net/eric/audita/internal/validators/protected_terms"
)
var builtInChains = map[string][]string{
@@ -70,7 +70,10 @@ func ResolveBuiltInChain(moduleKey string, registry *Registry) ([]contracts.Vali
if moduleKey == "glossary" && key == KeyProtectedTerms {
// Glossary stages preserve current stricter protection semantics while
// reporting the stable protected_terms key.
v := frameworkvalidators.GlossaryStageProtectedGlossaryTermValidator{}
v, buildErr := protected_terms.NewGlossaryStage()
if buildErr != nil {
return nil, buildErr
}
out = append(out, v)
continue
}

13
internal/validators/confidence_threshold/validator.go Normal file
View File

@@ -0,0 +1,13 @@
package confidence_threshold
import (
"gitea.maximumdirect.net/eric/audita/internal/framework/contracts"
frameworkvalidators "gitea.maximumdirect.net/eric/audita/internal/framework/validators"
validatormetadata "gitea.maximumdirect.net/eric/audita/internal/validators/metadata"
)
const Key = "confidence_threshold"
func New() (contracts.Validator, error) {
return validatormetadata.Wrap(frameworkvalidators.ConfidenceThresholdValidator{}, validatormetadata.ExecutionClassDeterministic), nil
}

17
internal/validators/editorial_review/validator.go Normal file
View File

@@ -0,0 +1,17 @@
package editorial_review
import (
"gitea.maximumdirect.net/eric/audita/internal/framework/contracts"
frameworkvalidators "gitea.maximumdirect.net/eric/audita/internal/framework/validators"
validatormetadata "gitea.maximumdirect.net/eric/audita/internal/validators/metadata"
)
const Key = "editorial_review"
func New() (contracts.Validator, error) {
v, err := frameworkvalidators.NewLLMBackedValidator(Key, frameworkvalidators.LLMValidatorTypeEditorialReview, "")
if err != nil {
return nil, err
}
return validatormetadata.Wrap(v, validatormetadata.ExecutionClassLLMBacked), nil
}

17
internal/validators/grammar_review/validator.go Normal file
View File

@@ -0,0 +1,17 @@
package grammar_review
import (
"gitea.maximumdirect.net/eric/audita/internal/framework/contracts"
frameworkvalidators "gitea.maximumdirect.net/eric/audita/internal/framework/validators"
validatormetadata "gitea.maximumdirect.net/eric/audita/internal/validators/metadata"
)
const Key = "grammar_review"
func New() (contracts.Validator, error) {
v, err := frameworkvalidators.NewLLMBackedValidator(Key, frameworkvalidators.LLMValidatorTypeGrammarReview, "")
if err != nil {
return nil, err
}
return validatormetadata.Wrap(v, validatormetadata.ExecutionClassLLMBacked), nil
}

17
internal/validators/meaning_reversal_review/validator.go Normal file
View File

@@ -0,0 +1,17 @@
package meaning_reversal_review
import (
"gitea.maximumdirect.net/eric/audita/internal/framework/contracts"
frameworkvalidators "gitea.maximumdirect.net/eric/audita/internal/framework/validators"
validatormetadata "gitea.maximumdirect.net/eric/audita/internal/validators/metadata"
)
const Key = "meaning_reversal_review"
func New() (contracts.Validator, error) {
v, err := frameworkvalidators.NewLLMBackedValidator(Key, frameworkvalidators.LLMValidatorTypeMeaningReversal, "")
if err != nil {
return nil, err
}
return validatormetadata.Wrap(v, validatormetadata.ExecutionClassLLMBacked), nil
}

55
internal/validators/metadata/metadata.go Normal file
View File

@@ -0,0 +1,55 @@
package metadata
import "gitea.maximumdirect.net/eric/audita/internal/framework/contracts"
type ExecutionClass string
const (
ExecutionClassDeterministic ExecutionClass = "deterministic"
ExecutionClassLLMBacked ExecutionClass = "llm_backed"
)
type ClassifiedValidator interface {
contracts.Validator
ExecutionClass() ExecutionClass
}
func ClassOf(v contracts.Validator) ExecutionClass {
if v == nil {
return ExecutionClassDeterministic
}
classified, ok := v.(ClassifiedValidator)
if !ok {
return ExecutionClassDeterministic
}
switch classified.ExecutionClass() {
case ExecutionClassLLMBacked:
return ExecutionClassLLMBacked
case ExecutionClassDeterministic:
return ExecutionClassDeterministic
default:
return ExecutionClassDeterministic
}
}
func Wrap(v contracts.Validator, class ExecutionClass) contracts.Validator {
if v == nil {
return nil
}
if class != ExecutionClassLLMBacked {
class = ExecutionClassDeterministic
}
return wrappedValidator{
Validator: v,
class: class,
}
}
type wrappedValidator struct {
contracts.Validator
class ExecutionClass
}
func (w wrappedValidator) ExecutionClass() ExecutionClass {
return w.class
}

30
internal/validators/metadata/metadata_test.go Normal file
View File

@@ -0,0 +1,30 @@
package metadata
import (
"context"
"testing"
"gitea.maximumdirect.net/eric/audita/internal/framework/contracts"
frameworkvalidators "gitea.maximumdirect.net/eric/audita/internal/framework/validators"
)
type unclassifiedValidator struct{}
func (u unclassifiedValidator) Name() string { return "unclassified" }
func (u unclassifiedValidator) Validate(_ context.Context, _ contracts.ValidationRequest) (frameworkvalidators.Result, error) {
return frameworkvalidators.Result{ValidatorName: u.Name(), Decisions: nil}, nil
}
func TestClassOfDefaultsToDeterministic(t *testing.T) {
if got := ClassOf(unclassifiedValidator{}); got != ExecutionClassDeterministic {
t.Fatalf("expected deterministic default class, got %q", got)
}
}
func TestWrapExposesExecutionClass(t *testing.T) {
wrapped := Wrap(unclassifiedValidator{}, ExecutionClassLLMBacked)
if got := ClassOf(wrapped); got != ExecutionClassLLMBacked {
t.Fatalf("expected llm_backed class, got %q", got)
}
}

13
internal/validators/no_effect/validator.go Normal file
View File

@@ -0,0 +1,13 @@
package no_effect
import (
"gitea.maximumdirect.net/eric/audita/internal/framework/contracts"
frameworkvalidators "gitea.maximumdirect.net/eric/audita/internal/framework/validators"
validatormetadata "gitea.maximumdirect.net/eric/audita/internal/validators/metadata"
)
const Key = "no_effect"
func New() (contracts.Validator, error) {
return validatormetadata.Wrap(frameworkvalidators.NoEffectValidator{}, validatormetadata.ExecutionClassDeterministic), nil
}

13
internal/validators/non_empty_corrected_text/validator.go Normal file
View File

@@ -0,0 +1,13 @@
package non_empty_corrected_text
import (
"gitea.maximumdirect.net/eric/audita/internal/framework/contracts"
frameworkvalidators "gitea.maximumdirect.net/eric/audita/internal/framework/validators"
validatormetadata "gitea.maximumdirect.net/eric/audita/internal/validators/metadata"
)
const Key = "non_empty_corrected_text"
func New() (contracts.Validator, error) {
return validatormetadata.Wrap(frameworkvalidators.NonEmptyCorrectionValidator{}, validatormetadata.ExecutionClassDeterministic), nil
}

13
internal/validators/original_text_presence/validator.go Normal file
View File

@@ -0,0 +1,13 @@
package original_text_presence
import (
"gitea.maximumdirect.net/eric/audita/internal/framework/contracts"
frameworkvalidators "gitea.maximumdirect.net/eric/audita/internal/framework/validators"
validatormetadata "gitea.maximumdirect.net/eric/audita/internal/validators/metadata"
)
const Key = "original_text_presence"
func New() (contracts.Validator, error) {
return validatormetadata.Wrap(frameworkvalidators.OriginalTextPresenceValidator{}, validatormetadata.ExecutionClassDeterministic), nil
}

44
internal/validators/protected_terms/validator.go Normal file
View File

@@ -0,0 +1,44 @@
package protected_terms
import (
"context"
"gitea.maximumdirect.net/eric/audita/internal/framework/contracts"
frameworkvalidators "gitea.maximumdirect.net/eric/audita/internal/framework/validators"
validatormetadata "gitea.maximumdirect.net/eric/audita/internal/validators/metadata"
)
const Key = "protected_terms"
func New() (contracts.Validator, error) {
return validatormetadata.Wrap(validator{}, validatormetadata.ExecutionClassDeterministic), nil
}
func NewGlossaryStage() (contracts.Validator, error) {
return validatormetadata.Wrap(validator{glossaryStage: true}, validatormetadata.ExecutionClassDeterministic), nil
}
type validator struct {
glossaryStage bool
}
func (v validator) Name() string {
return Key
}
func (v validator) Validate(_ context.Context, req contracts.ValidationRequest) (frameworkvalidators.Result, error) {
if !v.glossaryStage && req.ModuleKey == "glossary" {
decisions := make([]frameworkvalidators.Decision, 0, len(req.CandidateProposal))
for _, c := range req.CandidateProposal {
decisions = append(decisions, frameworkvalidators.Decision{
ProposalIndex: c.ProposalIndex,
Approved: true,
ReasonCode: frameworkvalidators.ReasonApproved,
Message: "approved",
})
}
return frameworkvalidators.Result{ValidatorName: Key, Decisions: decisions}, nil
}
return frameworkvalidators.ValidateProtectedTerms(req, v.glossaryStage)
}

164
internal/validators/protected_terms/validator_test.go Normal file
View File

@@ -0,0 +1,164 @@
package protected_terms
import (
"context"
"testing"
"gitea.maximumdirect.net/eric/audita/internal/core/schema"
"gitea.maximumdirect.net/eric/audita/internal/framework/contracts"
"gitea.maximumdirect.net/eric/audita/internal/framework/proposals"
frameworkvalidators "gitea.maximumdirect.net/eric/audita/internal/framework/validators"
validatormetadata "gitea.maximumdirect.net/eric/audita/internal/validators/metadata"
)
func proposal(index int, moduleKey, original, corrected string) proposals.EnrichedCorrectionProposal {
return proposals.EnrichedCorrectionProposal{
CorrectionProposal: proposals.CorrectionProposal{
TargetSegmentID: 1,
OriginalText: original,
CorrectedText: corrected,
Confidence: 0.95,
},
ProposalMetadata: proposals.ProposalMetadata{
ProposalIndex: index,
ModuleKey: moduleKey,
ModuleInstance: moduleKey,
},
}
}
func tinyGlossary() *schema.Glossary {
return &schema.Glossary{
Entries: []schema.GlossaryEntry{
{Name: "Jesters", Category: "faction", Summary: "Faction"},
},
}
}
func TestNewReportsStableKeyAndDeterministicClass(t *testing.T) {
v, err := New()
if err != nil {
t.Fatalf("New: %v", err)
}
if v.Name() != Key {
t.Fatalf("expected key %q, got %q", Key, v.Name())
}
if got := validatormetadata.ClassOf(v); got != validatormetadata.ExecutionClassDeterministic {
t.Fatalf("expected deterministic class, got %q", got)
}
}
func TestNewUsesGeneralBehaviorForGlossaryModule(t *testing.T) {
v, err := New()
if err != nil {
t.Fatalf("New: %v", err)
}
req := contracts.ValidationRequest{
ModuleKey: "glossary",
Glossary: tinyGlossary(),
CandidateProposal: []proposals.EnrichedCorrectionProposal{
proposal(0, "glossary", "Jesters", "Gestures"),
},
}
result, err := v.Validate(context.Background(), req)
if err != nil {
t.Fatalf("Validate: %v", err)
}
if len(result.Decisions) != 1 {
t.Fatalf("expected one decision, got %+v", result.Decisions)
}
if !result.Decisions[0].Approved {
t.Fatalf("expected glossary-module approval in general variant, got %+v", result.Decisions[0])
}
if result.Decisions[0].ReasonCode != frameworkvalidators.ReasonApproved {
t.Fatalf("expected reason %q, got %+v", frameworkvalidators.ReasonApproved, result.Decisions[0])
}
}
func TestNewRejectsProtectedTermChangeOutsideGlossaryStage(t *testing.T) {
v, err := New()
if err != nil {
t.Fatalf("New: %v", err)
}
req := contracts.ValidationRequest{
ModuleKey: "homophones",
Glossary: tinyGlossary(),
CandidateProposal: []proposals.EnrichedCorrectionProposal{
proposal(0, "homophones", "Jesters", "Gestures"),
},
}
result, err := v.Validate(context.Background(), req)
if err != nil {
t.Fatalf("Validate: %v", err)
}
if len(result.Decisions) != 1 {
t.Fatalf("expected one decision, got %+v", result.Decisions)
}
if result.Decisions[0].Approved {
t.Fatalf("expected protected_terms rejection, got %+v", result.Decisions[0])
}
if result.Decisions[0].ReasonCode != frameworkvalidators.ReasonProtectedGlossaryTerm {
t.Fatalf("expected reason %q, got %+v", frameworkvalidators.ReasonProtectedGlossaryTerm, result.Decisions[0])
}
}
func TestNewGlossaryStageRejectsProtectedTermChange(t *testing.T) {
v, err := NewGlossaryStage()
if err != nil {
t.Fatalf("NewGlossaryStage: %v", err)
}
if v.Name() != Key {
t.Fatalf("expected key %q, got %q", Key, v.Name())
}
if got := validatormetadata.ClassOf(v); got != validatormetadata.ExecutionClassDeterministic {
t.Fatalf("expected deterministic class, got %q", got)
}
req := contracts.ValidationRequest{
ModuleKey: "glossary",
Glossary: tinyGlossary(),
CandidateProposal: []proposals.EnrichedCorrectionProposal{
proposal(0, "glossary", "Jesters", "Gestures"),
},
}
result, err := v.Validate(context.Background(), req)
if err != nil {
t.Fatalf("Validate: %v", err)
}
if len(result.Decisions) != 1 {
t.Fatalf("expected one decision, got %+v", result.Decisions)
}
if result.Decisions[0].Approved {
t.Fatalf("expected glossary-stage protected_terms rejection, got %+v", result.Decisions[0])
}
if result.Decisions[0].ReasonCode != frameworkvalidators.ReasonProtectedGlossaryTerm {
t.Fatalf("expected reason %q, got %+v", frameworkvalidators.ReasonProtectedGlossaryTerm, result.Decisions[0])
}
}
func TestNewGlossaryStageAllowsProtectedTermInsertion(t *testing.T) {
v, err := NewGlossaryStage()
if err != nil {
t.Fatalf("NewGlossaryStage: %v", err)
}
req := contracts.ValidationRequest{
ModuleKey: "glossary",
Glossary: tinyGlossary(),
CandidateProposal: []proposals.EnrichedCorrectionProposal{
proposal(0, "glossary", "gestures", "Jesters"),
},
}
result, err := v.Validate(context.Background(), req)
if err != nil {
t.Fatalf("Validate: %v", err)
}
if len(result.Decisions) != 1 {
t.Fatalf("expected one decision, got %+v", result.Decisions)
}
if !result.Decisions[0].Approved {
t.Fatalf("expected glossary-stage approval for protected-term insertion, got %+v", result.Decisions[0])
}
}

41
internal/validators/registry.go
View File

@@ -5,7 +5,16 @@ import (
"strings"
"gitea.maximumdirect.net/eric/audita/internal/framework/contracts"
frameworkvalidators "gitea.maximumdirect.net/eric/audita/internal/framework/validators"
"gitea.maximumdirect.net/eric/audita/internal/validators/confidence_threshold"
"gitea.maximumdirect.net/eric/audita/internal/validators/editorial_review"
"gitea.maximumdirect.net/eric/audita/internal/validators/grammar_review"
"gitea.maximumdirect.net/eric/audita/internal/validators/meaning_reversal_review"
"gitea.maximumdirect.net/eric/audita/internal/validators/no_effect"
"gitea.maximumdirect.net/eric/audita/internal/validators/non_empty_corrected_text"
"gitea.maximumdirect.net/eric/audita/internal/validators/original_text_presence"
"gitea.maximumdirect.net/eric/audita/internal/validators/protected_terms"
"gitea.maximumdirect.net/eric/audita/internal/validators/spoken_form_plausibility"
"gitea.maximumdirect.net/eric/audita/internal/validators/spoken_word_review"
)
const (
@@ -34,26 +43,16 @@ type Registry struct {
func NewBuiltInRegistry() *Registry {
defs := []BuiltInValidatorDefinition{
{Key: KeyConfidenceThreshold, Build: func() (contracts.Validator, error) { return frameworkvalidators.ConfidenceThresholdValidator{}, nil }},
{Key: KeyOriginalTextPresence, Build: func() (contracts.Validator, error) { return frameworkvalidators.OriginalTextPresenceValidator{}, nil }},
{Key: KeyNonEmptyCorrectedText, Build: func() (contracts.Validator, error) { return frameworkvalidators.NonEmptyCorrectionValidator{}, nil }},
{Key: KeyNoEffect, Build: func() (contracts.Validator, error) { return frameworkvalidators.NoEffectValidator{}, nil }},
{Key: KeyProtectedTerms, Build: func() (contracts.Validator, error) { return frameworkvalidators.ProtectedGlossaryTermValidator{}, nil }},
{Key: KeySpokenFormPlausibility, LLMBacked: true, Build: func() (contracts.Validator, error) {
return frameworkvalidators.NewLLMBackedValidator(KeySpokenFormPlausibility, frameworkvalidators.LLMValidatorTypeSpokenFormPlausibility, "")
}},
{Key: KeyMeaningReversalReview, LLMBacked: true, Build: func() (contracts.Validator, error) {
return frameworkvalidators.NewLLMBackedValidator(KeyMeaningReversalReview, frameworkvalidators.LLMValidatorTypeMeaningReversal, "")
}},
{Key: KeyEditorialReview, LLMBacked: true, Build: func() (contracts.Validator, error) {
return frameworkvalidators.NewLLMBackedValidator(KeyEditorialReview, frameworkvalidators.LLMValidatorTypeEditorialReview, "")
}},
{Key: KeyGrammarReview, LLMBacked: true, Build: func() (contracts.Validator, error) {
return frameworkvalidators.NewLLMBackedValidator(KeyGrammarReview, frameworkvalidators.LLMValidatorTypeGrammarReview, "")
}},
{Key: KeySpokenWordReview, LLMBacked: true, Build: func() (contracts.Validator, error) {
return frameworkvalidators.NewLLMBackedValidator(KeySpokenWordReview, frameworkvalidators.LLMValidatorTypeSpokenWordReview, "")
}},
{Key: KeyConfidenceThreshold, Build: confidence_threshold.New},
{Key: KeyOriginalTextPresence, Build: original_text_presence.New},
{Key: KeyNonEmptyCorrectedText, Build: non_empty_corrected_text.New},
{Key: KeyNoEffect, Build: no_effect.New},
{Key: KeyProtectedTerms, Build: protected_terms.New},
{Key: KeySpokenFormPlausibility, LLMBacked: true, Build: spoken_form_plausibility.New},
{Key: KeyMeaningReversalReview, LLMBacked: true, Build: meaning_reversal_review.New},
{Key: KeyEditorialReview, LLMBacked: true, Build: editorial_review.New},
{Key: KeyGrammarReview, LLMBacked: true, Build: grammar_review.New},
{Key: KeySpokenWordReview, LLMBacked: true, Build: spoken_word_review.New},
}
m := make(map[string]BuiltInValidatorDefinition, len(defs))

165
internal/validators/registry_test.go
View File

@@ -1,7 +1,23 @@
package validators
import (
"context"
"testing"
"gitea.maximumdirect.net/eric/audita/internal/core/schema"
"gitea.maximumdirect.net/eric/audita/internal/framework/contracts"
"gitea.maximumdirect.net/eric/audita/internal/framework/proposals"
"gitea.maximumdirect.net/eric/audita/internal/validators/confidence_threshold"
"gitea.maximumdirect.net/eric/audita/internal/validators/editorial_review"
"gitea.maximumdirect.net/eric/audita/internal/validators/grammar_review"
"gitea.maximumdirect.net/eric/audita/internal/validators/meaning_reversal_review"
validatormetadata "gitea.maximumdirect.net/eric/audita/internal/validators/metadata"
"gitea.maximumdirect.net/eric/audita/internal/validators/no_effect"
"gitea.maximumdirect.net/eric/audita/internal/validators/non_empty_corrected_text"
"gitea.maximumdirect.net/eric/audita/internal/validators/original_text_presence"
"gitea.maximumdirect.net/eric/audita/internal/validators/protected_terms"
"gitea.maximumdirect.net/eric/audita/internal/validators/spoken_form_plausibility"
"gitea.maximumdirect.net/eric/audita/internal/validators/spoken_word_review"
)
func TestBuiltInRegistryRegistersAllKeys(t *testing.T) {
@@ -32,6 +48,155 @@ func TestBuiltInRegistryRegistersAllKeys(t *testing.T) {
}
}
func TestBuiltInValidatorPackagesConstruct(t *testing.T) {
type validatorCtor struct {
name string
key string
build func() (contracts.Validator, error)
wantClass validatormetadata.ExecutionClass
}
cases := []validatorCtor{
{name: "confidence_threshold", key: KeyConfidenceThreshold, build: confidence_threshold.New, wantClass: validatormetadata.ExecutionClassDeterministic},
{name: "original_text_presence", key: KeyOriginalTextPresence, build: original_text_presence.New, wantClass: validatormetadata.ExecutionClassDeterministic},
{name: "non_empty_corrected_text", key: KeyNonEmptyCorrectedText, build: non_empty_corrected_text.New, wantClass: validatormetadata.ExecutionClassDeterministic},
{name: "no_effect", key: KeyNoEffect, build: no_effect.New, wantClass: validatormetadata.ExecutionClassDeterministic},
{name: "protected_terms", key: KeyProtectedTerms, build: protected_terms.New, wantClass: validatormetadata.ExecutionClassDeterministic},
{name: "spoken_form_plausibility", key: KeySpokenFormPlausibility, build: spoken_form_plausibility.New, wantClass: validatormetadata.ExecutionClassLLMBacked},
{name: "meaning_reversal_review", key: KeyMeaningReversalReview, build: meaning_reversal_review.New, wantClass: validatormetadata.ExecutionClassLLMBacked},
{name: "editorial_review", key: KeyEditorialReview, build: editorial_review.New, wantClass: validatormetadata.ExecutionClassLLMBacked},
{name: "grammar_review", key: KeyGrammarReview, build: grammar_review.New, wantClass: validatormetadata.ExecutionClassLLMBacked},
{name: "spoken_word_review", key: KeySpokenWordReview, build: spoken_word_review.New, wantClass: validatormetadata.ExecutionClassLLMBacked},
}
for _, tc := range cases {
v, err := tc.build()
if err != nil {
t.Fatalf("%s: build: %v", tc.name, err)
}
if v.Name() != tc.key {
t.Fatalf("%s: expected key %q, got %q", tc.name, tc.key, v.Name())
}
if got := validatormetadata.ClassOf(v); got != tc.wantClass {
t.Fatalf("%s: expected class %q, got %q", tc.name, tc.wantClass, got)
}
}
}
func TestRegistryBuildsClassifiedValidators(t *testing.T) {
r := NewBuiltInRegistry()
llmKeys := map[string]bool{
KeySpokenFormPlausibility: true,
KeyMeaningReversalReview: true,
KeyEditorialReview: true,
KeyGrammarReview: true,
KeySpokenWordReview: true,
}
for _, key := range r.RegisteredKeys() {
v, err := r.MustBuild(key)
if err != nil {
t.Fatalf("must build %q: %v", key, err)
}
if _, ok := v.(validatormetadata.ClassifiedValidator); !ok {
t.Fatalf("expected built validator %q to expose execution classification metadata", key)
}
want := validatormetadata.ExecutionClassDeterministic
if llmKeys[key] {
want = validatormetadata.ExecutionClassLLMBacked
}
if got := validatormetadata.ClassOf(v); got != want {
t.Fatalf("expected class %q for %q, got %q", want, key, got)
}
}
}
func TestRegistryProtectedTermsUsesNonGlossaryStageBehavior(t *testing.T) {
r := NewBuiltInRegistry()
v, err := r.MustBuild(KeyProtectedTerms)
if err != nil {
t.Fatalf("must build protected_terms: %v", err)
}
req := contracts.ValidationRequest{
ModuleKey: "glossary",
Glossary: &schema.Glossary{
Entries: []schema.GlossaryEntry{{Name: "Jesters", Category: "faction", Summary: "Protected"}},
},
CandidateProposal: []proposals.EnrichedCorrectionProposal{
{
CorrectionProposal: proposals.CorrectionProposal{
TargetSegmentID: 1,
OriginalText: "Jesters",
CorrectedText: "Gestures",
Confidence: 1.0,
},
ProposalMetadata: proposals.ProposalMetadata{
ProposalIndex: 0,
ModuleKey: "glossary",
},
},
},
}
result, err := v.Validate(context.Background(), req)
if err != nil {
t.Fatalf("validate protected_terms: %v", err)
}
if len(result.Decisions) != 1 {
t.Fatalf("expected one decision, got %+v", result.Decisions)
}
if !result.Decisions[0].Approved {
t.Fatalf("expected glossary module proposal to be approved by non-glossary-stage protected_terms validator, got %+v", result.Decisions[0])
}
}
func TestResolveBuiltInChainUsesGlossaryStageProtectedTermsBehavior(t *testing.T) {
chain, err := ResolveBuiltInChain("glossary", NewBuiltInRegistry())
if err != nil {
t.Fatalf("resolve glossary chain: %v", err)
}
var v contracts.Validator
for _, candidate := range chain {
if candidate.Name() == KeyProtectedTerms {
v = candidate
break
}
}
if v == nil {
t.Fatalf("expected protected_terms validator in glossary chain")
}
req := contracts.ValidationRequest{
ModuleKey: "glossary",
Glossary: &schema.Glossary{
Entries: []schema.GlossaryEntry{{Name: "Jesters", Category: "faction", Summary: "Protected"}},
},
CandidateProposal: []proposals.EnrichedCorrectionProposal{
{
CorrectionProposal: proposals.CorrectionProposal{
TargetSegmentID: 1,
OriginalText: "Jesters",
CorrectedText: "Gestures",
Confidence: 1.0,
},
ProposalMetadata: proposals.ProposalMetadata{
ProposalIndex: 0,
ModuleKey: "glossary",
},
},
},
}
result, err := v.Validate(context.Background(), req)
if err != nil {
t.Fatalf("validate protected_terms: %v", err)
}
if len(result.Decisions) != 1 {
t.Fatalf("expected one decision, got %+v", result.Decisions)
}
if result.Decisions[0].Approved {
t.Fatalf("expected glossary-stage protected_terms rejection, got %+v", result.Decisions[0])
}
}
func TestBuiltInRegistryUnknownKeyFails(t *testing.T) {
r := NewBuiltInRegistry()
if _, err := r.MustBuild("missing"); err == nil {

17
internal/validators/spoken_form_plausibility/validator.go Normal file
View File

@@ -0,0 +1,17 @@
package spoken_form_plausibility
import (
"gitea.maximumdirect.net/eric/audita/internal/framework/contracts"
frameworkvalidators "gitea.maximumdirect.net/eric/audita/internal/framework/validators"
validatormetadata "gitea.maximumdirect.net/eric/audita/internal/validators/metadata"
)
const Key = "spoken_form_plausibility"
func New() (contracts.Validator, error) {
v, err := frameworkvalidators.NewLLMBackedValidator(Key, frameworkvalidators.LLMValidatorTypeSpokenFormPlausibility, "")
if err != nil {
return nil, err
}
return validatormetadata.Wrap(v, validatormetadata.ExecutionClassLLMBacked), nil
}

17
internal/validators/spoken_word_review/validator.go Normal file
View File

@@ -0,0 +1,17 @@
package spoken_word_review
import (
"gitea.maximumdirect.net/eric/audita/internal/framework/contracts"
frameworkvalidators "gitea.maximumdirect.net/eric/audita/internal/framework/validators"
validatormetadata "gitea.maximumdirect.net/eric/audita/internal/validators/metadata"
)
const Key = "spoken_word_review"
func New() (contracts.Validator, error) {
v, err := frameworkvalidators.NewLLMBackedValidator(Key, frameworkvalidators.LLMValidatorTypeSpokenWordReview, "")
if err != nil {
return nil, err
}
return validatormetadata.Wrap(v, validatormetadata.ExecutionClassLLMBacked), nil
}