Complete Phase 13 glossary module

This commit is contained in:
2026-05-12 11:20:02 +00:00
parent fc3a7b7a67
commit 543a7ff8ef
14 changed files with 1049 additions and 97 deletions

View File

@@ -34,19 +34,22 @@ Implemented today:
- Proposal-generation diagnostics artifacts with secret redaction. - Proposal-generation diagnostics artifacts with secret redaction.
- Production module registry scaffolding with known-key recognition and explicit unsupported/unimplemented errors. - Production module registry scaffolding with known-key recognition and explicit unsupported/unimplemented errors.
- Production `grammar` module implementation in `internal/modules/grammar`. - Production `grammar` module implementation in `internal/modules/grammar`.
- Production `glossary` module implementation in `internal/modules/glossary`.
- Explicit runtime support for `--modules grammar` through the production runner path. - Explicit runtime support for `--modules grammar` through the production runner path.
- Explicit runtime support for `--modules glossary`, including repeated stages such as `--modules glossary,glossary`.
Not implemented in CLI runtime path today: Not implemented in CLI runtime path today:
- Real module execution pipeline for `glossary`, `homophones`, and `spoken_word`. - Real module execution pipeline for `homophones` and `spoken_word`.
- Real domain proposal prompts for production modules. - Real domain proposal prompts for remaining production modules.
- End-to-end transcript polishing with real module behavior. - End-to-end transcript polishing with the full default module sequence.
Phase sequencing note: Phase sequencing note:
- Phase 9 LLM infrastructure is complete (structured client, scheduler, effective config resolution, diagnostics primitives); - Phase 9 LLM infrastructure is complete (structured client, scheduler, effective config resolution, diagnostics primitives);
- Phase 10 LLM-backed validator runtime integration is complete; - Phase 10 LLM-backed validator runtime integration is complete;
- Phase 11 shared proposal-generation framework and module-registry scaffolding are complete; - Phase 11 shared proposal-generation framework and module-registry scaffolding are complete;
- Phase 12 grammar module implementation and explicit runtime wiring are complete; - Phase 12 grammar module implementation and explicit runtime wiring are complete;
- next recommended phase is Phase 13 (glossary module and protected-term behavior). - Phase 13 glossary module and protected-term behavior are complete;
- next recommended phase is Phase 14 (homophones module).
## Actual Go package layout ## Actual Go package layout
@@ -109,6 +112,10 @@ internal/modules/grammar/
module.go module.go
prompt.go prompt.go
internal/modules/glossary/
module.go
prompt.go
internal/framework/validators/ internal/framework/validators/
models.go models.go
deterministic.go deterministic.go
@@ -144,17 +151,17 @@ Current runtime flow (`internal/cli/run.go`):
10. Chunk normalized transcript and compute chunk summaries. 10. Chunk normalized transcript and compute chunk summaries.
11. Write chunking summary artifact. 11. Write chunking summary artifact.
12. Execute runner modules sequentially when: 12. Execute runner modules sequentially when:
- `--modules` is explicitly provided (production grammar path); or - `--modules` is explicitly provided (production grammar/glossary paths); or
- a test/injected module factory is provided. - a test/injected module factory is provided.
13. Output working transcript to `--output` file or stdout. 13. Output working transcript to `--output` file or stdout.
14. Build process report (`phase` currently set to `phase12-grammar-module`). 14. Build process report (`phase` currently set to `phase13-glossary-module`).
15. Optionally write `--report-json`; always write run-dir `report.json`. 15. Optionally write `--report-json`; always write run-dir `report.json`.
16. Apply work-dir retention. 16. Apply work-dir retention.
Important behavior details: Important behavior details:
- Glossary is validated but not yet used for real correction module logic. - Glossary is validated and is used for explicit glossary/grammar module correction paths.
- Default production CLI behavior remains deterministic normalization/chunking/reporting unless modules are explicitly selected with `--modules`. - Default production CLI behavior remains deterministic normalization/chunking/reporting unless modules are explicitly selected with `--modules`.
- Explicit `--modules grammar` runs the production grammar module path with LLM-backed proposal generation and validator-chain execution. - Explicit `--modules grammar` and `--modules glossary` run production module paths with LLM-backed proposal generation and validator-chain execution.
- Default runs (without explicit module selection) do not perform LLM calls. - Default runs (without explicit module selection) do not perform LLM calls.
- Success path is generally quiet on stderr. - Success path is generally quiet on stderr.
- Source IDs are preserved into a canonical transcript before normalization; normalization then reassigns output IDs sequentially from `1`. - Source IDs are preserved into a canonical transcript before normalization; normalization then reassigns output IDs sequentially from `1`.
@@ -203,7 +210,7 @@ Implemented config surfaces include:
- work-dir and retention mode - work-dir and retention mode
Current caveat: Current caveat:
- LLM/module-related settings are active for explicit grammar runs; the default non-explicit path remains deterministic. - LLM/module-related settings are active for explicit grammar/glossary runs; the default non-explicit path remains deterministic.
## Implemented structured LLM infrastructure ## Implemented structured LLM infrastructure
`internal/framework/contracts` now defines a typed structured-completion contract: `internal/framework/contracts` now defines a typed structured-completion contract:
@@ -220,7 +227,7 @@ Current caveat:
Current runtime boundary: Current runtime boundary:
- the default CLI runtime path (without explicit module selection) still does not instantiate the full production module sequence. - the default CLI runtime path (without explicit module selection) still does not instantiate the full production module sequence.
- LLM calls are exercised in production when `--modules grammar` is explicitly requested and in tests when fake/injected clients are used. - LLM calls are exercised in production when `--modules grammar` or `--modules glossary` is explicitly requested and in tests when fake/injected clients are used.
`internal/framework/llm` also provides: `internal/framework/llm` also provides:
- a bounded `Scheduler` for controlled concurrent LLM calls with reliable permit release; - a bounded `Scheduler` for controlled concurrent LLM calls with reliable permit release;
@@ -262,7 +269,7 @@ Current behavior details:
`internal/framework/contracts` provides interfaces and run-spec metadata scaffolding, including deterministic repeated module instance naming (`ResolveModuleRunSpecs`). `internal/framework/contracts` provides interfaces and run-spec metadata scaffolding, including deterministic repeated module instance naming (`ResolveModuleRunSpecs`).
These primitives are wired into the production runner and report model. The grammar module is implemented; other production modules remain pending. These primitives are wired into the production runner and report model. The grammar and glossary modules are implemented; other production modules remain pending.
## Implemented validator runtime infrastructure ## Implemented validator runtime infrastructure
`internal/framework/validators` provides deterministic validator infrastructure: `internal/framework/validators` provides deterministic validator infrastructure:
@@ -327,7 +334,7 @@ This helper only produces candidate proposals; validator-chain execution and pro
- diagnostics directory context - diagnostics directory context
- returns explicit errors for unknown keys (`unsupported_module`) and recognized-but-unimplemented keys (`unimplemented_module`). - returns explicit errors for unknown keys (`unsupported_module`) and recognized-but-unimplemented keys (`unimplemented_module`).
The `grammar` module key is now registered and constructible. `glossary`, `homophones`, and `spoken_word` remain recognized-but-unimplemented. The `grammar` and `glossary` module keys are now registered and constructible. `homophones` and `spoken_word` remain recognized-but-unimplemented.
## Implemented grammar production module ## Implemented grammar production module
`internal/modules/grammar` now provides the first production module: `internal/modules/grammar` now provides the first production module:
@@ -340,6 +347,27 @@ The `grammar` module key is now registered and constructible. `glossary`, `homop
- grammar confidence threshold enforcement through existing validator/config infrastructure; - grammar confidence threshold enforcement through existing validator/config infrastructure;
- module-level reporting and diagnostics capture through existing runner/reporting paths. - module-level reporting and diagnostics capture through existing runner/reporting paths.
## Implemented glossary production module
`internal/modules/glossary` now provides the second production module:
- prompt builder aligned to Python glossary-module intent, constrained to glossary-backed domain/acoustic corrections;
- prompt context includes glossary names, aliases, categories, summaries, and plural forms where available;
- guardrails against broad style rewriting and against replacing unrelated terms simply because they appear in glossary entries;
- proposal generation through `internal/framework/proposal_generation` and `contracts.StructuredLLMClient`;
- scheduler-aware proposal calls through existing `contracts.LLMScheduler` hooks;
- replacement policy `replace_all` (matching Python glossary behavior);
- validator chain integration using existing deterministic + LLM-backed validators;
- glossary confidence threshold enforcement through existing validator/config infrastructure;
- module-level reporting and diagnostics capture through existing runner/reporting paths;
- explicit support for repeated glossary stages with deterministic instance names (`glossary_1`, `glossary_2`, ...), where later stages see prior-stage working transcript changes.
## Implemented protected-term behavior
`internal/framework/validators/protected_terms.go` provides deterministic glossary-derived protected vocabulary:
- extracts protected terms from glossary names and aliases;
- includes explicit plural fields and synthetic plural forms where safe;
- deduplicates and returns stable ordering for repeatable behavior/tests.
This vocabulary is used by deterministic validators for both glossary-stage and non-glossary-stage protection checks, keeping protected-term guardrails active across modules.
## Reports and diagnostics (implemented) ## Reports and diagnostics (implemented)
Current per-run artifacts include: Current per-run artifacts include:
- `source-transcript.json` - `source-transcript.json`
@@ -379,7 +407,7 @@ Retention modes implemented in `ApplyRetention`:
Current runtime note: Current runtime note:
- default non-explicit runs usually have no module-level skipped corrections, so `auto` commonly removes clean successful run directories. - default non-explicit runs usually have no module-level skipped corrections, so `auto` commonly removes clean successful run directories.
- explicit grammar runs can produce validator rejections and application skips, which are reflected in reports and retention input. - explicit grammar/glossary runs can produce validator rejections and application skips, which are reflected in reports and retention input.
Intentionally deferred to module/LLM phases: Intentionally deferred to module/LLM phases:
- real domain proposal prompts and production module implementations remain tied to later module phases. - real domain proposal prompts and production module implementations remain tied to later module phases.
@@ -401,13 +429,15 @@ Implemented tests currently cover:
- shared proposal-generation request/response parsing, deterministic indexing, scheduler hooks, and diagnostics redaction (`internal/framework/proposal_generation/*_test.go`, `internal/framework/runner/*_test.go`) - shared proposal-generation request/response parsing, deterministic indexing, scheduler hooks, and diagnostics redaction (`internal/framework/proposal_generation/*_test.go`, `internal/framework/runner/*_test.go`)
- production module-registry known-key recognition and unsupported/unimplemented error behavior (`internal/framework/modules/*_test.go`, `internal/cli/run_test.go`) - production module-registry known-key recognition and unsupported/unimplemented error behavior (`internal/framework/modules/*_test.go`, `internal/cli/run_test.go`)
- production grammar module prompt constraints, proposal mapping, validator-chain behavior, confidence-threshold enforcement, diagnostics redaction, and explicit CLI/runtime integration (`internal/modules/grammar/*_test.go`, `internal/cli/run_test.go`, `internal/framework/runner/*_test.go`) - production grammar module prompt constraints, proposal mapping, validator-chain behavior, confidence-threshold enforcement, diagnostics redaction, and explicit CLI/runtime integration (`internal/modules/grammar/*_test.go`, `internal/cli/run_test.go`, `internal/framework/runner/*_test.go`)
- production glossary module prompt constraints, proposal mapping, validator-chain behavior, confidence-threshold enforcement, diagnostics redaction, repeated-stage behavior, and explicit CLI/runtime integration (`internal/modules/glossary/*_test.go`, `internal/cli/run_test.go`, `internal/framework/runner/*_test.go`)
- glossary-derived protected-term extraction and stable behavior (`internal/framework/validators/protected_terms_test.go`)
Not covered yet (because not implemented): production `glossary`, `homophones`, and `spoken_word` modules plus full default-sequence transcript-polishing runtime behavior. Not covered yet (because not implemented): production `homophones` and `spoken_word` modules plus full default-sequence transcript-polishing runtime behavior.
## Intended final architecture (not yet implemented) ## Intended final architecture (not yet implemented)
The intended end-state still matches the rewrite plan: The intended end-state still matches the rewrite plan:
- sequential module pipeline over a mutable working transcript - sequential module pipeline over a mutable working transcript
- real module implementations (`glossary`, `homophones`, `spoken_word`, `grammar`) - real module implementations (`homophones`, `spoken_word`)
- structured LLM proposal generation - structured LLM proposal generation
- deterministic and LLM validators - deterministic and LLM validators
- validator cardinality enforcement in pipeline execution - validator cardinality enforcement in pipeline execution

View File

@@ -61,6 +61,10 @@ Implemented:
- Production module-registry scaffolding with known key recognition and explicit unsupported/unimplemented errors. - Production module-registry scaffolding with known key recognition and explicit unsupported/unimplemented errors.
- Production grammar module package with Python-aligned prompt intent and guardrails. - Production grammar module package with Python-aligned prompt intent and guardrails.
- Explicit `--modules grammar` runtime path through runner, shared proposal generation, validators, application, reporting, and diagnostics. - Explicit `--modules grammar` runtime path through runner, shared proposal generation, validators, application, reporting, and diagnostics.
- Production glossary module package with Python-aligned prompt intent and guardrails.
- Glossary-derived deterministic protected-term extraction and validator integration.
- Explicit `--modules glossary` runtime path through runner, shared proposal generation, validators, application, reporting, and diagnostics.
- Repeated glossary stage support with deterministic instance names (`glossary_1`, `glossary_2`), including mutable working-transcript handoff.
- Broad deterministic and CLI/subprocess test coverage for implemented phases through `go test ./...`. - Broad deterministic and CLI/subprocess test coverage for implemented phases through `go test ./...`.
- Internal typed structured LLM contract (`StructuredLLMClient.CompleteStructured(ctx, req, out)`). - Internal typed structured LLM contract (`StructuredLLMClient.CompleteStructured(ctx, req, out)`).
- `internal/framework/llm` instructor-go-backed adapter with: - `internal/framework/llm` instructor-go-backed adapter with:
@@ -76,9 +80,9 @@ Implemented:
- Generic JSON diagnostics primitives for LLM interactions (request metadata, request payload, response payload, optional error payload) with secret redaction. - Generic JSON diagnostics primitives for LLM interactions (request metadata, request payload, response payload, optional error payload) with secret redaction.
Not yet implemented in runtime pipeline: Not yet implemented in runtime pipeline:
- Real correction modules for `glossary`, `homophones`, and `spoken_word`. - Real correction modules for `homophones` and `spoken_word`.
- Domain proposal prompts for remaining real modules. - Domain proposal prompts for remaining real modules.
- End-to-end transcript polishing behavior. - End-to-end transcript polishing behavior with the full default module sequence.
## Completed phases ## Completed phases
@@ -216,7 +220,7 @@ Not implemented in Phase 8 (by design):
## Remaining work plan ## Remaining work plan
Next recommended phase: **Phase 13 (glossary module and protected-term behavior)**. Next recommended phase: **Phase 14 (homophones module)**.
## Phase 9: Structured LLM client and scheduler infrastructure ## Phase 9: Structured LLM client and scheduler infrastructure
@@ -362,44 +366,28 @@ Not implemented in Phase 12 (by design):
## Phase 13: Glossary module and protected-term behavior ## Phase 13: Glossary module and protected-term behavior
### Purpose Completed.
Implement the glossary correction module and the glossary-derived protection behavior needed by downstream modules. Implemented:
- Production glossary module package in `internal/modules/glossary`.
- Glossary prompt builder aligned to Python intent and constrained to glossary-supported domain/acoustic corrections.
- Prompt context using glossary names, aliases, categories, summaries, and plural forms where available.
- Glossary proposal generation through shared `internal/framework/proposal_generation` using `contracts.StructuredLLMClient`.
- Scheduler-aware glossary proposal generation through existing scheduler hooks.
- Glossary replacement policy `replace_all` (matching Python behavior).
- Glossary validator chain using existing deterministic and LLM-backed validators.
- Glossary confidence threshold enforcement through existing config + confidence-threshold validator behavior.
- Deterministic glossary-derived protected-term extraction (`internal/framework/validators/protected_terms.go`) from names, aliases, and plural forms, with stable deduplicated ordering.
- Protected-term validator behavior remaining available to non-glossary modules via existing deterministic validators.
- Explicit runtime support for `--modules glossary` through normalization, chunking, runner, proposal generation, validation, application, and reporting.
- Repeated glossary-stage support (`--modules glossary,glossary`) with deterministic instance naming and mutable working-transcript handoff across stages.
- Prompt/response diagnostics artifacts for glossary proposal + validator interactions with secret redaction.
- Module-level reports for glossary including generated proposals, validator decisions/rejections, applied changes, and application skips.
- CLI/runtime fake-client tests for approved proposals, validator rejection, application skips, repeated stages, diagnostics, failure/error.log behavior, and report outputs (`--report-json` and run-dir `report.json`).
### Scope Not implemented in Phase 13 (by design):
- Production `homophones` and `spoken_word` modules.
Implement: - Full default module sequence execution as a feature-complete claim.
- `glossary` module package.
- Glossary prompt builder ported from Python.
- Glossary structured response model.
- Glossary replacement policy.
- Glossary confidence threshold handling.
- Glossary validator chain.
- Protected-term extraction from parsed glossary.
- Protected-term validator behavior used by other modules where applicable.
- Prompt/response diagnostics.
- CLI support for `--modules glossary`.
- Fake LLM tests.
- Tests for repeated glossary stages using `glossary,glossary`.
Do not implement:
- Homophones module.
- Spoken-word module.
- Default full pipeline parity claim.
### Expected behavior at end of phase
Running `audita process ... --modules glossary` should perform real glossary-supported corrections. Repeated glossary stages should work and be reported as separate module instances.
### Definition of done
- Glossary module runs in the production runner.
- Glossary terms and aliases are used in prompts and validators.
- Protected-term behavior is implemented and tested.
- Repeated glossary module instances are reported correctly.
- Applied/skipped glossary changes appear in reports.
- Prompt/response diagnostics are written.
- `go test ./...` passes without requiring external LLM credentials.
## Phase 14: Homophones module ## Phase 14: Homophones module

View File

@@ -460,7 +460,7 @@ func extractErrorPhase(err error) (phase string, message string) {
func buildProcessReport(status string, inv processInvocation, runDir *diagnostics.RunDirectory, startedAt, completedAt time.Time, errorMessage string, errorPhase string, normalizationSummary *normalization.NormalizationSummary, chunkingSummary *chunking.Summary, runOutput *runner.RunOutput) reporting.ProcessReport { func buildProcessReport(status string, inv processInvocation, runDir *diagnostics.RunDirectory, startedAt, completedAt time.Time, errorMessage string, errorPhase string, normalizationSummary *normalization.NormalizationSummary, chunkingSummary *chunking.Summary, runOutput *runner.RunOutput) reporting.ProcessReport {
report := reporting.ProcessReport{ report := reporting.ProcessReport{
Phase: "phase12-grammar-module", Phase: "phase13-glossary-module",
Status: status, Status: status,
Operation: "process", Operation: "process",
TranscriptPath: inv.TranscriptPath, TranscriptPath: inv.TranscriptPath,

View File

@@ -616,8 +616,8 @@ func TestRunProcessReportJSONIncludesChunkingSummary(t *testing.T) {
if report.Chunking.MaxSectionTokens == 0 { if report.Chunking.MaxSectionTokens == 0 {
t.Errorf("expected max_section_tokens in report") t.Errorf("expected max_section_tokens in report")
} }
if report.Phase != "phase12-grammar-module" { if report.Phase != "phase13-glossary-module" {
t.Errorf("expected phase 'phase12-grammar-module', got %q", report.Phase) t.Errorf("expected phase 'phase13-glossary-module', got %q", report.Phase)
} }
} }
@@ -664,12 +664,13 @@ func (v fakeValidator) Validate(ctx context.Context, req contracts.ValidationReq
type fakeStructuredLLMClient struct { type fakeStructuredLLMClient struct {
validationResponses []validators.LLMValidationResponse validationResponses []validators.LLMValidationResponse
proposalResponses []proposal_generation.StructuredCorrectionSet proposalResponses []proposal_generation.StructuredCorrectionSet
calls []string
err error err error
} }
func (f *fakeStructuredLLMClient) CompleteStructured(ctx context.Context, req contracts.StructuredCompletionRequest, out any) (contracts.StructuredCompletionResponse, error) { func (f *fakeStructuredLLMClient) CompleteStructured(ctx context.Context, req contracts.StructuredCompletionRequest, out any) (contracts.StructuredCompletionResponse, error) {
_ = ctx _ = ctx
_ = req f.calls = append(f.calls, req.StageName)
if f.err != nil { if f.err != nil {
return contracts.StructuredCompletionResponse{}, f.err return contracts.StructuredCompletionResponse{}, f.err
} }
@@ -923,7 +924,7 @@ func TestRunProcessProductionRegistryUnimplementedModuleFailsCleanly(t *testing.
exitCode := Run([]string{ exitCode := Run([]string{
"process", transcriptPath, "process", transcriptPath,
"--glossary", fixturePath("tiny_glossary.yaml"), "--glossary", fixturePath("tiny_glossary.yaml"),
"--modules", "glossary", "--modules", "homophones",
"--work-dir", workDir, "--work-dir", workDir,
"--work-dir-retention", "always", "--work-dir-retention", "always",
"--report-json", reportPath, "--report-json", reportPath,
@@ -954,7 +955,7 @@ func TestRunProcessProductionRegistryUnimplementedModuleFailsCleanly(t *testing.
} }
func TestRunProcessExplicitUnimplementedModulesFailClearly(t *testing.T) { func TestRunProcessExplicitUnimplementedModulesFailClearly(t *testing.T) {
for _, moduleKey := range []string{"glossary", "homophones", "spoken_word"} { for _, moduleKey := range []string{"homophones", "spoken_word"} {
t.Run(moduleKey, func(t *testing.T) { t.Run(moduleKey, func(t *testing.T) {
var stdout, stderr bytes.Buffer var stdout, stderr bytes.Buffer
transcriptPath := writeFile(t, "transcript.json", `[ transcriptPath := writeFile(t, "transcript.json", `[
@@ -1196,6 +1197,281 @@ func TestRunProcessDefaultBehaviorRemainsDeterministicWithoutExplicitModules(t *
} }
} }
func TestRunProcessExplicitGlossaryAppliesCorrectionAndReportsDiagnostics(t *testing.T) {
secret := "phase13-secret"
proposalClient := &fakeStructuredLLMClient{
proposalResponses: []proposal_generation.StructuredCorrectionSet{
{
Corrections: []proposal_generation.StructuredCorrectionProposal{
{TargetSegmentID: 1, OriginalText: "gestures", CorrectedText: "Jesters", Confidence: 0.95},
},
},
},
}
validationClient := &fakeStructuredLLMClient{
validationResponses: []validators.LLMValidationResponse{
{Validations: []validators.LLMValidationDecision{{CorrectionIndex: 0, Approved: true, Confidence: 0.99, Reason: "spoken plausible"}}},
{Validations: []validators.LLMValidationDecision{{CorrectionIndex: 0, Approved: true, Confidence: 0.99, Reason: secret}}},
},
}
processProposalLLMClient = proposalClient
processValidationLLMClient = validationClient
t.Setenv("AUDITA_LLM_API_KEY", secret)
t.Setenv("AUDITA_VALIDATION_LLM_API_KEY", secret)
t.Cleanup(func() {
processProposalLLMClient = nil
processValidationLLMClient = nil
})
var stdout, stderr bytes.Buffer
workDir := t.TempDir()
reportPath := filepath.Join(t.TempDir(), "report.json")
outputPath := filepath.Join(t.TempDir(), "out.json")
transcriptPath := writeFile(t, "transcript.json", `[
{"id":1,"speaker":"Alice","start":0.0,"end":1.0,"text":"There were gestures in the hall."}
]`)
exitCode := Run([]string{
"process", transcriptPath,
"--glossary", fixturePath("tiny_glossary.yaml"),
"--modules", "glossary",
"--output", outputPath,
"--report-json", reportPath,
"--work-dir", workDir,
"--work-dir-retention", "always",
}, &stdout, &stderr)
if exitCode != 0 {
t.Fatalf("expected success, got %d stderr=%q", exitCode, stderr.String())
}
if stdout.Len() != 0 {
t.Fatalf("expected empty stdout with --output, got %q", stdout.String())
}
parsed, err := schema.ParseTranscriptJSON(readFile(t, outputPath))
if err != nil {
t.Fatalf("parse output: %v", err)
}
if parsed.Segments[0].Text != "There were Jesters in the hall." {
t.Fatalf("expected glossary correction applied, got %q", parsed.Segments[0].Text)
}
report := readProcessReport(t, reportPath)
if len(report.ModuleResults) != 1 || report.ModuleResults[0].ModuleKey != "glossary" {
t.Fatalf("expected one glossary module result, got %+v", report.ModuleResults)
}
if len(report.ModuleResults[0].AppliedChanges) != 1 {
t.Fatalf("expected one applied glossary change, got %+v", report.ModuleResults[0].AppliedChanges)
}
if len(report.ModuleResults[0].ValidatorDecisions) == 0 {
t.Fatalf("expected validator decisions in report")
}
runDir := onlyRunDir(t, workDir)
runReport := readProcessReport(t, filepath.Join(runDir, "report.json"))
if len(runReport.ModuleResults) != 1 {
t.Fatalf("expected module results in run-dir report")
}
diagFiles, globErr := filepath.Glob(filepath.Join(runDir, "glossary", "*response-payload.json"))
if globErr != nil {
t.Fatalf("glob diagnostics: %v", globErr)
}
if len(diagFiles) == 0 {
t.Fatalf("expected glossary diagnostics payload files in %s", filepath.Join(runDir, "glossary"))
}
for _, f := range diagFiles {
raw := string(readFile(t, f))
if strings.Contains(raw, secret) {
t.Fatalf("secret leaked in diagnostics %q: %s", f, raw)
}
}
}
func TestRunProcessExplicitGlossaryRejectedAndApplicationSkipAreDistinct(t *testing.T) {
processProposalLLMClient = &fakeStructuredLLMClient{
proposalResponses: []proposal_generation.StructuredCorrectionSet{
{
Corrections: []proposal_generation.StructuredCorrectionProposal{
{TargetSegmentID: 1, OriginalText: "there were gestures", CorrectedText: "There were gestures", Confidence: 0.99},
{TargetSegmentID: 1, OriginalText: "there were gestures", CorrectedText: "there were jesters", Confidence: 0.99},
{TargetSegmentID: 1, OriginalText: "hall", CorrectedText: "temple", Confidence: 0.99},
},
},
},
}
processValidationLLMClient = &fakeStructuredLLMClient{
validationResponses: []validators.LLMValidationResponse{
{
Validations: []validators.LLMValidationDecision{
{CorrectionIndex: 0, Approved: true, Confidence: 0.9, Reason: "ok"},
{CorrectionIndex: 1, Approved: true, Confidence: 0.9, Reason: "ok"},
{CorrectionIndex: 2, Approved: false, Confidence: 0.9, Reason: "reject"},
},
},
{
Validations: []validators.LLMValidationDecision{
{CorrectionIndex: 0, Approved: true, Confidence: 0.9, Reason: "ok"},
{CorrectionIndex: 1, Approved: true, Confidence: 0.9, Reason: "ok"},
},
},
},
}
t.Cleanup(func() {
processProposalLLMClient = nil
processValidationLLMClient = nil
})
var stdout, stderr bytes.Buffer
reportPath := filepath.Join(t.TempDir(), "report.json")
outputPath := filepath.Join(t.TempDir(), "out.json")
transcriptPath := writeFile(t, "transcript.json", `[
{"id":1,"speaker":"Alice","start":0.0,"end":1.0,"text":"there were gestures in the hall and there were gestures."}
]`)
exitCode := Run([]string{
"process", transcriptPath,
"--glossary", fixturePath("tiny_glossary.yaml"),
"--modules", "glossary",
"--output", outputPath,
"--report-json", reportPath,
}, &stdout, &stderr)
if exitCode != 0 {
t.Fatalf("expected success, got %d stderr=%q", exitCode, stderr.String())
}
report := readProcessReport(t, reportPath)
if len(report.ModuleResults) != 1 {
t.Fatalf("expected one module result")
}
module := report.ModuleResults[0]
if len(module.ValidatorRejected) != 1 {
t.Fatalf("expected one validator rejection, got %+v", module.ValidatorRejected)
}
if len(module.SkippedChanges) != 1 {
t.Fatalf("expected one application skip, got %+v", module.SkippedChanges)
}
if module.ValidatorRejected[0].ReasonCode == string(module.SkippedChanges[0].SkipReason) {
t.Fatalf("validator rejection and application skip should remain distinct")
}
}
func TestRunProcessExplicitGlossaryRepeatedStagesUseDeterministicInstanceNamesAndSeePriorChanges(t *testing.T) {
proposalClient := &fakeStructuredLLMClient{
proposalResponses: []proposal_generation.StructuredCorrectionSet{
{
Corrections: []proposal_generation.StructuredCorrectionProposal{
{TargetSegmentID: 1, OriginalText: "gestures", CorrectedText: "Jesters", Confidence: 0.99},
},
},
{
Corrections: []proposal_generation.StructuredCorrectionProposal{
{TargetSegmentID: 1, OriginalText: "Jesters", CorrectedText: "JESTERS", Confidence: 0.99},
},
},
},
}
validationClient := &fakeStructuredLLMClient{
validationResponses: []validators.LLMValidationResponse{
{Validations: []validators.LLMValidationDecision{{CorrectionIndex: 0, Approved: true, Confidence: 0.99, Reason: "ok"}}},
{Validations: []validators.LLMValidationDecision{{CorrectionIndex: 0, Approved: true, Confidence: 0.99, Reason: "ok"}}},
{Validations: []validators.LLMValidationDecision{{CorrectionIndex: 0, Approved: true, Confidence: 0.99, Reason: "ok"}}},
{Validations: []validators.LLMValidationDecision{{CorrectionIndex: 0, Approved: true, Confidence: 0.99, Reason: "ok"}}},
},
}
processProposalLLMClient = proposalClient
processValidationLLMClient = validationClient
t.Cleanup(func() {
processProposalLLMClient = nil
processValidationLLMClient = nil
})
var stdout, stderr bytes.Buffer
reportPath := filepath.Join(t.TempDir(), "report.json")
outputPath := filepath.Join(t.TempDir(), "out.json")
transcriptPath := writeFile(t, "transcript.json", `[
{"id":1,"speaker":"Alice","start":0.0,"end":1.0,"text":"There were gestures in the hall."}
]`)
exitCode := Run([]string{
"process", transcriptPath,
"--glossary", fixturePath("tiny_glossary.yaml"),
"--modules", "glossary,glossary",
"--output", outputPath,
"--report-json", reportPath,
}, &stdout, &stderr)
if exitCode != 0 {
t.Fatalf("expected success, got %d stderr=%q", exitCode, stderr.String())
}
parsed, err := schema.ParseTranscriptJSON(readFile(t, outputPath))
if err != nil {
t.Fatalf("parse output: %v", err)
}
if parsed.Segments[0].Text != "There were JESTERS in the hall." {
t.Fatalf("expected second stage to see first stage changes, got %q", parsed.Segments[0].Text)
}
report := readProcessReport(t, reportPath)
if len(report.ModuleResults) != 2 {
t.Fatalf("expected two module results, got %+v", report.ModuleResults)
}
if report.ModuleResults[0].ModuleInstance != "glossary_1" || report.ModuleResults[1].ModuleInstance != "glossary_2" {
t.Fatalf("expected deterministic glossary instance names, got %+v", report.ModuleResults)
}
if len(report.ModuleResults[0].AppliedChanges) != 1 || len(report.ModuleResults[1].AppliedChanges) != 1 {
t.Fatalf("expected one applied change per stage, got %+v", report.ModuleResults)
}
foundStage1 := false
foundStage2 := false
for _, call := range proposalClient.calls {
if strings.HasPrefix(call, "glossary_1:proposal") {
foundStage1 = true
}
if strings.HasPrefix(call, "glossary_2:proposal") {
foundStage2 = true
}
}
if !foundStage1 || !foundStage2 {
t.Fatalf("expected proposal calls for glossary_1 and glossary_2, got %v", proposalClient.calls)
}
}
func TestRunProcessExplicitGlossaryMalformedLLMOutputFailsWithErrorLog(t *testing.T) {
processProposalLLMClient = &fakeStructuredLLMClient{err: errors.New("malformed structured output")}
t.Cleanup(func() { processProposalLLMClient = nil })
var stdout, stderr bytes.Buffer
workDir := t.TempDir()
reportPath := filepath.Join(t.TempDir(), "report.json")
transcriptPath := writeFile(t, "transcript.json", `[
{"id":1,"speaker":"Alice","start":0.0,"end":1.0,"text":"hello"}
]`)
exitCode := Run([]string{
"process", transcriptPath,
"--glossary", fixturePath("tiny_glossary.yaml"),
"--modules", "glossary",
"--work-dir", workDir,
"--work-dir-retention", "always",
"--report-json", reportPath,
}, &stdout, &stderr)
if exitCode == 0 {
t.Fatal("expected failure")
}
if stdout.Len() != 0 {
t.Fatalf("expected empty stdout on failure, got %q", stdout.String())
}
if !strings.Contains(stderr.String(), "runner_execution") {
t.Fatalf("expected runner_execution error, got %q", stderr.String())
}
runDir := onlyRunDir(t, workDir)
if _, err := os.Stat(filepath.Join(runDir, "error.log")); err != nil {
t.Fatalf("expected error.log on failed glossary run: %v", err)
}
report := readProcessReport(t, reportPath)
if report.Status != "failed" || report.ErrorPhase != "runner_execution" {
t.Fatalf("expected failed runner_execution report, got %+v", report)
}
}
func TestRunProcessChunkingSummaryArtifactWritten(t *testing.T) { func TestRunProcessChunkingSummaryArtifactWritten(t *testing.T) {
var stdout bytes.Buffer var stdout bytes.Buffer
var stderr bytes.Buffer var stderr bytes.Buffer

View File

@@ -11,7 +11,7 @@ import (
func TestProcessReportModuleResultsJSONSuccessAndSkipped(t *testing.T) { func TestProcessReportModuleResultsJSONSuccessAndSkipped(t *testing.T) {
now := time.Now().UTC() now := time.Now().UTC()
report := ProcessReport{ report := ProcessReport{
Phase: "phase12-grammar-module", Phase: "phase13-glossary-module",
Status: "success", Status: "success",
ModuleResults: []ModuleReport{ ModuleResults: []ModuleReport{
{ {
@@ -80,7 +80,7 @@ func TestProcessReportModuleResultsJSONSuccessAndSkipped(t *testing.T) {
func TestProcessReportModuleResultsJSONFailedModule(t *testing.T) { func TestProcessReportModuleResultsJSONFailedModule(t *testing.T) {
now := time.Now().UTC() now := time.Now().UTC()
report := ProcessReport{ report := ProcessReport{
Phase: "phase12-grammar-module", Phase: "phase13-glossary-module",
Status: "failed", Status: "failed",
ModuleResults: []ModuleReport{ ModuleResults: []ModuleReport{
{ {

View File

@@ -8,6 +8,7 @@ import (
"gitea.maximumdirect.net/eric/audita/internal/core/config" "gitea.maximumdirect.net/eric/audita/internal/core/config"
"gitea.maximumdirect.net/eric/audita/internal/core/schema" "gitea.maximumdirect.net/eric/audita/internal/core/schema"
"gitea.maximumdirect.net/eric/audita/internal/framework/contracts" "gitea.maximumdirect.net/eric/audita/internal/framework/contracts"
glossarymodule "gitea.maximumdirect.net/eric/audita/internal/modules/glossary"
grammarmodule "gitea.maximumdirect.net/eric/audita/internal/modules/grammar" grammarmodule "gitea.maximumdirect.net/eric/audita/internal/modules/grammar"
) )
@@ -70,6 +71,7 @@ func NewFactory(deps Dependencies) *Factory {
deps: deps, deps: deps,
constructors: make(map[string]Constructor, len(knownModuleKeys)), constructors: make(map[string]Constructor, len(knownModuleKeys)),
} }
_ = factory.RegisterConstructor(ModuleKeyGlossary, constructGlossaryModule)
_ = factory.RegisterConstructor(ModuleKeyGrammar, constructGrammarModule) _ = factory.RegisterConstructor(ModuleKeyGrammar, constructGrammarModule)
return factory return factory
} }
@@ -96,6 +98,12 @@ func constructGrammarModule(ctx context.Context, req ConstructRequest) (contract
return grammarmodule.New() return grammarmodule.New()
} }
func constructGlossaryModule(ctx context.Context, req ConstructRequest) (contracts.TranscriptModule, error) {
_ = ctx
_ = req
return glossarymodule.New()
}
// ModuleForSpec resolves one configured run spec into a module instance. // ModuleForSpec resolves one configured run spec into a module instance.
func (f *Factory) ModuleForSpec(spec contracts.ModuleRunSpec) (contracts.TranscriptModule, error) { func (f *Factory) ModuleForSpec(spec contracts.ModuleRunSpec) (contracts.TranscriptModule, error) {
if f == nil { if f == nil {

View File

@@ -66,7 +66,7 @@ func TestUnsupportedUnknownModuleKeyFailsCleanly(t *testing.T) {
func TestRecognizedButUnimplementedModuleKeyFailsCleanly(t *testing.T) { func TestRecognizedButUnimplementedModuleKeyFailsCleanly(t *testing.T) {
factory := NewFactory(Dependencies{}) factory := NewFactory(Dependencies{})
for _, key := range []string{ModuleKeyGlossary, ModuleKeyHomophones, ModuleKeySpokenWord} { for _, key := range []string{ModuleKeyHomophones, ModuleKeySpokenWord} {
t.Run(key, func(t *testing.T) { t.Run(key, func(t *testing.T) {
_, err := factory.ModuleForSpec(contracts.ModuleRunSpec{ModuleKey: key, InstanceName: key}) _, err := factory.ModuleForSpec(contracts.ModuleRunSpec{ModuleKey: key, InstanceName: key})
if err == nil { if err == nil {
@@ -95,6 +95,17 @@ func TestGrammarIsRegisteredAndConstructibleByDefault(t *testing.T) {
} }
} }
func TestGlossaryIsRegisteredAndConstructibleByDefault(t *testing.T) {
factory := NewFactory(Dependencies{})
module, err := factory.ModuleForSpec(contracts.ModuleRunSpec{ModuleKey: ModuleKeyGlossary, InstanceName: ModuleKeyGlossary})
if err != nil {
t.Fatalf("ModuleForSpec error: %v", err)
}
if module.Key() != ModuleKeyGlossary {
t.Fatalf("expected glossary module key, got %q", module.Key())
}
}
func TestRegisterConstructorAndConstruct(t *testing.T) { func TestRegisterConstructorAndConstruct(t *testing.T) {
factory := NewFactory(Dependencies{}) factory := NewFactory(Dependencies{})
if err := factory.RegisterConstructor(ModuleKeyGlossary, func(ctx context.Context, req ConstructRequest) (contracts.TranscriptModule, error) { if err := factory.RegisterConstructor(ModuleKeyGlossary, func(ctx context.Context, req ConstructRequest) (contracts.TranscriptModule, error) {

View File

@@ -686,3 +686,37 @@ func TestRunnerGrammarModuleUsesConfidenceThreshold(t *testing.T) {
t.Fatalf("expected low confidence reason, got %+v", out.ModuleResults[0].ValidatorRejected[0]) t.Fatalf("expected low confidence reason, got %+v", out.ModuleResults[0].ValidatorRejected[0])
} }
} }
func TestRunnerGlossaryModuleUsesConfidenceThreshold(t *testing.T) {
client := &fakeProposalStructuredClient{
responses: []proposal_generation.StructuredCorrectionSet{
{
Corrections: []proposal_generation.StructuredCorrectionProposal{
{TargetSegmentID: 1, OriginalText: "gestures", CorrectedText: "Jesters", Confidence: 0.5},
},
},
},
}
cfg := config.Default()
cfg.Thresholds.Glossary = 0.9
factory := modules.NewFactory(modules.Dependencies{})
out, err := New(factory).Run(context.Background(), RunInput{
Config: &cfg,
Transcript: &schema.Transcript{Segments: []schema.Segment{{ID: 1, Text: "There were gestures"}}},
Glossary: &schema.Glossary{Entries: []schema.GlossaryEntry{{Name: "Jesters", Category: "faction", Summary: "Faction"}}},
ModuleSpecs: []contracts.ModuleRunSpec{{ModuleKey: "glossary", InstanceName: "glossary"}},
ProposalLLMClient: client,
})
if err != nil {
t.Fatalf("Run error: %v", err)
}
if out.FinalTranscript.Segments[0].Text != "There were gestures" {
t.Fatalf("expected no changes due to glossary confidence threshold, got %q", out.FinalTranscript.Segments[0].Text)
}
if len(out.ModuleResults) != 1 || len(out.ModuleResults[0].ValidatorRejected) == 0 {
t.Fatalf("expected validator rejection, got %+v", out.ModuleResults)
}
if out.ModuleResults[0].ValidatorRejected[0].ReasonCode != validators.ReasonLowConfidence {
t.Fatalf("expected low confidence reason, got %+v", out.ModuleResults[0].ValidatorRejected[0])
}
}

View File

@@ -4,8 +4,6 @@ import (
"context" "context"
"fmt" "fmt"
"strings" "strings"
"gitea.maximumdirect.net/eric/audita/internal/framework/proposals"
) )
type ConfidenceThresholdValidator struct{} type ConfidenceThresholdValidator struct{}
@@ -110,11 +108,12 @@ func (v ProtectedGlossaryTermValidator) Validate(_ context.Context, req Request)
return Result{ValidatorName: v.Name(), Decisions: decisions}, nil return Result{ValidatorName: v.Name(), Decisions: decisions}, nil
} }
terms := glossaryTerms(req) vocab := NewProtectedVocabulary(req.Glossary)
decisions := make([]Decision, 0, len(req.CandidateProposal)) decisions := make([]Decision, 0, len(req.CandidateProposal))
for _, c := range req.CandidateProposal { for _, c := range req.CandidateProposal {
if altersProtectedTerm(c.CorrectionProposal, terms) { reason := vocab.violationReason(c.OriginalText, c.CorrectedText)
decisions = append(decisions, rejection(c.ProposalIndex, ReasonProtectedGlossaryTerm, "proposal may alter protected glossary terminology")) if reason != "" {
decisions = append(decisions, rejection(c.ProposalIndex, ReasonProtectedGlossaryTerm, reason))
continue continue
} }
decisions = append(decisions, approval(c.ProposalIndex)) decisions = append(decisions, approval(c.ProposalIndex))
@@ -125,37 +124,25 @@ func (v ProtectedGlossaryTermValidator) Validate(_ context.Context, req Request)
return Result{ValidatorName: v.Name(), Decisions: decisions}, nil return Result{ValidatorName: v.Name(), Decisions: decisions}, nil
} }
func glossaryTerms(req Request) []string { type GlossaryStageProtectedGlossaryTermValidator struct{}
if req.Glossary == nil {
return nil func (v GlossaryStageProtectedGlossaryTermValidator) Name() string {
} return "glossary_stage_protected_glossary_terms"
out := make([]string, 0)
for _, e := range req.Glossary.Entries {
if t := strings.TrimSpace(strings.ToLower(e.Name)); t != "" {
out = append(out, t)
}
for _, a := range e.Aliases {
if t := strings.TrimSpace(strings.ToLower(a)); t != "" {
out = append(out, t)
}
}
if t := strings.TrimSpace(strings.ToLower(e.Plural)); t != "" {
out = append(out, t)
}
}
return out
} }
func altersProtectedTerm(p proposals.CorrectionProposal, terms []string) bool { func (v GlossaryStageProtectedGlossaryTermValidator) Validate(_ context.Context, req Request) (Result, error) {
if len(terms) == 0 { vocab := NewProtectedVocabulary(req.Glossary)
return false decisions := make([]Decision, 0, len(req.CandidateProposal))
} for _, c := range req.CandidateProposal {
orig := strings.ToLower(p.OriginalText) reason := vocab.glossaryStageViolationReason(c.OriginalText, c.CorrectedText)
corr := strings.ToLower(p.CorrectedText) if reason != "" {
for _, t := range terms { decisions = append(decisions, rejection(c.ProposalIndex, ReasonProtectedGlossaryTerm, reason))
if strings.Contains(orig, t) && !strings.Contains(corr, t) { continue
return true
} }
decisions = append(decisions, approval(c.ProposalIndex))
} }
return false if err := EnforceDecisionCardinality(req.CandidateProposal, decisions); err != nil {
return Result{}, err
}
return Result{ValidatorName: v.Name(), Decisions: decisions}, nil
} }

View File

@@ -0,0 +1,188 @@
package validators
import (
"regexp"
"sort"
"strings"
"gitea.maximumdirect.net/eric/audita/internal/core/schema"
)
type protectedTermDef struct {
identity int
canonical string
}
type protectedOccurrence struct {
text string
identity int
canonical string
}
// ProtectedVocabulary is a deterministic glossary-derived protected term set.
type ProtectedVocabulary struct {
termsByFolded map[string]protectedTermDef
pattern *regexp.Regexp
}
// ExtractProtectedTerms returns a stable, de-duplicated list of protected terms
// derived from glossary names, aliases, synthetic plural forms, and explicit
// plural fields.
func ExtractProtectedTerms(glossary *schema.Glossary) []string {
vocab := NewProtectedVocabulary(glossary)
out := make([]string, 0, len(vocab.termsByFolded))
for _, def := range vocab.termsByFolded {
out = append(out, def.canonical)
}
sort.SliceStable(out, func(i, j int) bool {
li := strings.ToLower(out[i])
lj := strings.ToLower(out[j])
if li == lj {
return out[i] < out[j]
}
return li < lj
})
return out
}
// NewProtectedVocabulary builds a deterministic protected vocabulary from a glossary.
func NewProtectedVocabulary(glossary *schema.Glossary) ProtectedVocabulary {
termsByFolded := make(map[string]protectedTermDef)
if glossary != nil {
for identity, entry := range glossary.Entries {
entryTerms := append([]string{entry.Name}, entry.Aliases...)
for _, term := range entryTerms {
trimmed := strings.TrimSpace(term)
if trimmed == "" {
continue
}
addProtectedTerm(termsByFolded, trimmed, identity)
addProtectedTerm(termsByFolded, trimmed+"s", identity)
}
addProtectedTerm(termsByFolded, entry.Plural, identity)
}
}
alternatives := make([]string, 0, len(termsByFolded))
for _, def := range termsByFolded {
alternatives = append(alternatives, regexp.QuoteMeta(def.canonical))
}
sort.SliceStable(alternatives, func(i, j int) bool { return len(alternatives[i]) > len(alternatives[j]) })
if len(alternatives) == 0 {
return ProtectedVocabulary{termsByFolded: termsByFolded}
}
pattern := regexp.MustCompile(`(?i)\b(?:` + strings.Join(alternatives, "|") + `)\b`)
return ProtectedVocabulary{termsByFolded: termsByFolded, pattern: pattern}
}
func addProtectedTerm(terms map[string]protectedTermDef, term string, identity int) {
trimmed := strings.TrimSpace(term)
if trimmed == "" {
return
}
folded := strings.ToLower(trimmed)
if _, ok := terms[folded]; ok {
return
}
terms[folded] = protectedTermDef{identity: identity, canonical: trimmed}
}
func (v ProtectedVocabulary) violationReason(before, after string) string {
beforeByID := v.occurrencesByIdentity(before)
afterByID := v.occurrencesByIdentity(after)
if reason := validateIdentityPreservation(beforeByID, afterByID); reason != "" {
return reason
}
if reason := validateCapitalizationTransitions(beforeByID, afterByID); reason != "" {
return reason
}
return ""
}
func (v ProtectedVocabulary) glossaryStageViolationReason(before, after string) string {
beforeByID := v.occurrencesByIdentity(before)
afterByID := v.occurrencesByIdentity(after)
if reason := validateGlossaryStageIdentityPreservation(beforeByID, afterByID); reason != "" {
return reason
}
if reason := validateCapitalizationTransitions(beforeByID, afterByID); reason != "" {
return reason
}
return ""
}
func (v ProtectedVocabulary) occurrencesByIdentity(text string) map[int][]protectedOccurrence {
byID := make(map[int][]protectedOccurrence)
for _, o := range v.occurrences(text) {
byID[o.identity] = append(byID[o.identity], o)
}
return byID
}
func (v ProtectedVocabulary) occurrences(text string) []protectedOccurrence {
if v.pattern == nil {
return nil
}
matches := v.pattern.FindAllStringIndex(text, -1)
if len(matches) == 0 {
return nil
}
out := make([]protectedOccurrence, 0, len(matches))
for _, idx := range matches {
matched := text[idx[0]:idx[1]]
def, ok := v.termsByFolded[strings.ToLower(matched)]
if !ok {
continue
}
out = append(out, protectedOccurrence{
text: matched,
identity: def.identity,
canonical: def.canonical,
})
}
return out
}
func validateIdentityPreservation(beforeByID, afterByID map[int][]protectedOccurrence) string {
for identity, beforeItems := range beforeByID {
if len(afterByID[identity]) < len(beforeItems) {
return "proposal may alter protected glossary terminology"
}
}
return ""
}
func validateGlossaryStageIdentityPreservation(beforeByID, afterByID map[int][]protectedOccurrence) string {
beforeTotal := 0
for _, items := range beforeByID {
beforeTotal += len(items)
}
afterTotal := 0
for _, items := range afterByID {
afterTotal += len(items)
}
if afterTotal < beforeTotal {
return "proposal may alter protected glossary terminology"
}
return ""
}
func validateCapitalizationTransitions(beforeByID, afterByID map[int][]protectedOccurrence) string {
for identity, afterItems := range afterByID {
beforeItems := beforeByID[identity]
for i, afterItem := range afterItems {
if i >= len(beforeItems) {
continue
}
beforeItem := beforeItems[i]
if afterItem.text == beforeItem.text {
continue
}
if afterItem.text == afterItem.canonical {
continue
}
return "proposal may alter protected glossary terminology"
}
}
return ""
}

View File

@@ -0,0 +1,86 @@
package validators
import (
"context"
"reflect"
"testing"
"gitea.maximumdirect.net/eric/audita/internal/core/schema"
"gitea.maximumdirect.net/eric/audita/internal/framework/proposals"
)
func TestExtractProtectedTermsStableDeduplicatedWithPlurals(t *testing.T) {
glossary := &schema.Glossary{
Entries: []schema.GlossaryEntry{
{Name: "Jesters", Aliases: []string{"Jester", " "}, Plural: "Jesters", Category: "faction", Summary: "Faction"},
{Name: "Hrank", Aliases: []string{"hrank", "Hrank"}, Plural: "Hranks", Category: "pc", Summary: "Character"},
{Name: "Lyra", Aliases: []string{}, Plural: "", Category: "npc", Summary: "NPC"},
},
}
got := ExtractProtectedTerms(glossary)
want := []string{"Hrank", "Hranks", "Jester", "Jesters", "Jesterss", "Lyra", "Lyras"}
if !reflect.DeepEqual(got, want) {
t.Fatalf("unexpected protected terms\n got: %v\nwant: %v", got, want)
}
}
func TestExtractProtectedTermsEmptyGlossary(t *testing.T) {
if terms := ExtractProtectedTerms(&schema.Glossary{}); len(terms) != 0 {
t.Fatalf("expected no terms, got %v", terms)
}
}
func TestProtectedGlossaryTermValidatorAppliesToNonGlossaryModule(t *testing.T) {
req := Request{
ModuleKey: "grammar",
Glossary: &schema.Glossary{
Entries: []schema.GlossaryEntry{{Name: "Jesters", Category: "faction", Summary: "Faction"}},
},
CandidateProposal: []proposals.EnrichedCorrectionProposal{
{
CorrectionProposal: proposals.CorrectionProposal{
TargetSegmentID: 1,
OriginalText: "Jesters",
CorrectedText: "Gestures",
Confidence: 0.9,
},
ProposalMetadata: proposals.ProposalMetadata{ProposalIndex: 0},
},
},
}
res, err := (ProtectedGlossaryTermValidator{}).Validate(context.Background(), req)
if err != nil {
t.Fatalf("validator error: %v", err)
}
if len(res.Decisions) != 1 || res.Decisions[0].Approved {
t.Fatalf("expected protected-term rejection, got %+v", res.Decisions)
}
}
func TestGlossaryStageProtectedGlossaryTermValidatorAllowsProtectedTermSwap(t *testing.T) {
req := Request{
ModuleKey: "glossary",
Glossary: &schema.Glossary{
Entries: []schema.GlossaryEntry{{Name: "Jesters", Category: "faction", Summary: "Faction"}},
},
CandidateProposal: []proposals.EnrichedCorrectionProposal{
{
CorrectionProposal: proposals.CorrectionProposal{
TargetSegmentID: 1,
OriginalText: "gestures",
CorrectedText: "Jesters",
Confidence: 0.9,
},
ProposalMetadata: proposals.ProposalMetadata{ProposalIndex: 0},
},
},
}
res, err := (GlossaryStageProtectedGlossaryTermValidator{}).Validate(context.Background(), req)
if err != nil {
t.Fatalf("validator error: %v", err)
}
if len(res.Decisions) != 1 || !res.Decisions[0].Approved {
t.Fatalf("expected approval for glossary-stage protected term correction, got %+v", res.Decisions)
}
}

View File

@@ -0,0 +1,76 @@
package glossary
import (
"context"
"fmt"
"gitea.maximumdirect.net/eric/audita/internal/framework/contracts"
"gitea.maximumdirect.net/eric/audita/internal/framework/proposal_generation"
"gitea.maximumdirect.net/eric/audita/internal/framework/proposals"
"gitea.maximumdirect.net/eric/audita/internal/framework/validators"
)
type Module struct {
validators []contracts.Validator
}
func New() (*Module, error) {
spokenForm, err := validators.NewLLMBackedValidator("spoken_form_plausibility_review", validators.LLMValidatorTypeSpokenFormPlausibility, "")
if err != nil {
return nil, err
}
meaningReversal, err := validators.NewLLMBackedValidator("meaning_reversal_review", validators.LLMValidatorTypeMeaningReversal, "")
if err != nil {
return nil, err
}
return &Module{
validators: []contracts.Validator{
validators.NoEffectValidator{},
validators.OriginalTextPresenceValidator{},
validators.ConfidenceThresholdValidator{},
validators.GlossaryStageProtectedGlossaryTermValidator{},
validators.NonEmptyCorrectionValidator{},
spokenForm,
meaningReversal,
},
}, nil
}
func (m *Module) Key() string { return "glossary" }
func (m *Module) ReplacementPolicy() proposals.ReplacementPolicy {
// Python glossary module uses replace_all to update repeated term occurrences.
return proposals.ReplacementPolicyReplaceAll
}
func (m *Module) Validators() []contracts.Validator {
return append([]contracts.Validator(nil), m.validators...)
}
func (m *Module) Propose(ctx context.Context, req contracts.ProposalRequest) ([]proposals.CorrectionProposal, error) {
messages, err := BuildProposalMessages(req.WorkingTranscript, req.Glossary)
if err != nil {
return nil, err
}
generated, err := proposal_generation.GenerateCandidates(ctx, proposal_generation.Request{
ModuleKey: req.RunSpec.ModuleKey,
ModuleInstance: req.RunSpec.InstanceName,
ReplacementPolicy: req.RunSpec.ReplacementPolicy,
WorkingTranscript: req.WorkingTranscript,
Section: req.Section,
Glossary: req.Glossary,
Config: req.Config,
Messages: messages,
StageName: fmt.Sprintf("%s:proposal", req.RunSpec.InstanceName),
StartIndex: 0,
LLMClient: req.LLMClient,
Scheduler: req.LLMScheduler,
DiagnosticsDir: req.DiagnosticsDir,
})
if err != nil {
return nil, err
}
return generated.Corrections, nil
}

View File

@@ -0,0 +1,188 @@
package glossary
import (
"context"
"errors"
"os"
"path/filepath"
"strings"
"testing"
"gitea.maximumdirect.net/eric/audita/internal/core/config"
"gitea.maximumdirect.net/eric/audita/internal/core/schema"
"gitea.maximumdirect.net/eric/audita/internal/framework/contracts"
"gitea.maximumdirect.net/eric/audita/internal/framework/proposal_generation"
"gitea.maximumdirect.net/eric/audita/internal/framework/proposals"
)
type fakeLLMClient struct {
responses []proposal_generation.StructuredCorrectionSet
err error
calls []contracts.StructuredCompletionRequest
}
func (f *fakeLLMClient) CompleteStructured(ctx context.Context, req contracts.StructuredCompletionRequest, out any) (contracts.StructuredCompletionResponse, error) {
_ = ctx
f.calls = append(f.calls, req)
if f.err != nil {
return contracts.StructuredCompletionResponse{}, f.err
}
target, ok := out.(*proposal_generation.StructuredCorrectionSet)
if !ok {
return contracts.StructuredCompletionResponse{}, errors.New("unexpected output type")
}
if len(f.responses) == 0 {
return contracts.StructuredCompletionResponse{}, errors.New("unexpected call")
}
*target = f.responses[0]
f.responses = f.responses[1:]
return contracts.StructuredCompletionResponse{}, nil
}
type countingScheduler struct {
runs int
}
func (s *countingScheduler) Run(ctx context.Context, fn func(context.Context) error) error {
s.runs++
return fn(ctx)
}
func tinyTranscript() *schema.Transcript {
return &schema.Transcript{Segments: []schema.Segment{
{ID: 1, Speaker: "A", Start: 0, End: 1, Text: "There were gestures in the hall.", Categories: []string{"session"}},
}}
}
func tinyGlossary() *schema.Glossary {
return &schema.Glossary{Entries: []schema.GlossaryEntry{
{Name: "Jesters", Aliases: []string{"Jester"}, Category: "faction", Summary: "Guild members", Plural: "Jesters"},
}}
}
func TestBuildProposalMessagesContainsGlossaryContextAndConstraints(t *testing.T) {
msgs, err := BuildProposalMessages(tinyTranscript(), tinyGlossary())
if err != nil {
t.Fatalf("BuildProposalMessages error: %v", err)
}
if len(msgs) != 2 {
t.Fatalf("expected 2 messages, got %d", len(msgs))
}
combined := msgs[0].Content + "\n" + msgs[1].Content
for _, want := range []string{
"Glossary:",
"Transcript section:",
`"Aliases":`,
`"Jester"`,
`"Category": "faction"`,
`"Summary": "Guild members"`,
`"Plural": "Jesters"`,
"glossary-supported corrections",
"acoustically similar",
"Do not make generic grammar, spelling, capitalization, style, or filler-word edits",
} {
if !strings.Contains(combined, want) {
t.Fatalf("expected prompt to contain %q", want)
}
}
if strings.Contains(strings.ToLower(combined), "summarize the transcript") {
t.Fatalf("prompt should not invite summarization")
}
}
func TestGlossaryModuleReplacementPolicy(t *testing.T) {
m, err := New()
if err != nil {
t.Fatalf("New error: %v", err)
}
if m.ReplacementPolicy() != proposals.ReplacementPolicyReplaceAll {
t.Fatalf("unexpected replacement policy: %q", m.ReplacementPolicy())
}
}
func TestGlossaryModuleValidatorChain(t *testing.T) {
m, err := New()
if err != nil {
t.Fatalf("New error: %v", err)
}
got := make([]string, 0, len(m.Validators()))
for _, v := range m.Validators() {
got = append(got, v.Name())
}
want := []string{
"no_effect",
"original_text_presence",
"confidence_threshold",
"glossary_stage_protected_glossary_terms",
"non_empty_correction",
"spoken_form_plausibility_review",
"meaning_reversal_review",
}
if strings.Join(got, ",") != strings.Join(want, ",") {
t.Fatalf("unexpected validator chain\n got: %v\nwant: %v", got, want)
}
}
func TestGlossaryModuleProposeMapsCorrectionsAndWritesDiagnostics(t *testing.T) {
secret := "glossary-secret"
client := &fakeLLMClient{
responses: []proposal_generation.StructuredCorrectionSet{
{
Corrections: []proposal_generation.StructuredCorrectionProposal{
{TargetSegmentID: 1, OriginalText: "gestures", CorrectedText: "Jesters", Confidence: 0.94},
},
},
},
}
scheduler := &countingScheduler{}
cfg := config.Default()
cfg.PrimaryLLM.APIKey = secret
m, err := New()
if err != nil {
t.Fatalf("New error: %v", err)
}
diagDir := t.TempDir()
out, err := m.Propose(context.Background(), contracts.ProposalRequest{
ExecutionContext: contracts.ExecutionContext{
Config: &cfg,
WorkingTranscript: tinyTranscript(),
Glossary: tinyGlossary(),
DiagnosticsDir: diagDir,
},
RunSpec: contracts.ModuleRunSpec{
ModuleKey: "glossary",
InstanceName: "glossary",
ReplacementPolicy: proposals.ReplacementPolicyReplaceAll,
},
LLMClient: client,
LLMScheduler: scheduler,
})
if err != nil {
t.Fatalf("Propose error: %v", err)
}
if scheduler.runs != 1 {
t.Fatalf("expected scheduler run count 1, got %d", scheduler.runs)
}
if len(client.calls) != 1 || client.calls[0].StageName != "glossary:proposal" {
t.Fatalf("expected one glossary:proposal call, got %+v", client.calls)
}
if len(out) != 1 || out[0].CorrectedText != "Jesters" {
t.Fatalf("unexpected proposals: %+v", out)
}
diagFiles, globErr := filepath.Glob(filepath.Join(diagDir, "glossary", "*proposal*response-payload.json"))
if globErr != nil {
t.Fatalf("glob error: %v", globErr)
}
if len(diagFiles) == 0 {
t.Fatalf("expected diagnostics response payload under %s", filepath.Join(diagDir, "glossary"))
}
for _, f := range diagFiles {
raw, readErr := os.ReadFile(f)
if readErr != nil {
t.Fatalf("read diag %q: %v", f, readErr)
}
if strings.Contains(string(raw), secret) {
t.Fatalf("secret leaked in diagnostics: %s", string(raw))
}
}
}

View File

@@ -0,0 +1,80 @@
package glossary
import (
"encoding/json"
"fmt"
"gitea.maximumdirect.net/eric/audita/internal/core/schema"
"gitea.maximumdirect.net/eric/audita/internal/framework/contracts"
)
type promptSegment struct {
ID int `json:"id"`
Speaker string `json:"speaker"`
Start float64 `json:"start"`
End float64 `json:"end"`
Text string `json:"text"`
Categories []string `json:"categories,omitempty"`
}
type promptTranscriptSection struct {
SectionIndex int `json:"section_index"`
Segments []promptSegment `json:"segments"`
}
func BuildProposalMessages(transcript *schema.Transcript, glossary *schema.Glossary) ([]contracts.LLMMessage, error) {
glossaryJSON, err := json.MarshalIndent(glossary, "", " ")
if err != nil {
return nil, fmt.Errorf("marshal glossary prompt context: %w", err)
}
sectionPayload := promptTranscriptSection{
SectionIndex: 0,
Segments: make([]promptSegment, 0),
}
if transcript != nil {
for _, s := range transcript.Segments {
sectionPayload.Segments = append(sectionPayload.Segments, promptSegment{
ID: s.ID,
Speaker: s.Speaker,
Start: s.Start,
End: s.End,
Text: s.Text,
Categories: append([]string(nil), s.Categories...),
})
}
}
sectionJSON, err := json.MarshalIndent(sectionPayload, "", " ")
if err != nil {
return nil, fmt.Errorf("marshal transcript prompt context: %w", err)
}
system := "You are Audita, a careful transcript correction assistant. Identify only transcription errors that are strongly supported by the glossary. A valid correction must be acoustically plausible: the original transcript text should sound similar to the proposed correction when spoken aloud. Do not make generic grammar, spelling, capitalization, style, or filler-word edits. Do not substitute an unrelated glossary term just because it could fit the topic. Preserve speaker names, timestamps, and meaning."
user := "Review this transcript section and return only glossary-supported corrections that should be applied.\n\n" +
"Rules:\n" +
"- Correct domain-specific names, aliases, jargon, deities, locations, NPCs, players, factions, and similar terms only when both the glossary and surrounding transcript context support the correction.\n" +
"- The correction must plausibly fix a transcription error: the original words should be phonetically or acoustically similar to the corrected words in spoken English.\n" +
"- Appropriate example: correcting \"gestures\" to \"Jesters\" can be valid if \"Jesters\" appears in the glossary and nearby context supports that inference.\n" +
"- Inappropriate example: correcting \"Lyra\" to \"Jesters\" should be omitted because those words are not similar in spoken English, even if \"Jesters\" appears in the glossary.\n" +
"- Do not replace one clear glossary term, character name, location, or ordinary word with a different glossary term unless it is a plausible mishearing.\n" +
"- Treat glossary names and aliases already present in the transcript as protected spellings.\n" +
"- Do not replace, Anglicize, normalize, lowercase, or otherwise alter protected glossary names or aliases away from their glossary spelling.\n" +
"- Preserve canonical glossary capitalization for protected names and aliases, even if they look unusual.\n" +
"- If a segment includes categories, treat them as additional transcript context.\n" +
"- Plural forms of glossary names and aliases are allowed targets when spoken similarity and context support them, even if the plural is not explicitly listed in the glossary.\n" +
"- Use the exact id from the input segment.\n" +
"- For returned corrections, original_text must be only the exact text span that needs replacement, not the full segment text unless the whole segment is the replacement span.\n" +
"- corrected_text must be only the replacement text for that span, not the full corrected segment text unless the whole segment is the replacement span.\n" +
"- Each returned correction must contain only id, original_text, corrected_text, and confidence.\n" +
"- Do not return corrections where original_text and corrected_text are identical.\n" +
"- Do not return speaker, start, or end fields.\n" +
"- Return only changed segments; do not return entries for unchanged segments.\n" +
"- confidence must be between 0.0 and 1.0.\n" +
"- If no corrections are needed, return an empty corrections list.\n\n" +
fmt.Sprintf("Glossary:\n%s\n\nTranscript section:\n%s", string(glossaryJSON), string(sectionJSON))
return []contracts.LLMMessage{
{Role: "system", Content: system},
{Role: "user", Content: user},
}, nil
}