diff --git a/docs/architecture.md b/docs/architecture.md index 0fe5c45..589c56a 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -34,19 +34,22 @@ Implemented today: - Proposal-generation diagnostics artifacts with secret redaction. - Production module registry scaffolding with known-key recognition and explicit unsupported/unimplemented errors. - Production `grammar` module implementation in `internal/modules/grammar`. +- Production `glossary` module implementation in `internal/modules/glossary`. - Explicit runtime support for `--modules grammar` through the production runner path. +- Explicit runtime support for `--modules glossary`, including repeated stages such as `--modules glossary,glossary`. Not implemented in CLI runtime path today: -- Real module execution pipeline for `glossary`, `homophones`, and `spoken_word`. -- Real domain proposal prompts for production modules. -- End-to-end transcript polishing with real module behavior. +- Real module execution pipeline for `homophones` and `spoken_word`. +- Real domain proposal prompts for remaining production modules. +- End-to-end transcript polishing with the full default module sequence. Phase sequencing note: - Phase 9 LLM infrastructure is complete (structured client, scheduler, effective config resolution, diagnostics primitives); - Phase 10 LLM-backed validator runtime integration is complete; - Phase 11 shared proposal-generation framework and module-registry scaffolding are complete; - Phase 12 grammar module implementation and explicit runtime wiring are complete; -- next recommended phase is Phase 13 (glossary module and protected-term behavior). +- Phase 13 glossary module and protected-term behavior are complete; +- next recommended phase is Phase 14 (homophones module). ## Actual Go package layout @@ -109,6 +112,10 @@ internal/modules/grammar/ module.go prompt.go +internal/modules/glossary/ + module.go + prompt.go + internal/framework/validators/ models.go deterministic.go @@ -144,17 +151,17 @@ Current runtime flow (`internal/cli/run.go`): 10. Chunk normalized transcript and compute chunk summaries. 11. Write chunking summary artifact. 12. Execute runner modules sequentially when: - - `--modules` is explicitly provided (production grammar path); or + - `--modules` is explicitly provided (production grammar/glossary paths); or - a test/injected module factory is provided. 13. Output working transcript to `--output` file or stdout. -14. Build process report (`phase` currently set to `phase12-grammar-module`). +14. Build process report (`phase` currently set to `phase13-glossary-module`). 15. Optionally write `--report-json`; always write run-dir `report.json`. 16. Apply work-dir retention. Important behavior details: -- Glossary is validated but not yet used for real correction module logic. +- Glossary is validated and is used for explicit glossary/grammar module correction paths. - Default production CLI behavior remains deterministic normalization/chunking/reporting unless modules are explicitly selected with `--modules`. -- Explicit `--modules grammar` runs the production grammar module path with LLM-backed proposal generation and validator-chain execution. +- Explicit `--modules grammar` and `--modules glossary` run production module paths with LLM-backed proposal generation and validator-chain execution. - Default runs (without explicit module selection) do not perform LLM calls. - Success path is generally quiet on stderr. - Source IDs are preserved into a canonical transcript before normalization; normalization then reassigns output IDs sequentially from `1`. @@ -203,7 +210,7 @@ Implemented config surfaces include: - work-dir and retention mode Current caveat: -- LLM/module-related settings are active for explicit grammar runs; the default non-explicit path remains deterministic. +- LLM/module-related settings are active for explicit grammar/glossary runs; the default non-explicit path remains deterministic. ## Implemented structured LLM infrastructure `internal/framework/contracts` now defines a typed structured-completion contract: @@ -220,7 +227,7 @@ Current caveat: Current runtime boundary: - the default CLI runtime path (without explicit module selection) still does not instantiate the full production module sequence. -- LLM calls are exercised in production when `--modules grammar` is explicitly requested and in tests when fake/injected clients are used. +- LLM calls are exercised in production when `--modules grammar` or `--modules glossary` is explicitly requested and in tests when fake/injected clients are used. `internal/framework/llm` also provides: - a bounded `Scheduler` for controlled concurrent LLM calls with reliable permit release; @@ -262,7 +269,7 @@ Current behavior details: `internal/framework/contracts` provides interfaces and run-spec metadata scaffolding, including deterministic repeated module instance naming (`ResolveModuleRunSpecs`). -These primitives are wired into the production runner and report model. The grammar module is implemented; other production modules remain pending. +These primitives are wired into the production runner and report model. The grammar and glossary modules are implemented; other production modules remain pending. ## Implemented validator runtime infrastructure `internal/framework/validators` provides deterministic validator infrastructure: @@ -327,7 +334,7 @@ This helper only produces candidate proposals; validator-chain execution and pro - diagnostics directory context - returns explicit errors for unknown keys (`unsupported_module`) and recognized-but-unimplemented keys (`unimplemented_module`). -The `grammar` module key is now registered and constructible. `glossary`, `homophones`, and `spoken_word` remain recognized-but-unimplemented. +The `grammar` and `glossary` module keys are now registered and constructible. `homophones` and `spoken_word` remain recognized-but-unimplemented. ## Implemented grammar production module `internal/modules/grammar` now provides the first production module: @@ -340,6 +347,27 @@ The `grammar` module key is now registered and constructible. `glossary`, `homop - grammar confidence threshold enforcement through existing validator/config infrastructure; - module-level reporting and diagnostics capture through existing runner/reporting paths. +## Implemented glossary production module +`internal/modules/glossary` now provides the second production module: +- prompt builder aligned to Python glossary-module intent, constrained to glossary-backed domain/acoustic corrections; +- prompt context includes glossary names, aliases, categories, summaries, and plural forms where available; +- guardrails against broad style rewriting and against replacing unrelated terms simply because they appear in glossary entries; +- proposal generation through `internal/framework/proposal_generation` and `contracts.StructuredLLMClient`; +- scheduler-aware proposal calls through existing `contracts.LLMScheduler` hooks; +- replacement policy `replace_all` (matching Python glossary behavior); +- validator chain integration using existing deterministic + LLM-backed validators; +- glossary confidence threshold enforcement through existing validator/config infrastructure; +- module-level reporting and diagnostics capture through existing runner/reporting paths; +- explicit support for repeated glossary stages with deterministic instance names (`glossary_1`, `glossary_2`, ...), where later stages see prior-stage working transcript changes. + +## Implemented protected-term behavior +`internal/framework/validators/protected_terms.go` provides deterministic glossary-derived protected vocabulary: +- extracts protected terms from glossary names and aliases; +- includes explicit plural fields and synthetic plural forms where safe; +- deduplicates and returns stable ordering for repeatable behavior/tests. + +This vocabulary is used by deterministic validators for both glossary-stage and non-glossary-stage protection checks, keeping protected-term guardrails active across modules. + ## Reports and diagnostics (implemented) Current per-run artifacts include: - `source-transcript.json` @@ -379,7 +407,7 @@ Retention modes implemented in `ApplyRetention`: Current runtime note: - default non-explicit runs usually have no module-level skipped corrections, so `auto` commonly removes clean successful run directories. -- explicit grammar runs can produce validator rejections and application skips, which are reflected in reports and retention input. +- explicit grammar/glossary runs can produce validator rejections and application skips, which are reflected in reports and retention input. Intentionally deferred to module/LLM phases: - real domain proposal prompts and production module implementations remain tied to later module phases. @@ -401,13 +429,15 @@ Implemented tests currently cover: - shared proposal-generation request/response parsing, deterministic indexing, scheduler hooks, and diagnostics redaction (`internal/framework/proposal_generation/*_test.go`, `internal/framework/runner/*_test.go`) - production module-registry known-key recognition and unsupported/unimplemented error behavior (`internal/framework/modules/*_test.go`, `internal/cli/run_test.go`) - production grammar module prompt constraints, proposal mapping, validator-chain behavior, confidence-threshold enforcement, diagnostics redaction, and explicit CLI/runtime integration (`internal/modules/grammar/*_test.go`, `internal/cli/run_test.go`, `internal/framework/runner/*_test.go`) +- production glossary module prompt constraints, proposal mapping, validator-chain behavior, confidence-threshold enforcement, diagnostics redaction, repeated-stage behavior, and explicit CLI/runtime integration (`internal/modules/glossary/*_test.go`, `internal/cli/run_test.go`, `internal/framework/runner/*_test.go`) +- glossary-derived protected-term extraction and stable behavior (`internal/framework/validators/protected_terms_test.go`) -Not covered yet (because not implemented): production `glossary`, `homophones`, and `spoken_word` modules plus full default-sequence transcript-polishing runtime behavior. +Not covered yet (because not implemented): production `homophones` and `spoken_word` modules plus full default-sequence transcript-polishing runtime behavior. ## Intended final architecture (not yet implemented) The intended end-state still matches the rewrite plan: - sequential module pipeline over a mutable working transcript -- real module implementations (`glossary`, `homophones`, `spoken_word`, `grammar`) +- real module implementations (`homophones`, `spoken_word`) - structured LLM proposal generation - deterministic and LLM validators - validator cardinality enforcement in pipeline execution diff --git a/docs/rewrite-notes.md b/docs/rewrite-notes.md index b3c2047..6b4d65a 100644 --- a/docs/rewrite-notes.md +++ b/docs/rewrite-notes.md @@ -61,6 +61,10 @@ Implemented: - Production module-registry scaffolding with known key recognition and explicit unsupported/unimplemented errors. - Production grammar module package with Python-aligned prompt intent and guardrails. - Explicit `--modules grammar` runtime path through runner, shared proposal generation, validators, application, reporting, and diagnostics. +- Production glossary module package with Python-aligned prompt intent and guardrails. +- Glossary-derived deterministic protected-term extraction and validator integration. +- Explicit `--modules glossary` runtime path through runner, shared proposal generation, validators, application, reporting, and diagnostics. +- Repeated glossary stage support with deterministic instance names (`glossary_1`, `glossary_2`), including mutable working-transcript handoff. - Broad deterministic and CLI/subprocess test coverage for implemented phases through `go test ./...`. - Internal typed structured LLM contract (`StructuredLLMClient.CompleteStructured(ctx, req, out)`). - `internal/framework/llm` instructor-go-backed adapter with: @@ -76,9 +80,9 @@ Implemented: - Generic JSON diagnostics primitives for LLM interactions (request metadata, request payload, response payload, optional error payload) with secret redaction. Not yet implemented in runtime pipeline: -- Real correction modules for `glossary`, `homophones`, and `spoken_word`. +- Real correction modules for `homophones` and `spoken_word`. - Domain proposal prompts for remaining real modules. -- End-to-end transcript polishing behavior. +- End-to-end transcript polishing behavior with the full default module sequence. ## Completed phases @@ -216,7 +220,7 @@ Not implemented in Phase 8 (by design): ## Remaining work plan -Next recommended phase: **Phase 13 (glossary module and protected-term behavior)**. +Next recommended phase: **Phase 14 (homophones module)**. ## Phase 9: Structured LLM client and scheduler infrastructure @@ -362,44 +366,28 @@ Not implemented in Phase 12 (by design): ## Phase 13: Glossary module and protected-term behavior -### Purpose +Completed. -Implement the glossary correction module and the glossary-derived protection behavior needed by downstream modules. +Implemented: +- Production glossary module package in `internal/modules/glossary`. +- Glossary prompt builder aligned to Python intent and constrained to glossary-supported domain/acoustic corrections. +- Prompt context using glossary names, aliases, categories, summaries, and plural forms where available. +- Glossary proposal generation through shared `internal/framework/proposal_generation` using `contracts.StructuredLLMClient`. +- Scheduler-aware glossary proposal generation through existing scheduler hooks. +- Glossary replacement policy `replace_all` (matching Python behavior). +- Glossary validator chain using existing deterministic and LLM-backed validators. +- Glossary confidence threshold enforcement through existing config + confidence-threshold validator behavior. +- Deterministic glossary-derived protected-term extraction (`internal/framework/validators/protected_terms.go`) from names, aliases, and plural forms, with stable deduplicated ordering. +- Protected-term validator behavior remaining available to non-glossary modules via existing deterministic validators. +- Explicit runtime support for `--modules glossary` through normalization, chunking, runner, proposal generation, validation, application, and reporting. +- Repeated glossary-stage support (`--modules glossary,glossary`) with deterministic instance naming and mutable working-transcript handoff across stages. +- Prompt/response diagnostics artifacts for glossary proposal + validator interactions with secret redaction. +- Module-level reports for glossary including generated proposals, validator decisions/rejections, applied changes, and application skips. +- CLI/runtime fake-client tests for approved proposals, validator rejection, application skips, repeated stages, diagnostics, failure/error.log behavior, and report outputs (`--report-json` and run-dir `report.json`). -### Scope - -Implement: -- `glossary` module package. -- Glossary prompt builder ported from Python. -- Glossary structured response model. -- Glossary replacement policy. -- Glossary confidence threshold handling. -- Glossary validator chain. -- Protected-term extraction from parsed glossary. -- Protected-term validator behavior used by other modules where applicable. -- Prompt/response diagnostics. -- CLI support for `--modules glossary`. -- Fake LLM tests. -- Tests for repeated glossary stages using `glossary,glossary`. - -Do not implement: -- Homophones module. -- Spoken-word module. -- Default full pipeline parity claim. - -### Expected behavior at end of phase - -Running `audita process ... --modules glossary` should perform real glossary-supported corrections. Repeated glossary stages should work and be reported as separate module instances. - -### Definition of done - -- Glossary module runs in the production runner. -- Glossary terms and aliases are used in prompts and validators. -- Protected-term behavior is implemented and tested. -- Repeated glossary module instances are reported correctly. -- Applied/skipped glossary changes appear in reports. -- Prompt/response diagnostics are written. -- `go test ./...` passes without requiring external LLM credentials. +Not implemented in Phase 13 (by design): +- Production `homophones` and `spoken_word` modules. +- Full default module sequence execution as a feature-complete claim. ## Phase 14: Homophones module diff --git a/internal/cli/run.go b/internal/cli/run.go index cc38fdd..f7c00ce 100644 --- a/internal/cli/run.go +++ b/internal/cli/run.go @@ -460,7 +460,7 @@ func extractErrorPhase(err error) (phase string, message string) { func buildProcessReport(status string, inv processInvocation, runDir *diagnostics.RunDirectory, startedAt, completedAt time.Time, errorMessage string, errorPhase string, normalizationSummary *normalization.NormalizationSummary, chunkingSummary *chunking.Summary, runOutput *runner.RunOutput) reporting.ProcessReport { report := reporting.ProcessReport{ - Phase: "phase12-grammar-module", + Phase: "phase13-glossary-module", Status: status, Operation: "process", TranscriptPath: inv.TranscriptPath, diff --git a/internal/cli/run_test.go b/internal/cli/run_test.go index 3aa54f3..3995242 100644 --- a/internal/cli/run_test.go +++ b/internal/cli/run_test.go @@ -616,8 +616,8 @@ func TestRunProcessReportJSONIncludesChunkingSummary(t *testing.T) { if report.Chunking.MaxSectionTokens == 0 { t.Errorf("expected max_section_tokens in report") } - if report.Phase != "phase12-grammar-module" { - t.Errorf("expected phase 'phase12-grammar-module', got %q", report.Phase) + if report.Phase != "phase13-glossary-module" { + t.Errorf("expected phase 'phase13-glossary-module', got %q", report.Phase) } } @@ -664,12 +664,13 @@ func (v fakeValidator) Validate(ctx context.Context, req contracts.ValidationReq type fakeStructuredLLMClient struct { validationResponses []validators.LLMValidationResponse proposalResponses []proposal_generation.StructuredCorrectionSet + calls []string err error } func (f *fakeStructuredLLMClient) CompleteStructured(ctx context.Context, req contracts.StructuredCompletionRequest, out any) (contracts.StructuredCompletionResponse, error) { _ = ctx - _ = req + f.calls = append(f.calls, req.StageName) if f.err != nil { return contracts.StructuredCompletionResponse{}, f.err } @@ -923,7 +924,7 @@ func TestRunProcessProductionRegistryUnimplementedModuleFailsCleanly(t *testing. exitCode := Run([]string{ "process", transcriptPath, "--glossary", fixturePath("tiny_glossary.yaml"), - "--modules", "glossary", + "--modules", "homophones", "--work-dir", workDir, "--work-dir-retention", "always", "--report-json", reportPath, @@ -954,7 +955,7 @@ func TestRunProcessProductionRegistryUnimplementedModuleFailsCleanly(t *testing. } func TestRunProcessExplicitUnimplementedModulesFailClearly(t *testing.T) { - for _, moduleKey := range []string{"glossary", "homophones", "spoken_word"} { + for _, moduleKey := range []string{"homophones", "spoken_word"} { t.Run(moduleKey, func(t *testing.T) { var stdout, stderr bytes.Buffer transcriptPath := writeFile(t, "transcript.json", `[ @@ -1196,6 +1197,281 @@ func TestRunProcessDefaultBehaviorRemainsDeterministicWithoutExplicitModules(t * } } +func TestRunProcessExplicitGlossaryAppliesCorrectionAndReportsDiagnostics(t *testing.T) { + secret := "phase13-secret" + proposalClient := &fakeStructuredLLMClient{ + proposalResponses: []proposal_generation.StructuredCorrectionSet{ + { + Corrections: []proposal_generation.StructuredCorrectionProposal{ + {TargetSegmentID: 1, OriginalText: "gestures", CorrectedText: "Jesters", Confidence: 0.95}, + }, + }, + }, + } + validationClient := &fakeStructuredLLMClient{ + validationResponses: []validators.LLMValidationResponse{ + {Validations: []validators.LLMValidationDecision{{CorrectionIndex: 0, Approved: true, Confidence: 0.99, Reason: "spoken plausible"}}}, + {Validations: []validators.LLMValidationDecision{{CorrectionIndex: 0, Approved: true, Confidence: 0.99, Reason: secret}}}, + }, + } + processProposalLLMClient = proposalClient + processValidationLLMClient = validationClient + t.Setenv("AUDITA_LLM_API_KEY", secret) + t.Setenv("AUDITA_VALIDATION_LLM_API_KEY", secret) + t.Cleanup(func() { + processProposalLLMClient = nil + processValidationLLMClient = nil + }) + + var stdout, stderr bytes.Buffer + workDir := t.TempDir() + reportPath := filepath.Join(t.TempDir(), "report.json") + outputPath := filepath.Join(t.TempDir(), "out.json") + transcriptPath := writeFile(t, "transcript.json", `[ + {"id":1,"speaker":"Alice","start":0.0,"end":1.0,"text":"There were gestures in the hall."} +]`) + + exitCode := Run([]string{ + "process", transcriptPath, + "--glossary", fixturePath("tiny_glossary.yaml"), + "--modules", "glossary", + "--output", outputPath, + "--report-json", reportPath, + "--work-dir", workDir, + "--work-dir-retention", "always", + }, &stdout, &stderr) + if exitCode != 0 { + t.Fatalf("expected success, got %d stderr=%q", exitCode, stderr.String()) + } + if stdout.Len() != 0 { + t.Fatalf("expected empty stdout with --output, got %q", stdout.String()) + } + + parsed, err := schema.ParseTranscriptJSON(readFile(t, outputPath)) + if err != nil { + t.Fatalf("parse output: %v", err) + } + if parsed.Segments[0].Text != "There were Jesters in the hall." { + t.Fatalf("expected glossary correction applied, got %q", parsed.Segments[0].Text) + } + + report := readProcessReport(t, reportPath) + if len(report.ModuleResults) != 1 || report.ModuleResults[0].ModuleKey != "glossary" { + t.Fatalf("expected one glossary module result, got %+v", report.ModuleResults) + } + if len(report.ModuleResults[0].AppliedChanges) != 1 { + t.Fatalf("expected one applied glossary change, got %+v", report.ModuleResults[0].AppliedChanges) + } + if len(report.ModuleResults[0].ValidatorDecisions) == 0 { + t.Fatalf("expected validator decisions in report") + } + + runDir := onlyRunDir(t, workDir) + runReport := readProcessReport(t, filepath.Join(runDir, "report.json")) + if len(runReport.ModuleResults) != 1 { + t.Fatalf("expected module results in run-dir report") + } + + diagFiles, globErr := filepath.Glob(filepath.Join(runDir, "glossary", "*response-payload.json")) + if globErr != nil { + t.Fatalf("glob diagnostics: %v", globErr) + } + if len(diagFiles) == 0 { + t.Fatalf("expected glossary diagnostics payload files in %s", filepath.Join(runDir, "glossary")) + } + for _, f := range diagFiles { + raw := string(readFile(t, f)) + if strings.Contains(raw, secret) { + t.Fatalf("secret leaked in diagnostics %q: %s", f, raw) + } + } +} + +func TestRunProcessExplicitGlossaryRejectedAndApplicationSkipAreDistinct(t *testing.T) { + processProposalLLMClient = &fakeStructuredLLMClient{ + proposalResponses: []proposal_generation.StructuredCorrectionSet{ + { + Corrections: []proposal_generation.StructuredCorrectionProposal{ + {TargetSegmentID: 1, OriginalText: "there were gestures", CorrectedText: "There were gestures", Confidence: 0.99}, + {TargetSegmentID: 1, OriginalText: "there were gestures", CorrectedText: "there were jesters", Confidence: 0.99}, + {TargetSegmentID: 1, OriginalText: "hall", CorrectedText: "temple", Confidence: 0.99}, + }, + }, + }, + } + processValidationLLMClient = &fakeStructuredLLMClient{ + validationResponses: []validators.LLMValidationResponse{ + { + Validations: []validators.LLMValidationDecision{ + {CorrectionIndex: 0, Approved: true, Confidence: 0.9, Reason: "ok"}, + {CorrectionIndex: 1, Approved: true, Confidence: 0.9, Reason: "ok"}, + {CorrectionIndex: 2, Approved: false, Confidence: 0.9, Reason: "reject"}, + }, + }, + { + Validations: []validators.LLMValidationDecision{ + {CorrectionIndex: 0, Approved: true, Confidence: 0.9, Reason: "ok"}, + {CorrectionIndex: 1, Approved: true, Confidence: 0.9, Reason: "ok"}, + }, + }, + }, + } + t.Cleanup(func() { + processProposalLLMClient = nil + processValidationLLMClient = nil + }) + + var stdout, stderr bytes.Buffer + reportPath := filepath.Join(t.TempDir(), "report.json") + outputPath := filepath.Join(t.TempDir(), "out.json") + transcriptPath := writeFile(t, "transcript.json", `[ + {"id":1,"speaker":"Alice","start":0.0,"end":1.0,"text":"there were gestures in the hall and there were gestures."} +]`) + exitCode := Run([]string{ + "process", transcriptPath, + "--glossary", fixturePath("tiny_glossary.yaml"), + "--modules", "glossary", + "--output", outputPath, + "--report-json", reportPath, + }, &stdout, &stderr) + if exitCode != 0 { + t.Fatalf("expected success, got %d stderr=%q", exitCode, stderr.String()) + } + + report := readProcessReport(t, reportPath) + if len(report.ModuleResults) != 1 { + t.Fatalf("expected one module result") + } + module := report.ModuleResults[0] + if len(module.ValidatorRejected) != 1 { + t.Fatalf("expected one validator rejection, got %+v", module.ValidatorRejected) + } + if len(module.SkippedChanges) != 1 { + t.Fatalf("expected one application skip, got %+v", module.SkippedChanges) + } + if module.ValidatorRejected[0].ReasonCode == string(module.SkippedChanges[0].SkipReason) { + t.Fatalf("validator rejection and application skip should remain distinct") + } +} + +func TestRunProcessExplicitGlossaryRepeatedStagesUseDeterministicInstanceNamesAndSeePriorChanges(t *testing.T) { + proposalClient := &fakeStructuredLLMClient{ + proposalResponses: []proposal_generation.StructuredCorrectionSet{ + { + Corrections: []proposal_generation.StructuredCorrectionProposal{ + {TargetSegmentID: 1, OriginalText: "gestures", CorrectedText: "Jesters", Confidence: 0.99}, + }, + }, + { + Corrections: []proposal_generation.StructuredCorrectionProposal{ + {TargetSegmentID: 1, OriginalText: "Jesters", CorrectedText: "JESTERS", Confidence: 0.99}, + }, + }, + }, + } + validationClient := &fakeStructuredLLMClient{ + validationResponses: []validators.LLMValidationResponse{ + {Validations: []validators.LLMValidationDecision{{CorrectionIndex: 0, Approved: true, Confidence: 0.99, Reason: "ok"}}}, + {Validations: []validators.LLMValidationDecision{{CorrectionIndex: 0, Approved: true, Confidence: 0.99, Reason: "ok"}}}, + {Validations: []validators.LLMValidationDecision{{CorrectionIndex: 0, Approved: true, Confidence: 0.99, Reason: "ok"}}}, + {Validations: []validators.LLMValidationDecision{{CorrectionIndex: 0, Approved: true, Confidence: 0.99, Reason: "ok"}}}, + }, + } + processProposalLLMClient = proposalClient + processValidationLLMClient = validationClient + t.Cleanup(func() { + processProposalLLMClient = nil + processValidationLLMClient = nil + }) + + var stdout, stderr bytes.Buffer + reportPath := filepath.Join(t.TempDir(), "report.json") + outputPath := filepath.Join(t.TempDir(), "out.json") + transcriptPath := writeFile(t, "transcript.json", `[ + {"id":1,"speaker":"Alice","start":0.0,"end":1.0,"text":"There were gestures in the hall."} +]`) + exitCode := Run([]string{ + "process", transcriptPath, + "--glossary", fixturePath("tiny_glossary.yaml"), + "--modules", "glossary,glossary", + "--output", outputPath, + "--report-json", reportPath, + }, &stdout, &stderr) + if exitCode != 0 { + t.Fatalf("expected success, got %d stderr=%q", exitCode, stderr.String()) + } + + parsed, err := schema.ParseTranscriptJSON(readFile(t, outputPath)) + if err != nil { + t.Fatalf("parse output: %v", err) + } + if parsed.Segments[0].Text != "There were JESTERS in the hall." { + t.Fatalf("expected second stage to see first stage changes, got %q", parsed.Segments[0].Text) + } + + report := readProcessReport(t, reportPath) + if len(report.ModuleResults) != 2 { + t.Fatalf("expected two module results, got %+v", report.ModuleResults) + } + if report.ModuleResults[0].ModuleInstance != "glossary_1" || report.ModuleResults[1].ModuleInstance != "glossary_2" { + t.Fatalf("expected deterministic glossary instance names, got %+v", report.ModuleResults) + } + if len(report.ModuleResults[0].AppliedChanges) != 1 || len(report.ModuleResults[1].AppliedChanges) != 1 { + t.Fatalf("expected one applied change per stage, got %+v", report.ModuleResults) + } + foundStage1 := false + foundStage2 := false + for _, call := range proposalClient.calls { + if strings.HasPrefix(call, "glossary_1:proposal") { + foundStage1 = true + } + if strings.HasPrefix(call, "glossary_2:proposal") { + foundStage2 = true + } + } + if !foundStage1 || !foundStage2 { + t.Fatalf("expected proposal calls for glossary_1 and glossary_2, got %v", proposalClient.calls) + } +} + +func TestRunProcessExplicitGlossaryMalformedLLMOutputFailsWithErrorLog(t *testing.T) { + processProposalLLMClient = &fakeStructuredLLMClient{err: errors.New("malformed structured output")} + t.Cleanup(func() { processProposalLLMClient = nil }) + + var stdout, stderr bytes.Buffer + workDir := t.TempDir() + reportPath := filepath.Join(t.TempDir(), "report.json") + transcriptPath := writeFile(t, "transcript.json", `[ + {"id":1,"speaker":"Alice","start":0.0,"end":1.0,"text":"hello"} +]`) + + exitCode := Run([]string{ + "process", transcriptPath, + "--glossary", fixturePath("tiny_glossary.yaml"), + "--modules", "glossary", + "--work-dir", workDir, + "--work-dir-retention", "always", + "--report-json", reportPath, + }, &stdout, &stderr) + if exitCode == 0 { + t.Fatal("expected failure") + } + if stdout.Len() != 0 { + t.Fatalf("expected empty stdout on failure, got %q", stdout.String()) + } + if !strings.Contains(stderr.String(), "runner_execution") { + t.Fatalf("expected runner_execution error, got %q", stderr.String()) + } + runDir := onlyRunDir(t, workDir) + if _, err := os.Stat(filepath.Join(runDir, "error.log")); err != nil { + t.Fatalf("expected error.log on failed glossary run: %v", err) + } + report := readProcessReport(t, reportPath) + if report.Status != "failed" || report.ErrorPhase != "runner_execution" { + t.Fatalf("expected failed runner_execution report, got %+v", report) + } +} + func TestRunProcessChunkingSummaryArtifactWritten(t *testing.T) { var stdout bytes.Buffer var stderr bytes.Buffer diff --git a/internal/core/reporting/report_test.go b/internal/core/reporting/report_test.go index e53658b..71e72fd 100644 --- a/internal/core/reporting/report_test.go +++ b/internal/core/reporting/report_test.go @@ -11,7 +11,7 @@ import ( func TestProcessReportModuleResultsJSONSuccessAndSkipped(t *testing.T) { now := time.Now().UTC() report := ProcessReport{ - Phase: "phase12-grammar-module", + Phase: "phase13-glossary-module", Status: "success", ModuleResults: []ModuleReport{ { @@ -80,7 +80,7 @@ func TestProcessReportModuleResultsJSONSuccessAndSkipped(t *testing.T) { func TestProcessReportModuleResultsJSONFailedModule(t *testing.T) { now := time.Now().UTC() report := ProcessReport{ - Phase: "phase12-grammar-module", + Phase: "phase13-glossary-module", Status: "failed", ModuleResults: []ModuleReport{ { diff --git a/internal/framework/modules/registry.go b/internal/framework/modules/registry.go index 4a702d8..28373b9 100644 --- a/internal/framework/modules/registry.go +++ b/internal/framework/modules/registry.go @@ -8,6 +8,7 @@ import ( "gitea.maximumdirect.net/eric/audita/internal/core/config" "gitea.maximumdirect.net/eric/audita/internal/core/schema" "gitea.maximumdirect.net/eric/audita/internal/framework/contracts" + glossarymodule "gitea.maximumdirect.net/eric/audita/internal/modules/glossary" grammarmodule "gitea.maximumdirect.net/eric/audita/internal/modules/grammar" ) @@ -70,6 +71,7 @@ func NewFactory(deps Dependencies) *Factory { deps: deps, constructors: make(map[string]Constructor, len(knownModuleKeys)), } + _ = factory.RegisterConstructor(ModuleKeyGlossary, constructGlossaryModule) _ = factory.RegisterConstructor(ModuleKeyGrammar, constructGrammarModule) return factory } @@ -96,6 +98,12 @@ func constructGrammarModule(ctx context.Context, req ConstructRequest) (contract return grammarmodule.New() } +func constructGlossaryModule(ctx context.Context, req ConstructRequest) (contracts.TranscriptModule, error) { + _ = ctx + _ = req + return glossarymodule.New() +} + // ModuleForSpec resolves one configured run spec into a module instance. func (f *Factory) ModuleForSpec(spec contracts.ModuleRunSpec) (contracts.TranscriptModule, error) { if f == nil { diff --git a/internal/framework/modules/registry_test.go b/internal/framework/modules/registry_test.go index 120bad0..9757ade 100644 --- a/internal/framework/modules/registry_test.go +++ b/internal/framework/modules/registry_test.go @@ -66,7 +66,7 @@ func TestUnsupportedUnknownModuleKeyFailsCleanly(t *testing.T) { func TestRecognizedButUnimplementedModuleKeyFailsCleanly(t *testing.T) { factory := NewFactory(Dependencies{}) - for _, key := range []string{ModuleKeyGlossary, ModuleKeyHomophones, ModuleKeySpokenWord} { + for _, key := range []string{ModuleKeyHomophones, ModuleKeySpokenWord} { t.Run(key, func(t *testing.T) { _, err := factory.ModuleForSpec(contracts.ModuleRunSpec{ModuleKey: key, InstanceName: key}) if err == nil { @@ -95,6 +95,17 @@ func TestGrammarIsRegisteredAndConstructibleByDefault(t *testing.T) { } } +func TestGlossaryIsRegisteredAndConstructibleByDefault(t *testing.T) { + factory := NewFactory(Dependencies{}) + module, err := factory.ModuleForSpec(contracts.ModuleRunSpec{ModuleKey: ModuleKeyGlossary, InstanceName: ModuleKeyGlossary}) + if err != nil { + t.Fatalf("ModuleForSpec error: %v", err) + } + if module.Key() != ModuleKeyGlossary { + t.Fatalf("expected glossary module key, got %q", module.Key()) + } +} + func TestRegisterConstructorAndConstruct(t *testing.T) { factory := NewFactory(Dependencies{}) if err := factory.RegisterConstructor(ModuleKeyGlossary, func(ctx context.Context, req ConstructRequest) (contracts.TranscriptModule, error) { diff --git a/internal/framework/runner/runner_test.go b/internal/framework/runner/runner_test.go index d2e96f7..d14d63b 100644 --- a/internal/framework/runner/runner_test.go +++ b/internal/framework/runner/runner_test.go @@ -686,3 +686,37 @@ func TestRunnerGrammarModuleUsesConfidenceThreshold(t *testing.T) { t.Fatalf("expected low confidence reason, got %+v", out.ModuleResults[0].ValidatorRejected[0]) } } + +func TestRunnerGlossaryModuleUsesConfidenceThreshold(t *testing.T) { + client := &fakeProposalStructuredClient{ + responses: []proposal_generation.StructuredCorrectionSet{ + { + Corrections: []proposal_generation.StructuredCorrectionProposal{ + {TargetSegmentID: 1, OriginalText: "gestures", CorrectedText: "Jesters", Confidence: 0.5}, + }, + }, + }, + } + cfg := config.Default() + cfg.Thresholds.Glossary = 0.9 + factory := modules.NewFactory(modules.Dependencies{}) + out, err := New(factory).Run(context.Background(), RunInput{ + Config: &cfg, + Transcript: &schema.Transcript{Segments: []schema.Segment{{ID: 1, Text: "There were gestures"}}}, + Glossary: &schema.Glossary{Entries: []schema.GlossaryEntry{{Name: "Jesters", Category: "faction", Summary: "Faction"}}}, + ModuleSpecs: []contracts.ModuleRunSpec{{ModuleKey: "glossary", InstanceName: "glossary"}}, + ProposalLLMClient: client, + }) + if err != nil { + t.Fatalf("Run error: %v", err) + } + if out.FinalTranscript.Segments[0].Text != "There were gestures" { + t.Fatalf("expected no changes due to glossary confidence threshold, got %q", out.FinalTranscript.Segments[0].Text) + } + if len(out.ModuleResults) != 1 || len(out.ModuleResults[0].ValidatorRejected) == 0 { + t.Fatalf("expected validator rejection, got %+v", out.ModuleResults) + } + if out.ModuleResults[0].ValidatorRejected[0].ReasonCode != validators.ReasonLowConfidence { + t.Fatalf("expected low confidence reason, got %+v", out.ModuleResults[0].ValidatorRejected[0]) + } +} diff --git a/internal/framework/validators/deterministic.go b/internal/framework/validators/deterministic.go index 2e980fc..dbc99d9 100644 --- a/internal/framework/validators/deterministic.go +++ b/internal/framework/validators/deterministic.go @@ -4,8 +4,6 @@ import ( "context" "fmt" "strings" - - "gitea.maximumdirect.net/eric/audita/internal/framework/proposals" ) type ConfidenceThresholdValidator struct{} @@ -110,11 +108,12 @@ func (v ProtectedGlossaryTermValidator) Validate(_ context.Context, req Request) return Result{ValidatorName: v.Name(), Decisions: decisions}, nil } - terms := glossaryTerms(req) + vocab := NewProtectedVocabulary(req.Glossary) decisions := make([]Decision, 0, len(req.CandidateProposal)) for _, c := range req.CandidateProposal { - if altersProtectedTerm(c.CorrectionProposal, terms) { - decisions = append(decisions, rejection(c.ProposalIndex, ReasonProtectedGlossaryTerm, "proposal may alter protected glossary terminology")) + reason := vocab.violationReason(c.OriginalText, c.CorrectedText) + if reason != "" { + decisions = append(decisions, rejection(c.ProposalIndex, ReasonProtectedGlossaryTerm, reason)) continue } decisions = append(decisions, approval(c.ProposalIndex)) @@ -125,37 +124,25 @@ func (v ProtectedGlossaryTermValidator) Validate(_ context.Context, req Request) return Result{ValidatorName: v.Name(), Decisions: decisions}, nil } -func glossaryTerms(req Request) []string { - if req.Glossary == nil { - return nil - } - out := make([]string, 0) - for _, e := range req.Glossary.Entries { - if t := strings.TrimSpace(strings.ToLower(e.Name)); t != "" { - out = append(out, t) - } - for _, a := range e.Aliases { - if t := strings.TrimSpace(strings.ToLower(a)); t != "" { - out = append(out, t) - } - } - if t := strings.TrimSpace(strings.ToLower(e.Plural)); t != "" { - out = append(out, t) - } - } - return out +type GlossaryStageProtectedGlossaryTermValidator struct{} + +func (v GlossaryStageProtectedGlossaryTermValidator) Name() string { + return "glossary_stage_protected_glossary_terms" } -func altersProtectedTerm(p proposals.CorrectionProposal, terms []string) bool { - if len(terms) == 0 { - return false - } - orig := strings.ToLower(p.OriginalText) - corr := strings.ToLower(p.CorrectedText) - for _, t := range terms { - if strings.Contains(orig, t) && !strings.Contains(corr, t) { - return true +func (v GlossaryStageProtectedGlossaryTermValidator) Validate(_ context.Context, req Request) (Result, error) { + vocab := NewProtectedVocabulary(req.Glossary) + decisions := make([]Decision, 0, len(req.CandidateProposal)) + for _, c := range req.CandidateProposal { + reason := vocab.glossaryStageViolationReason(c.OriginalText, c.CorrectedText) + if reason != "" { + decisions = append(decisions, rejection(c.ProposalIndex, ReasonProtectedGlossaryTerm, reason)) + continue } + decisions = append(decisions, approval(c.ProposalIndex)) } - return false + if err := EnforceDecisionCardinality(req.CandidateProposal, decisions); err != nil { + return Result{}, err + } + return Result{ValidatorName: v.Name(), Decisions: decisions}, nil } diff --git a/internal/framework/validators/protected_terms.go b/internal/framework/validators/protected_terms.go new file mode 100644 index 0000000..5a2fe9b --- /dev/null +++ b/internal/framework/validators/protected_terms.go @@ -0,0 +1,188 @@ +package validators + +import ( + "regexp" + "sort" + "strings" + + "gitea.maximumdirect.net/eric/audita/internal/core/schema" +) + +type protectedTermDef struct { + identity int + canonical string +} + +type protectedOccurrence struct { + text string + identity int + canonical string +} + +// ProtectedVocabulary is a deterministic glossary-derived protected term set. +type ProtectedVocabulary struct { + termsByFolded map[string]protectedTermDef + pattern *regexp.Regexp +} + +// ExtractProtectedTerms returns a stable, de-duplicated list of protected terms +// derived from glossary names, aliases, synthetic plural forms, and explicit +// plural fields. +func ExtractProtectedTerms(glossary *schema.Glossary) []string { + vocab := NewProtectedVocabulary(glossary) + out := make([]string, 0, len(vocab.termsByFolded)) + for _, def := range vocab.termsByFolded { + out = append(out, def.canonical) + } + sort.SliceStable(out, func(i, j int) bool { + li := strings.ToLower(out[i]) + lj := strings.ToLower(out[j]) + if li == lj { + return out[i] < out[j] + } + return li < lj + }) + return out +} + +// NewProtectedVocabulary builds a deterministic protected vocabulary from a glossary. +func NewProtectedVocabulary(glossary *schema.Glossary) ProtectedVocabulary { + termsByFolded := make(map[string]protectedTermDef) + if glossary != nil { + for identity, entry := range glossary.Entries { + entryTerms := append([]string{entry.Name}, entry.Aliases...) + for _, term := range entryTerms { + trimmed := strings.TrimSpace(term) + if trimmed == "" { + continue + } + addProtectedTerm(termsByFolded, trimmed, identity) + addProtectedTerm(termsByFolded, trimmed+"s", identity) + } + addProtectedTerm(termsByFolded, entry.Plural, identity) + } + } + + alternatives := make([]string, 0, len(termsByFolded)) + for _, def := range termsByFolded { + alternatives = append(alternatives, regexp.QuoteMeta(def.canonical)) + } + sort.SliceStable(alternatives, func(i, j int) bool { return len(alternatives[i]) > len(alternatives[j]) }) + if len(alternatives) == 0 { + return ProtectedVocabulary{termsByFolded: termsByFolded} + } + pattern := regexp.MustCompile(`(?i)\b(?:` + strings.Join(alternatives, "|") + `)\b`) + return ProtectedVocabulary{termsByFolded: termsByFolded, pattern: pattern} +} + +func addProtectedTerm(terms map[string]protectedTermDef, term string, identity int) { + trimmed := strings.TrimSpace(term) + if trimmed == "" { + return + } + folded := strings.ToLower(trimmed) + if _, ok := terms[folded]; ok { + return + } + terms[folded] = protectedTermDef{identity: identity, canonical: trimmed} +} + +func (v ProtectedVocabulary) violationReason(before, after string) string { + beforeByID := v.occurrencesByIdentity(before) + afterByID := v.occurrencesByIdentity(after) + if reason := validateIdentityPreservation(beforeByID, afterByID); reason != "" { + return reason + } + if reason := validateCapitalizationTransitions(beforeByID, afterByID); reason != "" { + return reason + } + return "" +} + +func (v ProtectedVocabulary) glossaryStageViolationReason(before, after string) string { + beforeByID := v.occurrencesByIdentity(before) + afterByID := v.occurrencesByIdentity(after) + if reason := validateGlossaryStageIdentityPreservation(beforeByID, afterByID); reason != "" { + return reason + } + if reason := validateCapitalizationTransitions(beforeByID, afterByID); reason != "" { + return reason + } + return "" +} + +func (v ProtectedVocabulary) occurrencesByIdentity(text string) map[int][]protectedOccurrence { + byID := make(map[int][]protectedOccurrence) + for _, o := range v.occurrences(text) { + byID[o.identity] = append(byID[o.identity], o) + } + return byID +} + +func (v ProtectedVocabulary) occurrences(text string) []protectedOccurrence { + if v.pattern == nil { + return nil + } + matches := v.pattern.FindAllStringIndex(text, -1) + if len(matches) == 0 { + return nil + } + out := make([]protectedOccurrence, 0, len(matches)) + for _, idx := range matches { + matched := text[idx[0]:idx[1]] + def, ok := v.termsByFolded[strings.ToLower(matched)] + if !ok { + continue + } + out = append(out, protectedOccurrence{ + text: matched, + identity: def.identity, + canonical: def.canonical, + }) + } + return out +} + +func validateIdentityPreservation(beforeByID, afterByID map[int][]protectedOccurrence) string { + for identity, beforeItems := range beforeByID { + if len(afterByID[identity]) < len(beforeItems) { + return "proposal may alter protected glossary terminology" + } + } + return "" +} + +func validateGlossaryStageIdentityPreservation(beforeByID, afterByID map[int][]protectedOccurrence) string { + beforeTotal := 0 + for _, items := range beforeByID { + beforeTotal += len(items) + } + afterTotal := 0 + for _, items := range afterByID { + afterTotal += len(items) + } + if afterTotal < beforeTotal { + return "proposal may alter protected glossary terminology" + } + return "" +} + +func validateCapitalizationTransitions(beforeByID, afterByID map[int][]protectedOccurrence) string { + for identity, afterItems := range afterByID { + beforeItems := beforeByID[identity] + for i, afterItem := range afterItems { + if i >= len(beforeItems) { + continue + } + beforeItem := beforeItems[i] + if afterItem.text == beforeItem.text { + continue + } + if afterItem.text == afterItem.canonical { + continue + } + return "proposal may alter protected glossary terminology" + } + } + return "" +} diff --git a/internal/framework/validators/protected_terms_test.go b/internal/framework/validators/protected_terms_test.go new file mode 100644 index 0000000..e5e3e12 --- /dev/null +++ b/internal/framework/validators/protected_terms_test.go @@ -0,0 +1,86 @@ +package validators + +import ( + "context" + "reflect" + "testing" + + "gitea.maximumdirect.net/eric/audita/internal/core/schema" + "gitea.maximumdirect.net/eric/audita/internal/framework/proposals" +) + +func TestExtractProtectedTermsStableDeduplicatedWithPlurals(t *testing.T) { + glossary := &schema.Glossary{ + Entries: []schema.GlossaryEntry{ + {Name: "Jesters", Aliases: []string{"Jester", " "}, Plural: "Jesters", Category: "faction", Summary: "Faction"}, + {Name: "Hrank", Aliases: []string{"hrank", "Hrank"}, Plural: "Hranks", Category: "pc", Summary: "Character"}, + {Name: "Lyra", Aliases: []string{}, Plural: "", Category: "npc", Summary: "NPC"}, + }, + } + + got := ExtractProtectedTerms(glossary) + want := []string{"Hrank", "Hranks", "Jester", "Jesters", "Jesterss", "Lyra", "Lyras"} + if !reflect.DeepEqual(got, want) { + t.Fatalf("unexpected protected terms\n got: %v\nwant: %v", got, want) + } +} + +func TestExtractProtectedTermsEmptyGlossary(t *testing.T) { + if terms := ExtractProtectedTerms(&schema.Glossary{}); len(terms) != 0 { + t.Fatalf("expected no terms, got %v", terms) + } +} + +func TestProtectedGlossaryTermValidatorAppliesToNonGlossaryModule(t *testing.T) { + req := Request{ + ModuleKey: "grammar", + Glossary: &schema.Glossary{ + Entries: []schema.GlossaryEntry{{Name: "Jesters", Category: "faction", Summary: "Faction"}}, + }, + CandidateProposal: []proposals.EnrichedCorrectionProposal{ + { + CorrectionProposal: proposals.CorrectionProposal{ + TargetSegmentID: 1, + OriginalText: "Jesters", + CorrectedText: "Gestures", + Confidence: 0.9, + }, + ProposalMetadata: proposals.ProposalMetadata{ProposalIndex: 0}, + }, + }, + } + res, err := (ProtectedGlossaryTermValidator{}).Validate(context.Background(), req) + if err != nil { + t.Fatalf("validator error: %v", err) + } + if len(res.Decisions) != 1 || res.Decisions[0].Approved { + t.Fatalf("expected protected-term rejection, got %+v", res.Decisions) + } +} + +func TestGlossaryStageProtectedGlossaryTermValidatorAllowsProtectedTermSwap(t *testing.T) { + req := Request{ + ModuleKey: "glossary", + Glossary: &schema.Glossary{ + Entries: []schema.GlossaryEntry{{Name: "Jesters", Category: "faction", Summary: "Faction"}}, + }, + CandidateProposal: []proposals.EnrichedCorrectionProposal{ + { + CorrectionProposal: proposals.CorrectionProposal{ + TargetSegmentID: 1, + OriginalText: "gestures", + CorrectedText: "Jesters", + Confidence: 0.9, + }, + ProposalMetadata: proposals.ProposalMetadata{ProposalIndex: 0}, + }, + }, + } + res, err := (GlossaryStageProtectedGlossaryTermValidator{}).Validate(context.Background(), req) + if err != nil { + t.Fatalf("validator error: %v", err) + } + if len(res.Decisions) != 1 || !res.Decisions[0].Approved { + t.Fatalf("expected approval for glossary-stage protected term correction, got %+v", res.Decisions) + } +} diff --git a/internal/modules/glossary/module.go b/internal/modules/glossary/module.go new file mode 100644 index 0000000..5e77ea0 --- /dev/null +++ b/internal/modules/glossary/module.go @@ -0,0 +1,76 @@ +package glossary + +import ( + "context" + "fmt" + + "gitea.maximumdirect.net/eric/audita/internal/framework/contracts" + "gitea.maximumdirect.net/eric/audita/internal/framework/proposal_generation" + "gitea.maximumdirect.net/eric/audita/internal/framework/proposals" + "gitea.maximumdirect.net/eric/audita/internal/framework/validators" +) + +type Module struct { + validators []contracts.Validator +} + +func New() (*Module, error) { + spokenForm, err := validators.NewLLMBackedValidator("spoken_form_plausibility_review", validators.LLMValidatorTypeSpokenFormPlausibility, "") + if err != nil { + return nil, err + } + meaningReversal, err := validators.NewLLMBackedValidator("meaning_reversal_review", validators.LLMValidatorTypeMeaningReversal, "") + if err != nil { + return nil, err + } + + return &Module{ + validators: []contracts.Validator{ + validators.NoEffectValidator{}, + validators.OriginalTextPresenceValidator{}, + validators.ConfidenceThresholdValidator{}, + validators.GlossaryStageProtectedGlossaryTermValidator{}, + validators.NonEmptyCorrectionValidator{}, + spokenForm, + meaningReversal, + }, + }, nil +} + +func (m *Module) Key() string { return "glossary" } + +func (m *Module) ReplacementPolicy() proposals.ReplacementPolicy { + // Python glossary module uses replace_all to update repeated term occurrences. + return proposals.ReplacementPolicyReplaceAll +} + +func (m *Module) Validators() []contracts.Validator { + return append([]contracts.Validator(nil), m.validators...) +} + +func (m *Module) Propose(ctx context.Context, req contracts.ProposalRequest) ([]proposals.CorrectionProposal, error) { + messages, err := BuildProposalMessages(req.WorkingTranscript, req.Glossary) + if err != nil { + return nil, err + } + + generated, err := proposal_generation.GenerateCandidates(ctx, proposal_generation.Request{ + ModuleKey: req.RunSpec.ModuleKey, + ModuleInstance: req.RunSpec.InstanceName, + ReplacementPolicy: req.RunSpec.ReplacementPolicy, + WorkingTranscript: req.WorkingTranscript, + Section: req.Section, + Glossary: req.Glossary, + Config: req.Config, + Messages: messages, + StageName: fmt.Sprintf("%s:proposal", req.RunSpec.InstanceName), + StartIndex: 0, + LLMClient: req.LLMClient, + Scheduler: req.LLMScheduler, + DiagnosticsDir: req.DiagnosticsDir, + }) + if err != nil { + return nil, err + } + return generated.Corrections, nil +} diff --git a/internal/modules/glossary/module_test.go b/internal/modules/glossary/module_test.go new file mode 100644 index 0000000..fe9bc64 --- /dev/null +++ b/internal/modules/glossary/module_test.go @@ -0,0 +1,188 @@ +package glossary + +import ( + "context" + "errors" + "os" + "path/filepath" + "strings" + "testing" + + "gitea.maximumdirect.net/eric/audita/internal/core/config" + "gitea.maximumdirect.net/eric/audita/internal/core/schema" + "gitea.maximumdirect.net/eric/audita/internal/framework/contracts" + "gitea.maximumdirect.net/eric/audita/internal/framework/proposal_generation" + "gitea.maximumdirect.net/eric/audita/internal/framework/proposals" +) + +type fakeLLMClient struct { + responses []proposal_generation.StructuredCorrectionSet + err error + calls []contracts.StructuredCompletionRequest +} + +func (f *fakeLLMClient) CompleteStructured(ctx context.Context, req contracts.StructuredCompletionRequest, out any) (contracts.StructuredCompletionResponse, error) { + _ = ctx + f.calls = append(f.calls, req) + if f.err != nil { + return contracts.StructuredCompletionResponse{}, f.err + } + target, ok := out.(*proposal_generation.StructuredCorrectionSet) + if !ok { + return contracts.StructuredCompletionResponse{}, errors.New("unexpected output type") + } + if len(f.responses) == 0 { + return contracts.StructuredCompletionResponse{}, errors.New("unexpected call") + } + *target = f.responses[0] + f.responses = f.responses[1:] + return contracts.StructuredCompletionResponse{}, nil +} + +type countingScheduler struct { + runs int +} + +func (s *countingScheduler) Run(ctx context.Context, fn func(context.Context) error) error { + s.runs++ + return fn(ctx) +} + +func tinyTranscript() *schema.Transcript { + return &schema.Transcript{Segments: []schema.Segment{ + {ID: 1, Speaker: "A", Start: 0, End: 1, Text: "There were gestures in the hall.", Categories: []string{"session"}}, + }} +} + +func tinyGlossary() *schema.Glossary { + return &schema.Glossary{Entries: []schema.GlossaryEntry{ + {Name: "Jesters", Aliases: []string{"Jester"}, Category: "faction", Summary: "Guild members", Plural: "Jesters"}, + }} +} + +func TestBuildProposalMessagesContainsGlossaryContextAndConstraints(t *testing.T) { + msgs, err := BuildProposalMessages(tinyTranscript(), tinyGlossary()) + if err != nil { + t.Fatalf("BuildProposalMessages error: %v", err) + } + if len(msgs) != 2 { + t.Fatalf("expected 2 messages, got %d", len(msgs)) + } + combined := msgs[0].Content + "\n" + msgs[1].Content + for _, want := range []string{ + "Glossary:", + "Transcript section:", + `"Aliases":`, + `"Jester"`, + `"Category": "faction"`, + `"Summary": "Guild members"`, + `"Plural": "Jesters"`, + "glossary-supported corrections", + "acoustically similar", + "Do not make generic grammar, spelling, capitalization, style, or filler-word edits", + } { + if !strings.Contains(combined, want) { + t.Fatalf("expected prompt to contain %q", want) + } + } + if strings.Contains(strings.ToLower(combined), "summarize the transcript") { + t.Fatalf("prompt should not invite summarization") + } +} + +func TestGlossaryModuleReplacementPolicy(t *testing.T) { + m, err := New() + if err != nil { + t.Fatalf("New error: %v", err) + } + if m.ReplacementPolicy() != proposals.ReplacementPolicyReplaceAll { + t.Fatalf("unexpected replacement policy: %q", m.ReplacementPolicy()) + } +} + +func TestGlossaryModuleValidatorChain(t *testing.T) { + m, err := New() + if err != nil { + t.Fatalf("New error: %v", err) + } + got := make([]string, 0, len(m.Validators())) + for _, v := range m.Validators() { + got = append(got, v.Name()) + } + want := []string{ + "no_effect", + "original_text_presence", + "confidence_threshold", + "glossary_stage_protected_glossary_terms", + "non_empty_correction", + "spoken_form_plausibility_review", + "meaning_reversal_review", + } + if strings.Join(got, ",") != strings.Join(want, ",") { + t.Fatalf("unexpected validator chain\n got: %v\nwant: %v", got, want) + } +} + +func TestGlossaryModuleProposeMapsCorrectionsAndWritesDiagnostics(t *testing.T) { + secret := "glossary-secret" + client := &fakeLLMClient{ + responses: []proposal_generation.StructuredCorrectionSet{ + { + Corrections: []proposal_generation.StructuredCorrectionProposal{ + {TargetSegmentID: 1, OriginalText: "gestures", CorrectedText: "Jesters", Confidence: 0.94}, + }, + }, + }, + } + scheduler := &countingScheduler{} + cfg := config.Default() + cfg.PrimaryLLM.APIKey = secret + m, err := New() + if err != nil { + t.Fatalf("New error: %v", err) + } + diagDir := t.TempDir() + out, err := m.Propose(context.Background(), contracts.ProposalRequest{ + ExecutionContext: contracts.ExecutionContext{ + Config: &cfg, + WorkingTranscript: tinyTranscript(), + Glossary: tinyGlossary(), + DiagnosticsDir: diagDir, + }, + RunSpec: contracts.ModuleRunSpec{ + ModuleKey: "glossary", + InstanceName: "glossary", + ReplacementPolicy: proposals.ReplacementPolicyReplaceAll, + }, + LLMClient: client, + LLMScheduler: scheduler, + }) + if err != nil { + t.Fatalf("Propose error: %v", err) + } + if scheduler.runs != 1 { + t.Fatalf("expected scheduler run count 1, got %d", scheduler.runs) + } + if len(client.calls) != 1 || client.calls[0].StageName != "glossary:proposal" { + t.Fatalf("expected one glossary:proposal call, got %+v", client.calls) + } + if len(out) != 1 || out[0].CorrectedText != "Jesters" { + t.Fatalf("unexpected proposals: %+v", out) + } + diagFiles, globErr := filepath.Glob(filepath.Join(diagDir, "glossary", "*proposal*response-payload.json")) + if globErr != nil { + t.Fatalf("glob error: %v", globErr) + } + if len(diagFiles) == 0 { + t.Fatalf("expected diagnostics response payload under %s", filepath.Join(diagDir, "glossary")) + } + for _, f := range diagFiles { + raw, readErr := os.ReadFile(f) + if readErr != nil { + t.Fatalf("read diag %q: %v", f, readErr) + } + if strings.Contains(string(raw), secret) { + t.Fatalf("secret leaked in diagnostics: %s", string(raw)) + } + } +} diff --git a/internal/modules/glossary/prompt.go b/internal/modules/glossary/prompt.go new file mode 100644 index 0000000..b8dab53 --- /dev/null +++ b/internal/modules/glossary/prompt.go @@ -0,0 +1,80 @@ +package glossary + +import ( + "encoding/json" + "fmt" + + "gitea.maximumdirect.net/eric/audita/internal/core/schema" + "gitea.maximumdirect.net/eric/audita/internal/framework/contracts" +) + +type promptSegment struct { + ID int `json:"id"` + Speaker string `json:"speaker"` + Start float64 `json:"start"` + End float64 `json:"end"` + Text string `json:"text"` + Categories []string `json:"categories,omitempty"` +} + +type promptTranscriptSection struct { + SectionIndex int `json:"section_index"` + Segments []promptSegment `json:"segments"` +} + +func BuildProposalMessages(transcript *schema.Transcript, glossary *schema.Glossary) ([]contracts.LLMMessage, error) { + glossaryJSON, err := json.MarshalIndent(glossary, "", " ") + if err != nil { + return nil, fmt.Errorf("marshal glossary prompt context: %w", err) + } + + sectionPayload := promptTranscriptSection{ + SectionIndex: 0, + Segments: make([]promptSegment, 0), + } + if transcript != nil { + for _, s := range transcript.Segments { + sectionPayload.Segments = append(sectionPayload.Segments, promptSegment{ + ID: s.ID, + Speaker: s.Speaker, + Start: s.Start, + End: s.End, + Text: s.Text, + Categories: append([]string(nil), s.Categories...), + }) + } + } + sectionJSON, err := json.MarshalIndent(sectionPayload, "", " ") + if err != nil { + return nil, fmt.Errorf("marshal transcript prompt context: %w", err) + } + + system := "You are Audita, a careful transcript correction assistant. Identify only transcription errors that are strongly supported by the glossary. A valid correction must be acoustically plausible: the original transcript text should sound similar to the proposed correction when spoken aloud. Do not make generic grammar, spelling, capitalization, style, or filler-word edits. Do not substitute an unrelated glossary term just because it could fit the topic. Preserve speaker names, timestamps, and meaning." + user := "Review this transcript section and return only glossary-supported corrections that should be applied.\n\n" + + "Rules:\n" + + "- Correct domain-specific names, aliases, jargon, deities, locations, NPCs, players, factions, and similar terms only when both the glossary and surrounding transcript context support the correction.\n" + + "- The correction must plausibly fix a transcription error: the original words should be phonetically or acoustically similar to the corrected words in spoken English.\n" + + "- Appropriate example: correcting \"gestures\" to \"Jesters\" can be valid if \"Jesters\" appears in the glossary and nearby context supports that inference.\n" + + "- Inappropriate example: correcting \"Lyra\" to \"Jesters\" should be omitted because those words are not similar in spoken English, even if \"Jesters\" appears in the glossary.\n" + + "- Do not replace one clear glossary term, character name, location, or ordinary word with a different glossary term unless it is a plausible mishearing.\n" + + "- Treat glossary names and aliases already present in the transcript as protected spellings.\n" + + "- Do not replace, Anglicize, normalize, lowercase, or otherwise alter protected glossary names or aliases away from their glossary spelling.\n" + + "- Preserve canonical glossary capitalization for protected names and aliases, even if they look unusual.\n" + + "- If a segment includes categories, treat them as additional transcript context.\n" + + "- Plural forms of glossary names and aliases are allowed targets when spoken similarity and context support them, even if the plural is not explicitly listed in the glossary.\n" + + "- Use the exact id from the input segment.\n" + + "- For returned corrections, original_text must be only the exact text span that needs replacement, not the full segment text unless the whole segment is the replacement span.\n" + + "- corrected_text must be only the replacement text for that span, not the full corrected segment text unless the whole segment is the replacement span.\n" + + "- Each returned correction must contain only id, original_text, corrected_text, and confidence.\n" + + "- Do not return corrections where original_text and corrected_text are identical.\n" + + "- Do not return speaker, start, or end fields.\n" + + "- Return only changed segments; do not return entries for unchanged segments.\n" + + "- confidence must be between 0.0 and 1.0.\n" + + "- If no corrections are needed, return an empty corrections list.\n\n" + + fmt.Sprintf("Glossary:\n%s\n\nTranscript section:\n%s", string(glossaryJSON), string(sectionJSON)) + + return []contracts.LLMMessage{ + {Role: "system", Content: system}, + {Role: "user", Content: user}, + }, nil +}