eric/audita
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 2 Wiki Activity

Complete Phase 13 glossary module

Browse Source
This commit is contained in:
Eric Rakestraw
2026-05-12 11:20:02 +00:00
parent fc3a7b7a67
commit 543a7ff8ef
14 changed files with 1049 additions and 97 deletions
Download Patch File Download Diff File Expand all files Collapse all files

60
docs/architecture.md
View File

@@ -34,19 +34,22 @@ Implemented today:
- Proposal-generation diagnostics artifacts with secret redaction.
- Production module registry scaffolding with known-key recognition and explicit unsupported/unimplemented errors.
- Production `grammar` module implementation in `internal/modules/grammar`.
- Production `glossary` module implementation in `internal/modules/glossary`.
- Explicit runtime support for `--modules grammar` through the production runner path.
- Explicit runtime support for `--modules glossary`, including repeated stages such as `--modules glossary,glossary`.
Not implemented in CLI runtime path today:
- Real module execution pipeline for `glossary`, `homophones`, and `spoken_word`.
- Real domain proposal prompts for production modules.
- End-to-end transcript polishing with real module behavior.
- Real module execution pipeline for `homophones` and `spoken_word`.
- Real domain proposal prompts for remaining production modules.
- End-to-end transcript polishing with the full default module sequence.
Phase sequencing note:
- Phase 9 LLM infrastructure is complete (structured client, scheduler, effective config resolution, diagnostics primitives);
- Phase 10 LLM-backed validator runtime integration is complete;
- Phase 11 shared proposal-generation framework and module-registry scaffolding are complete;
- Phase 12 grammar module implementation and explicit runtime wiring are complete;
- next recommended phase is Phase 13 (glossary module and protected-term behavior).
- Phase 13 glossary module and protected-term behavior are complete;
- next recommended phase is Phase 14 (homophones module).
## Actual Go package layout
@@ -109,6 +112,10 @@ internal/modules/grammar/
module.go
prompt.go
internal/modules/glossary/
module.go
prompt.go
internal/framework/validators/
models.go
deterministic.go
@@ -144,17 +151,17 @@ Current runtime flow (`internal/cli/run.go`):
10. Chunk normalized transcript and compute chunk summaries.
11. Write chunking summary artifact.
12. Execute runner modules sequentially when:
- `--modules` is explicitly provided (production grammar path); or
- `--modules` is explicitly provided (production grammar/glossary paths); or
- a test/injected module factory is provided.
13. Output working transcript to `--output` file or stdout.
14. Build process report (`phase` currently set to `phase12-grammar-module`).
14. Build process report (`phase` currently set to `phase13-glossary-module`).
15. Optionally write `--report-json`; always write run-dir `report.json`.
16. Apply work-dir retention.
Important behavior details:
- Glossary is validated but not yet used for real correction module logic.
- Glossary is validated and is used for explicit glossary/grammar module correction paths.
- Default production CLI behavior remains deterministic normalization/chunking/reporting unless modules are explicitly selected with `--modules`.
- Explicit `--modules grammar` runs the production grammar module path with LLM-backed proposal generation and validator-chain execution.
- Explicit `--modules grammar` and `--modules glossary` run production module paths with LLM-backed proposal generation and validator-chain execution.
- Default runs (without explicit module selection) do not perform LLM calls.
- Success path is generally quiet on stderr.
- Source IDs are preserved into a canonical transcript before normalization; normalization then reassigns output IDs sequentially from `1`.
@@ -203,7 +210,7 @@ Implemented config surfaces include:
- work-dir and retention mode
Current caveat:
- LLM/module-related settings are active for explicit grammar runs; the default non-explicit path remains deterministic.
- LLM/module-related settings are active for explicit grammar/glossary runs; the default non-explicit path remains deterministic.
## Implemented structured LLM infrastructure
`internal/framework/contracts` now defines a typed structured-completion contract:
@@ -220,7 +227,7 @@ Current caveat:
Current runtime boundary:
- the default CLI runtime path (without explicit module selection) still does not instantiate the full production module sequence.
- LLM calls are exercised in production when `--modules grammar` is explicitly requested and in tests when fake/injected clients are used.
- LLM calls are exercised in production when `--modules grammar` or `--modules glossary` is explicitly requested and in tests when fake/injected clients are used.
`internal/framework/llm` also provides:
- a bounded `Scheduler` for controlled concurrent LLM calls with reliable permit release;
@@ -262,7 +269,7 @@ Current behavior details:
`internal/framework/contracts` provides interfaces and run-spec metadata scaffolding, including deterministic repeated module instance naming (`ResolveModuleRunSpecs`).
These primitives are wired into the production runner and report model. The grammar module is implemented; other production modules remain pending.
These primitives are wired into the production runner and report model. The grammar and glossary modules are implemented; other production modules remain pending.
## Implemented validator runtime infrastructure
`internal/framework/validators` provides deterministic validator infrastructure:
@@ -327,7 +334,7 @@ This helper only produces candidate proposals; validator-chain execution and pro
- diagnostics directory context
- returns explicit errors for unknown keys (`unsupported_module`) and recognized-but-unimplemented keys (`unimplemented_module`).
The `grammar` module key is now registered and constructible. `glossary`, `homophones`, and `spoken_word` remain recognized-but-unimplemented.
The `grammar` and `glossary` module keys are now registered and constructible. `homophones` and `spoken_word` remain recognized-but-unimplemented.
## Implemented grammar production module
`internal/modules/grammar` now provides the first production module:
@@ -340,6 +347,27 @@ The `grammar` module key is now registered and constructible. `glossary`, `homop
- grammar confidence threshold enforcement through existing validator/config infrastructure;
- module-level reporting and diagnostics capture through existing runner/reporting paths.
## Implemented glossary production module
`internal/modules/glossary` now provides the second production module:
- prompt builder aligned to Python glossary-module intent, constrained to glossary-backed domain/acoustic corrections;
- prompt context includes glossary names, aliases, categories, summaries, and plural forms where available;
- guardrails against broad style rewriting and against replacing unrelated terms simply because they appear in glossary entries;
- proposal generation through `internal/framework/proposal_generation` and `contracts.StructuredLLMClient`;
- scheduler-aware proposal calls through existing `contracts.LLMScheduler` hooks;
- replacement policy `replace_all` (matching Python glossary behavior);
- validator chain integration using existing deterministic + LLM-backed validators;
- glossary confidence threshold enforcement through existing validator/config infrastructure;
- module-level reporting and diagnostics capture through existing runner/reporting paths;
- explicit support for repeated glossary stages with deterministic instance names (`glossary_1`, `glossary_2`, ...), where later stages see prior-stage working transcript changes.
## Implemented protected-term behavior
`internal/framework/validators/protected_terms.go` provides deterministic glossary-derived protected vocabulary:
- extracts protected terms from glossary names and aliases;
- includes explicit plural fields and synthetic plural forms where safe;
- deduplicates and returns stable ordering for repeatable behavior/tests.
This vocabulary is used by deterministic validators for both glossary-stage and non-glossary-stage protection checks, keeping protected-term guardrails active across modules.
## Reports and diagnostics (implemented)
Current per-run artifacts include:
- `source-transcript.json`
@@ -379,7 +407,7 @@ Retention modes implemented in `ApplyRetention`:
Current runtime note:
- default non-explicit runs usually have no module-level skipped corrections, so `auto` commonly removes clean successful run directories.
- explicit grammar runs can produce validator rejections and application skips, which are reflected in reports and retention input.
- explicit grammar/glossary runs can produce validator rejections and application skips, which are reflected in reports and retention input.
Intentionally deferred to module/LLM phases:
- real domain proposal prompts and production module implementations remain tied to later module phases.
@@ -401,13 +429,15 @@ Implemented tests currently cover:
- shared proposal-generation request/response parsing, deterministic indexing, scheduler hooks, and diagnostics redaction (`internal/framework/proposal_generation/*_test.go`, `internal/framework/runner/*_test.go`)
- production module-registry known-key recognition and unsupported/unimplemented error behavior (`internal/framework/modules/*_test.go`, `internal/cli/run_test.go`)
- production grammar module prompt constraints, proposal mapping, validator-chain behavior, confidence-threshold enforcement, diagnostics redaction, and explicit CLI/runtime integration (`internal/modules/grammar/*_test.go`, `internal/cli/run_test.go`, `internal/framework/runner/*_test.go`)
- production glossary module prompt constraints, proposal mapping, validator-chain behavior, confidence-threshold enforcement, diagnostics redaction, repeated-stage behavior, and explicit CLI/runtime integration (`internal/modules/glossary/*_test.go`, `internal/cli/run_test.go`, `internal/framework/runner/*_test.go`)
- glossary-derived protected-term extraction and stable behavior (`internal/framework/validators/protected_terms_test.go`)
Not covered yet (because not implemented): production `glossary`, `homophones`, and `spoken_word` modules plus full default-sequence transcript-polishing runtime behavior.
Not covered yet (because not implemented): production `homophones` and `spoken_word` modules plus full default-sequence transcript-polishing runtime behavior.
## Intended final architecture (not yet implemented)
The intended end-state still matches the rewrite plan:
- sequential module pipeline over a mutable working transcript
- real module implementations (`glossary`, `homophones`, `spoken_word`, `grammar`)
- real module implementations (`homophones`, `spoken_word`)
- structured LLM proposal generation
- deterministic and LLM validators
- validator cardinality enforcement in pipeline execution

66
docs/rewrite-notes.md
View File

@@ -61,6 +61,10 @@ Implemented:
- Production module-registry scaffolding with known key recognition and explicit unsupported/unimplemented errors.
- Production grammar module package with Python-aligned prompt intent and guardrails.
- Explicit `--modules grammar` runtime path through runner, shared proposal generation, validators, application, reporting, and diagnostics.
- Production glossary module package with Python-aligned prompt intent and guardrails.
- Glossary-derived deterministic protected-term extraction and validator integration.
- Explicit `--modules glossary` runtime path through runner, shared proposal generation, validators, application, reporting, and diagnostics.
- Repeated glossary stage support with deterministic instance names (`glossary_1`, `glossary_2`), including mutable working-transcript handoff.
- Broad deterministic and CLI/subprocess test coverage for implemented phases through `go test ./...`.
- Internal typed structured LLM contract (`StructuredLLMClient.CompleteStructured(ctx, req, out)`).
- `internal/framework/llm` instructor-go-backed adapter with:
@@ -76,9 +80,9 @@ Implemented:
- Generic JSON diagnostics primitives for LLM interactions (request metadata, request payload, response payload, optional error payload) with secret redaction.
Not yet implemented in runtime pipeline:
- Real correction modules for `glossary`, `homophones`, and `spoken_word`.
- Real correction modules for `homophones` and `spoken_word`.
- Domain proposal prompts for remaining real modules.
- End-to-end transcript polishing behavior.
- End-to-end transcript polishing behavior with the full default module sequence.
## Completed phases
@@ -216,7 +220,7 @@ Not implemented in Phase 8 (by design):
## Remaining work plan
Next recommended phase: **Phase 13 (glossary module and protected-term behavior)**.
Next recommended phase: **Phase 14 (homophones module)**.
## Phase 9: Structured LLM client and scheduler infrastructure
@@ -362,44 +366,28 @@ Not implemented in Phase 12 (by design):
## Phase 13: Glossary module and protected-term behavior
### Purpose
Completed.
Implement the glossary correction module and the glossary-derived protection behavior needed by downstream modules.
Implemented:
- Production glossary module package in `internal/modules/glossary`.
- Glossary prompt builder aligned to Python intent and constrained to glossary-supported domain/acoustic corrections.
- Prompt context using glossary names, aliases, categories, summaries, and plural forms where available.
- Glossary proposal generation through shared `internal/framework/proposal_generation` using `contracts.StructuredLLMClient`.
- Scheduler-aware glossary proposal generation through existing scheduler hooks.
- Glossary replacement policy `replace_all` (matching Python behavior).
- Glossary validator chain using existing deterministic and LLM-backed validators.
- Glossary confidence threshold enforcement through existing config + confidence-threshold validator behavior.
- Deterministic glossary-derived protected-term extraction (`internal/framework/validators/protected_terms.go`) from names, aliases, and plural forms, with stable deduplicated ordering.
- Protected-term validator behavior remaining available to non-glossary modules via existing deterministic validators.
- Explicit runtime support for `--modules glossary` through normalization, chunking, runner, proposal generation, validation, application, and reporting.
- Repeated glossary-stage support (`--modules glossary,glossary`) with deterministic instance naming and mutable working-transcript handoff across stages.
- Prompt/response diagnostics artifacts for glossary proposal + validator interactions with secret redaction.
- Module-level reports for glossary including generated proposals, validator decisions/rejections, applied changes, and application skips.
- CLI/runtime fake-client tests for approved proposals, validator rejection, application skips, repeated stages, diagnostics, failure/error.log behavior, and report outputs (`--report-json` and run-dir `report.json`).
### Scope
Implement:
- `glossary` module package.
- Glossary prompt builder ported from Python.
- Glossary structured response model.
- Glossary replacement policy.
- Glossary confidence threshold handling.
- Glossary validator chain.
- Protected-term extraction from parsed glossary.
- Protected-term validator behavior used by other modules where applicable.
- Prompt/response diagnostics.
- CLI support for `--modules glossary`.
- Fake LLM tests.
- Tests for repeated glossary stages using `glossary,glossary`.
Do not implement:
- Homophones module.
- Spoken-word module.
- Default full pipeline parity claim.
### Expected behavior at end of phase
Running `audita process ... --modules glossary` should perform real glossary-supported corrections. Repeated glossary stages should work and be reported as separate module instances.
### Definition of done
- Glossary module runs in the production runner.
- Glossary terms and aliases are used in prompts and validators.
- Protected-term behavior is implemented and tested.
- Repeated glossary module instances are reported correctly.
- Applied/skipped glossary changes appear in reports.
- Prompt/response diagnostics are written.
- `go test ./...` passes without requiring external LLM credentials.
Not implemented in Phase 13 (by design):
- Production `homophones` and `spoken_word` modules.
- Full default module sequence execution as a feature-complete claim.
## Phase 14: Homophones module

2
internal/cli/run.go
View File

@@ -460,7 +460,7 @@ func extractErrorPhase(err error) (phase string, message string) {
func buildProcessReport(status string, inv processInvocation, runDir *diagnostics.RunDirectory, startedAt, completedAt time.Time, errorMessage string, errorPhase string, normalizationSummary *normalization.NormalizationSummary, chunkingSummary *chunking.Summary, runOutput *runner.RunOutput) reporting.ProcessReport {
report := reporting.ProcessReport{
Phase: "phase12-grammar-module",
Phase: "phase13-glossary-module",
Status: status,
Operation: "process",
TranscriptPath: inv.TranscriptPath,

286
internal/cli/run_test.go
View File

@@ -616,8 +616,8 @@ func TestRunProcessReportJSONIncludesChunkingSummary(t *testing.T) {
if report.Chunking.MaxSectionTokens == 0 {
t.Errorf("expected max_section_tokens in report")
}
if report.Phase != "phase12-grammar-module" {
t.Errorf("expected phase 'phase12-grammar-module', got %q", report.Phase)
if report.Phase != "phase13-glossary-module" {
t.Errorf("expected phase 'phase13-glossary-module', got %q", report.Phase)
}
}
@@ -664,12 +664,13 @@ func (v fakeValidator) Validate(ctx context.Context, req contracts.ValidationReq
type fakeStructuredLLMClient struct {
validationResponses []validators.LLMValidationResponse
proposalResponses []proposal_generation.StructuredCorrectionSet
calls []string
err error
}
func (f *fakeStructuredLLMClient) CompleteStructured(ctx context.Context, req contracts.StructuredCompletionRequest, out any) (contracts.StructuredCompletionResponse, error) {
_ = ctx
_ = req
f.calls = append(f.calls, req.StageName)
if f.err != nil {
return contracts.StructuredCompletionResponse{}, f.err
}
@@ -923,7 +924,7 @@ func TestRunProcessProductionRegistryUnimplementedModuleFailsCleanly(t *testing.
exitCode := Run([]string{
"process", transcriptPath,
"--glossary", fixturePath("tiny_glossary.yaml"),
"--modules", "glossary",
"--modules", "homophones",
"--work-dir", workDir,
"--work-dir-retention", "always",
"--report-json", reportPath,
@@ -954,7 +955,7 @@ func TestRunProcessProductionRegistryUnimplementedModuleFailsCleanly(t *testing.
}
func TestRunProcessExplicitUnimplementedModulesFailClearly(t *testing.T) {
for _, moduleKey := range []string{"glossary", "homophones", "spoken_word"} {
for _, moduleKey := range []string{"homophones", "spoken_word"} {
t.Run(moduleKey, func(t *testing.T) {
var stdout, stderr bytes.Buffer
transcriptPath := writeFile(t, "transcript.json", `[
@@ -1196,6 +1197,281 @@ func TestRunProcessDefaultBehaviorRemainsDeterministicWithoutExplicitModules(t *
}
}
func TestRunProcessExplicitGlossaryAppliesCorrectionAndReportsDiagnostics(t *testing.T) {
secret := "phase13-secret"
proposalClient := &fakeStructuredLLMClient{
proposalResponses: []proposal_generation.StructuredCorrectionSet{
{
Corrections: []proposal_generation.StructuredCorrectionProposal{
{TargetSegmentID: 1, OriginalText: "gestures", CorrectedText: "Jesters", Confidence: 0.95},
},
},
},
}
validationClient := &fakeStructuredLLMClient{
validationResponses: []validators.LLMValidationResponse{
{Validations: []validators.LLMValidationDecision{{CorrectionIndex: 0, Approved: true, Confidence: 0.99, Reason: "spoken plausible"}}},
{Validations: []validators.LLMValidationDecision{{CorrectionIndex: 0, Approved: true, Confidence: 0.99, Reason: secret}}},
},
}
processProposalLLMClient = proposalClient
processValidationLLMClient = validationClient
t.Setenv("AUDITA_LLM_API_KEY", secret)
t.Setenv("AUDITA_VALIDATION_LLM_API_KEY", secret)
t.Cleanup(func() {
processProposalLLMClient = nil
processValidationLLMClient = nil
})
var stdout, stderr bytes.Buffer
workDir := t.TempDir()
reportPath := filepath.Join(t.TempDir(), "report.json")
outputPath := filepath.Join(t.TempDir(), "out.json")
transcriptPath := writeFile(t, "transcript.json", `[
{"id":1,"speaker":"Alice","start":0.0,"end":1.0,"text":"There were gestures in the hall."}
]`)
exitCode := Run([]string{
"process", transcriptPath,
"--glossary", fixturePath("tiny_glossary.yaml"),
"--modules", "glossary",
"--output", outputPath,
"--report-json", reportPath,
"--work-dir", workDir,
"--work-dir-retention", "always",
}, &stdout, &stderr)
if exitCode != 0 {
t.Fatalf("expected success, got %d stderr=%q", exitCode, stderr.String())
}
if stdout.Len() != 0 {
t.Fatalf("expected empty stdout with --output, got %q", stdout.String())
}
parsed, err := schema.ParseTranscriptJSON(readFile(t, outputPath))
if err != nil {
t.Fatalf("parse output: %v", err)
}
if parsed.Segments[0].Text != "There were Jesters in the hall." {
t.Fatalf("expected glossary correction applied, got %q", parsed.Segments[0].Text)
}
report := readProcessReport(t, reportPath)
if len(report.ModuleResults) != 1 || report.ModuleResults[0].ModuleKey != "glossary" {
t.Fatalf("expected one glossary module result, got %+v", report.ModuleResults)
}
if len(report.ModuleResults[0].AppliedChanges) != 1 {
t.Fatalf("expected one applied glossary change, got %+v", report.ModuleResults[0].AppliedChanges)
}
if len(report.ModuleResults[0].ValidatorDecisions) == 0 {
t.Fatalf("expected validator decisions in report")
}
runDir := onlyRunDir(t, workDir)
runReport := readProcessReport(t, filepath.Join(runDir, "report.json"))
if len(runReport.ModuleResults) != 1 {
t.Fatalf("expected module results in run-dir report")
}
diagFiles, globErr := filepath.Glob(filepath.Join(runDir, "glossary", "*response-payload.json"))
if globErr != nil {
t.Fatalf("glob diagnostics: %v", globErr)
}
if len(diagFiles) == 0 {
t.Fatalf("expected glossary diagnostics payload files in %s", filepath.Join(runDir, "glossary"))
}
for _, f := range diagFiles {
raw := string(readFile(t, f))
if strings.Contains(raw, secret) {
t.Fatalf("secret leaked in diagnostics %q: %s", f, raw)
}
}
}
func TestRunProcessExplicitGlossaryRejectedAndApplicationSkipAreDistinct(t *testing.T) {
processProposalLLMClient = &fakeStructuredLLMClient{
proposalResponses: []proposal_generation.StructuredCorrectionSet{
{
Corrections: []proposal_generation.StructuredCorrectionProposal{
{TargetSegmentID: 1, OriginalText: "there were gestures", CorrectedText: "There were gestures", Confidence: 0.99},
{TargetSegmentID: 1, OriginalText: "there were gestures", CorrectedText: "there were jesters", Confidence: 0.99},
{TargetSegmentID: 1, OriginalText: "hall", CorrectedText: "temple", Confidence: 0.99},
},
},
},
}
processValidationLLMClient = &fakeStructuredLLMClient{
validationResponses: []validators.LLMValidationResponse{
{
Validations: []validators.LLMValidationDecision{
{CorrectionIndex: 0, Approved: true, Confidence: 0.9, Reason: "ok"},
{CorrectionIndex: 1, Approved: true, Confidence: 0.9, Reason: "ok"},
{CorrectionIndex: 2, Approved: false, Confidence: 0.9, Reason: "reject"},
},
},
{
Validations: []validators.LLMValidationDecision{
{CorrectionIndex: 0, Approved: true, Confidence: 0.9, Reason: "ok"},
{CorrectionIndex: 1, Approved: true, Confidence: 0.9, Reason: "ok"},
},
},
},
}
t.Cleanup(func() {
processProposalLLMClient = nil
processValidationLLMClient = nil
})
var stdout, stderr bytes.Buffer
reportPath := filepath.Join(t.TempDir(), "report.json")
outputPath := filepath.Join(t.TempDir(), "out.json")
transcriptPath := writeFile(t, "transcript.json", `[
{"id":1,"speaker":"Alice","start":0.0,"end":1.0,"text":"there were gestures in the hall and there were gestures."}
]`)
exitCode := Run([]string{
"process", transcriptPath,
"--glossary", fixturePath("tiny_glossary.yaml"),
"--modules", "glossary",
"--output", outputPath,
"--report-json", reportPath,
}, &stdout, &stderr)
if exitCode != 0 {
t.Fatalf("expected success, got %d stderr=%q", exitCode, stderr.String())
}
report := readProcessReport(t, reportPath)
if len(report.ModuleResults) != 1 {
t.Fatalf("expected one module result")
}
module := report.ModuleResults[0]
if len(module.ValidatorRejected) != 1 {
t.Fatalf("expected one validator rejection, got %+v", module.ValidatorRejected)
}
if len(module.SkippedChanges) != 1 {
t.Fatalf("expected one application skip, got %+v", module.SkippedChanges)
}
if module.ValidatorRejected[0].ReasonCode == string(module.SkippedChanges[0].SkipReason) {
t.Fatalf("validator rejection and application skip should remain distinct")
}
}
func TestRunProcessExplicitGlossaryRepeatedStagesUseDeterministicInstanceNamesAndSeePriorChanges(t *testing.T) {
proposalClient := &fakeStructuredLLMClient{
proposalResponses: []proposal_generation.StructuredCorrectionSet{
{
Corrections: []proposal_generation.StructuredCorrectionProposal{
{TargetSegmentID: 1, OriginalText: "gestures", CorrectedText: "Jesters", Confidence: 0.99},
},
},
{
Corrections: []proposal_generation.StructuredCorrectionProposal{
{TargetSegmentID: 1, OriginalText: "Jesters", CorrectedText: "JESTERS", Confidence: 0.99},
},
},
},
}
validationClient := &fakeStructuredLLMClient{
validationResponses: []validators.LLMValidationResponse{
{Validations: []validators.LLMValidationDecision{{CorrectionIndex: 0, Approved: true, Confidence: 0.99, Reason: "ok"}}},
{Validations: []validators.LLMValidationDecision{{CorrectionIndex: 0, Approved: true, Confidence: 0.99, Reason: "ok"}}},
{Validations: []validators.LLMValidationDecision{{CorrectionIndex: 0, Approved: true, Confidence: 0.99, Reason: "ok"}}},
{Validations: []validators.LLMValidationDecision{{CorrectionIndex: 0, Approved: true, Confidence: 0.99, Reason: "ok"}}},
},
}
processProposalLLMClient = proposalClient
processValidationLLMClient = validationClient
t.Cleanup(func() {
processProposalLLMClient = nil
processValidationLLMClient = nil
})
var stdout, stderr bytes.Buffer
reportPath := filepath.Join(t.TempDir(), "report.json")
outputPath := filepath.Join(t.TempDir(), "out.json")
transcriptPath := writeFile(t, "transcript.json", `[
{"id":1,"speaker":"Alice","start":0.0,"end":1.0,"text":"There were gestures in the hall."}
]`)
exitCode := Run([]string{
"process", transcriptPath,
"--glossary", fixturePath("tiny_glossary.yaml"),
"--modules", "glossary,glossary",
"--output", outputPath,
"--report-json", reportPath,
}, &stdout, &stderr)
if exitCode != 0 {
t.Fatalf("expected success, got %d stderr=%q", exitCode, stderr.String())
}
parsed, err := schema.ParseTranscriptJSON(readFile(t, outputPath))
if err != nil {
t.Fatalf("parse output: %v", err)
}
if parsed.Segments[0].Text != "There were JESTERS in the hall." {
t.Fatalf("expected second stage to see first stage changes, got %q", parsed.Segments[0].Text)
}
report := readProcessReport(t, reportPath)
if len(report.ModuleResults) != 2 {
t.Fatalf("expected two module results, got %+v", report.ModuleResults)
}
if report.ModuleResults[0].ModuleInstance != "glossary_1" || report.ModuleResults[1].ModuleInstance != "glossary_2" {
t.Fatalf("expected deterministic glossary instance names, got %+v", report.ModuleResults)
}
if len(report.ModuleResults[0].AppliedChanges) != 1 || len(report.ModuleResults[1].AppliedChanges) != 1 {
t.Fatalf("expected one applied change per stage, got %+v", report.ModuleResults)
}
foundStage1 := false
foundStage2 := false
for _, call := range proposalClient.calls {
if strings.HasPrefix(call, "glossary_1:proposal") {
foundStage1 = true
}
if strings.HasPrefix(call, "glossary_2:proposal") {
foundStage2 = true
}
}
if !foundStage1 || !foundStage2 {
t.Fatalf("expected proposal calls for glossary_1 and glossary_2, got %v", proposalClient.calls)
}
}
func TestRunProcessExplicitGlossaryMalformedLLMOutputFailsWithErrorLog(t *testing.T) {
processProposalLLMClient = &fakeStructuredLLMClient{err: errors.New("malformed structured output")}
t.Cleanup(func() { processProposalLLMClient = nil })
var stdout, stderr bytes.Buffer
workDir := t.TempDir()
reportPath := filepath.Join(t.TempDir(), "report.json")
transcriptPath := writeFile(t, "transcript.json", `[
{"id":1,"speaker":"Alice","start":0.0,"end":1.0,"text":"hello"}
]`)
exitCode := Run([]string{
"process", transcriptPath,
"--glossary", fixturePath("tiny_glossary.yaml"),
"--modules", "glossary",
"--work-dir", workDir,
"--work-dir-retention", "always",
"--report-json", reportPath,
}, &stdout, &stderr)
if exitCode == 0 {
t.Fatal("expected failure")
}
if stdout.Len() != 0 {
t.Fatalf("expected empty stdout on failure, got %q", stdout.String())
}
if !strings.Contains(stderr.String(), "runner_execution") {
t.Fatalf("expected runner_execution error, got %q", stderr.String())
}
runDir := onlyRunDir(t, workDir)
if _, err := os.Stat(filepath.Join(runDir, "error.log")); err != nil {
t.Fatalf("expected error.log on failed glossary run: %v", err)
}
report := readProcessReport(t, reportPath)
if report.Status != "failed" || report.ErrorPhase != "runner_execution" {
t.Fatalf("expected failed runner_execution report, got %+v", report)
}
}
func TestRunProcessChunkingSummaryArtifactWritten(t *testing.T) {
var stdout bytes.Buffer
var stderr bytes.Buffer

4
internal/core/reporting/report_test.go
View File

@@ -11,7 +11,7 @@ import (
func TestProcessReportModuleResultsJSONSuccessAndSkipped(t *testing.T) {
now := time.Now().UTC()
report := ProcessReport{
Phase: "phase12-grammar-module",
Phase: "phase13-glossary-module",
Status: "success",
ModuleResults: []ModuleReport{
{
@@ -80,7 +80,7 @@ func TestProcessReportModuleResultsJSONSuccessAndSkipped(t *testing.T) {
func TestProcessReportModuleResultsJSONFailedModule(t *testing.T) {
now := time.Now().UTC()
report := ProcessReport{
Phase: "phase12-grammar-module",
Phase: "phase13-glossary-module",
Status: "failed",
ModuleResults: []ModuleReport{
{

8
internal/framework/modules/registry.go
View File

@@ -8,6 +8,7 @@ import (
"gitea.maximumdirect.net/eric/audita/internal/core/config"
"gitea.maximumdirect.net/eric/audita/internal/core/schema"
"gitea.maximumdirect.net/eric/audita/internal/framework/contracts"
glossarymodule "gitea.maximumdirect.net/eric/audita/internal/modules/glossary"
grammarmodule "gitea.maximumdirect.net/eric/audita/internal/modules/grammar"
)
@@ -70,6 +71,7 @@ func NewFactory(deps Dependencies) *Factory {
deps: deps,
constructors: make(map[string]Constructor, len(knownModuleKeys)),
}
_ = factory.RegisterConstructor(ModuleKeyGlossary, constructGlossaryModule)
_ = factory.RegisterConstructor(ModuleKeyGrammar, constructGrammarModule)
return factory
}
@@ -96,6 +98,12 @@ func constructGrammarModule(ctx context.Context, req ConstructRequest) (contract
return grammarmodule.New()
}
func constructGlossaryModule(ctx context.Context, req ConstructRequest) (contracts.TranscriptModule, error) {
_ = ctx
_ = req
return glossarymodule.New()
}
// ModuleForSpec resolves one configured run spec into a module instance.
func (f *Factory) ModuleForSpec(spec contracts.ModuleRunSpec) (contracts.TranscriptModule, error) {
if f == nil {

13
internal/framework/modules/registry_test.go
View File

@@ -66,7 +66,7 @@ func TestUnsupportedUnknownModuleKeyFailsCleanly(t *testing.T) {
func TestRecognizedButUnimplementedModuleKeyFailsCleanly(t *testing.T) {
factory := NewFactory(Dependencies{})
for _, key := range []string{ModuleKeyGlossary, ModuleKeyHomophones, ModuleKeySpokenWord} {
for _, key := range []string{ModuleKeyHomophones, ModuleKeySpokenWord} {
t.Run(key, func(t *testing.T) {
_, err := factory.ModuleForSpec(contracts.ModuleRunSpec{ModuleKey: key, InstanceName: key})
if err == nil {
@@ -95,6 +95,17 @@ func TestGrammarIsRegisteredAndConstructibleByDefault(t *testing.T) {
}
}
func TestGlossaryIsRegisteredAndConstructibleByDefault(t *testing.T) {
factory := NewFactory(Dependencies{})
module, err := factory.ModuleForSpec(contracts.ModuleRunSpec{ModuleKey: ModuleKeyGlossary, InstanceName: ModuleKeyGlossary})
if err != nil {
t.Fatalf("ModuleForSpec error: %v", err)
}
if module.Key() != ModuleKeyGlossary {
t.Fatalf("expected glossary module key, got %q", module.Key())
}
}
func TestRegisterConstructorAndConstruct(t *testing.T) {
factory := NewFactory(Dependencies{})
if err := factory.RegisterConstructor(ModuleKeyGlossary, func(ctx context.Context, req ConstructRequest) (contracts.TranscriptModule, error) {

34
internal/framework/runner/runner_test.go
View File

@@ -686,3 +686,37 @@ func TestRunnerGrammarModuleUsesConfidenceThreshold(t *testing.T) {
t.Fatalf("expected low confidence reason, got %+v", out.ModuleResults[0].ValidatorRejected[0])
}
}
func TestRunnerGlossaryModuleUsesConfidenceThreshold(t *testing.T) {
client := &fakeProposalStructuredClient{
responses: []proposal_generation.StructuredCorrectionSet{
{
Corrections: []proposal_generation.StructuredCorrectionProposal{
{TargetSegmentID: 1, OriginalText: "gestures", CorrectedText: "Jesters", Confidence: 0.5},
},
},
},
}
cfg := config.Default()
cfg.Thresholds.Glossary = 0.9
factory := modules.NewFactory(modules.Dependencies{})
out, err := New(factory).Run(context.Background(), RunInput{
Config: &cfg,
Transcript: &schema.Transcript{Segments: []schema.Segment{{ID: 1, Text: "There were gestures"}}},
Glossary: &schema.Glossary{Entries: []schema.GlossaryEntry{{Name: "Jesters", Category: "faction", Summary: "Faction"}}},
ModuleSpecs: []contracts.ModuleRunSpec{{ModuleKey: "glossary", InstanceName: "glossary"}},
ProposalLLMClient: client,
})
if err != nil {
t.Fatalf("Run error: %v", err)
}
if out.FinalTranscript.Segments[0].Text != "There were gestures" {
t.Fatalf("expected no changes due to glossary confidence threshold, got %q", out.FinalTranscript.Segments[0].Text)
}
if len(out.ModuleResults) != 1 || len(out.ModuleResults[0].ValidatorRejected) == 0 {
t.Fatalf("expected validator rejection, got %+v", out.ModuleResults)
}
if out.ModuleResults[0].ValidatorRejected[0].ReasonCode != validators.ReasonLowConfidence {
t.Fatalf("expected low confidence reason, got %+v", out.ModuleResults[0].ValidatorRejected[0])
}
}

55
internal/framework/validators/deterministic.go
View File

@@ -4,8 +4,6 @@ import (
"context"
"fmt"
"strings"
"gitea.maximumdirect.net/eric/audita/internal/framework/proposals"
)
type ConfidenceThresholdValidator struct{}
@@ -110,11 +108,12 @@ func (v ProtectedGlossaryTermValidator) Validate(_ context.Context, req Request)
return Result{ValidatorName: v.Name(), Decisions: decisions}, nil
}
terms := glossaryTerms(req)
vocab := NewProtectedVocabulary(req.Glossary)
decisions := make([]Decision, 0, len(req.CandidateProposal))
for _, c := range req.CandidateProposal {
if altersProtectedTerm(c.CorrectionProposal, terms) {
decisions = append(decisions, rejection(c.ProposalIndex, ReasonProtectedGlossaryTerm, "proposal may alter protected glossary terminology"))
reason := vocab.violationReason(c.OriginalText, c.CorrectedText)
if reason != "" {
decisions = append(decisions, rejection(c.ProposalIndex, ReasonProtectedGlossaryTerm, reason))
continue
}
decisions = append(decisions, approval(c.ProposalIndex))
@@ -125,37 +124,25 @@ func (v ProtectedGlossaryTermValidator) Validate(_ context.Context, req Request)
return Result{ValidatorName: v.Name(), Decisions: decisions}, nil
}
func glossaryTerms(req Request) []string {
if req.Glossary == nil {
return nil
}
out := make([]string, 0)
for _, e := range req.Glossary.Entries {
if t := strings.TrimSpace(strings.ToLower(e.Name)); t != "" {
out = append(out, t)
}
for _, a := range e.Aliases {
if t := strings.TrimSpace(strings.ToLower(a)); t != "" {
out = append(out, t)
}
}
if t := strings.TrimSpace(strings.ToLower(e.Plural)); t != "" {
out = append(out, t)
}
}
return out
type GlossaryStageProtectedGlossaryTermValidator struct{}
func (v GlossaryStageProtectedGlossaryTermValidator) Name() string {
return "glossary_stage_protected_glossary_terms"
}
func altersProtectedTerm(p proposals.CorrectionProposal, terms []string) bool {
if len(terms) == 0 {
return false
}
orig := strings.ToLower(p.OriginalText)
corr := strings.ToLower(p.CorrectedText)
for _, t := range terms {
if strings.Contains(orig, t) && !strings.Contains(corr, t) {
return true
func (v GlossaryStageProtectedGlossaryTermValidator) Validate(_ context.Context, req Request) (Result, error) {
vocab := NewProtectedVocabulary(req.Glossary)
decisions := make([]Decision, 0, len(req.CandidateProposal))
for _, c := range req.CandidateProposal {
reason := vocab.glossaryStageViolationReason(c.OriginalText, c.CorrectedText)
if reason != "" {
decisions = append(decisions, rejection(c.ProposalIndex, ReasonProtectedGlossaryTerm, reason))
continue
}
decisions = append(decisions, approval(c.ProposalIndex))
}
return false
if err := EnforceDecisionCardinality(req.CandidateProposal, decisions); err != nil {
return Result{}, err
}
return Result{ValidatorName: v.Name(), Decisions: decisions}, nil
}

188
internal/framework/validators/protected_terms.go Normal file
View File

@@ -0,0 +1,188 @@
package validators
import (
"regexp"
"sort"
"strings"
"gitea.maximumdirect.net/eric/audita/internal/core/schema"
)
type protectedTermDef struct {
identity int
canonical string
}
type protectedOccurrence struct {
text string
identity int
canonical string
}
// ProtectedVocabulary is a deterministic glossary-derived protected term set.
type ProtectedVocabulary struct {
termsByFolded map[string]protectedTermDef
pattern *regexp.Regexp
}
// ExtractProtectedTerms returns a stable, de-duplicated list of protected terms
// derived from glossary names, aliases, synthetic plural forms, and explicit
// plural fields.
func ExtractProtectedTerms(glossary *schema.Glossary) []string {
vocab := NewProtectedVocabulary(glossary)
out := make([]string, 0, len(vocab.termsByFolded))
for _, def := range vocab.termsByFolded {
out = append(out, def.canonical)
}
sort.SliceStable(out, func(i, j int) bool {
li := strings.ToLower(out[i])
lj := strings.ToLower(out[j])
if li == lj {
return out[i] < out[j]
}
return li < lj
})
return out
}
// NewProtectedVocabulary builds a deterministic protected vocabulary from a glossary.
func NewProtectedVocabulary(glossary *schema.Glossary) ProtectedVocabulary {
termsByFolded := make(map[string]protectedTermDef)
if glossary != nil {
for identity, entry := range glossary.Entries {
entryTerms := append([]string{entry.Name}, entry.Aliases...)
for _, term := range entryTerms {
trimmed := strings.TrimSpace(term)
if trimmed == "" {
continue
}
addProtectedTerm(termsByFolded, trimmed, identity)
addProtectedTerm(termsByFolded, trimmed+"s", identity)
}
addProtectedTerm(termsByFolded, entry.Plural, identity)
}
}
alternatives := make([]string, 0, len(termsByFolded))
for _, def := range termsByFolded {
alternatives = append(alternatives, regexp.QuoteMeta(def.canonical))
}
sort.SliceStable(alternatives, func(i, j int) bool { return len(alternatives[i]) > len(alternatives[j]) })
if len(alternatives) == 0 {
return ProtectedVocabulary{termsByFolded: termsByFolded}
}
pattern := regexp.MustCompile(`(?i)\b(?:` + strings.Join(alternatives, "|") + `)\b`)
return ProtectedVocabulary{termsByFolded: termsByFolded, pattern: pattern}
}
func addProtectedTerm(terms map[string]protectedTermDef, term string, identity int) {
trimmed := strings.TrimSpace(term)
if trimmed == "" {
return
}
folded := strings.ToLower(trimmed)
if _, ok := terms[folded]; ok {
return
}
terms[folded] = protectedTermDef{identity: identity, canonical: trimmed}
}
func (v ProtectedVocabulary) violationReason(before, after string) string {
beforeByID := v.occurrencesByIdentity(before)
afterByID := v.occurrencesByIdentity(after)
if reason := validateIdentityPreservation(beforeByID, afterByID); reason != "" {
return reason
}
if reason := validateCapitalizationTransitions(beforeByID, afterByID); reason != "" {
return reason
}
return ""
}
func (v ProtectedVocabulary) glossaryStageViolationReason(before, after string) string {
beforeByID := v.occurrencesByIdentity(before)
afterByID := v.occurrencesByIdentity(after)
if reason := validateGlossaryStageIdentityPreservation(beforeByID, afterByID); reason != "" {
return reason
}
if reason := validateCapitalizationTransitions(beforeByID, afterByID); reason != "" {
return reason
}
return ""
}
func (v ProtectedVocabulary) occurrencesByIdentity(text string) map[int][]protectedOccurrence {
byID := make(map[int][]protectedOccurrence)
for _, o := range v.occurrences(text) {
byID[o.identity] = append(byID[o.identity], o)
}
return byID
}
func (v ProtectedVocabulary) occurrences(text string) []protectedOccurrence {
if v.pattern == nil {
return nil
}
matches := v.pattern.FindAllStringIndex(text, -1)
if len(matches) == 0 {
return nil
}
out := make([]protectedOccurrence, 0, len(matches))
for _, idx := range matches {
matched := text[idx[0]:idx[1]]
def, ok := v.termsByFolded[strings.ToLower(matched)]
if !ok {
continue
}
out = append(out, protectedOccurrence{
text: matched,
identity: def.identity,
canonical: def.canonical,
})
}
return out
}
func validateIdentityPreservation(beforeByID, afterByID map[int][]protectedOccurrence) string {
for identity, beforeItems := range beforeByID {
if len(afterByID[identity]) < len(beforeItems) {
return "proposal may alter protected glossary terminology"
}
}
return ""
}
func validateGlossaryStageIdentityPreservation(beforeByID, afterByID map[int][]protectedOccurrence) string {
beforeTotal := 0
for _, items := range beforeByID {
beforeTotal += len(items)
}
afterTotal := 0
for _, items := range afterByID {
afterTotal += len(items)
}
if afterTotal < beforeTotal {
return "proposal may alter protected glossary terminology"
}
return ""
}
func validateCapitalizationTransitions(beforeByID, afterByID map[int][]protectedOccurrence) string {
for identity, afterItems := range afterByID {
beforeItems := beforeByID[identity]
for i, afterItem := range afterItems {
if i >= len(beforeItems) {
continue
}
beforeItem := beforeItems[i]
if afterItem.text == beforeItem.text {
continue
}
if afterItem.text == afterItem.canonical {
continue
}
return "proposal may alter protected glossary terminology"
}
}
return ""
}

86
internal/framework/validators/protected_terms_test.go Normal file
View File

@@ -0,0 +1,86 @@
package validators
import (
"context"
"reflect"
"testing"
"gitea.maximumdirect.net/eric/audita/internal/core/schema"
"gitea.maximumdirect.net/eric/audita/internal/framework/proposals"
)
func TestExtractProtectedTermsStableDeduplicatedWithPlurals(t *testing.T) {
glossary := &schema.Glossary{
Entries: []schema.GlossaryEntry{
{Name: "Jesters", Aliases: []string{"Jester", " "}, Plural: "Jesters", Category: "faction", Summary: "Faction"},
{Name: "Hrank", Aliases: []string{"hrank", "Hrank"}, Plural: "Hranks", Category: "pc", Summary: "Character"},
{Name: "Lyra", Aliases: []string{}, Plural: "", Category: "npc", Summary: "NPC"},
},
}
got := ExtractProtectedTerms(glossary)
want := []string{"Hrank", "Hranks", "Jester", "Jesters", "Jesterss", "Lyra", "Lyras"}
if !reflect.DeepEqual(got, want) {
t.Fatalf("unexpected protected terms\n got: %v\nwant: %v", got, want)
}
}
func TestExtractProtectedTermsEmptyGlossary(t *testing.T) {
if terms := ExtractProtectedTerms(&schema.Glossary{}); len(terms) != 0 {
t.Fatalf("expected no terms, got %v", terms)
}
}
func TestProtectedGlossaryTermValidatorAppliesToNonGlossaryModule(t *testing.T) {
req := Request{
ModuleKey: "grammar",
Glossary: &schema.Glossary{
Entries: []schema.GlossaryEntry{{Name: "Jesters", Category: "faction", Summary: "Faction"}},
},
CandidateProposal: []proposals.EnrichedCorrectionProposal{
{
CorrectionProposal: proposals.CorrectionProposal{
TargetSegmentID: 1,
OriginalText: "Jesters",
CorrectedText: "Gestures",
Confidence: 0.9,
},
ProposalMetadata: proposals.ProposalMetadata{ProposalIndex: 0},
},
},
}
res, err := (ProtectedGlossaryTermValidator{}).Validate(context.Background(), req)
if err != nil {
t.Fatalf("validator error: %v", err)
}
if len(res.Decisions) != 1 || res.Decisions[0].Approved {
t.Fatalf("expected protected-term rejection, got %+v", res.Decisions)
}
}
func TestGlossaryStageProtectedGlossaryTermValidatorAllowsProtectedTermSwap(t *testing.T) {
req := Request{
ModuleKey: "glossary",
Glossary: &schema.Glossary{
Entries: []schema.GlossaryEntry{{Name: "Jesters", Category: "faction", Summary: "Faction"}},
},
CandidateProposal: []proposals.EnrichedCorrectionProposal{
{
CorrectionProposal: proposals.CorrectionProposal{
TargetSegmentID: 1,
OriginalText: "gestures",
CorrectedText: "Jesters",
Confidence: 0.9,
},
ProposalMetadata: proposals.ProposalMetadata{ProposalIndex: 0},
},
},
}
res, err := (GlossaryStageProtectedGlossaryTermValidator{}).Validate(context.Background(), req)
if err != nil {
t.Fatalf("validator error: %v", err)
}
if len(res.Decisions) != 1 || !res.Decisions[0].Approved {
t.Fatalf("expected approval for glossary-stage protected term correction, got %+v", res.Decisions)
}
}

76
internal/modules/glossary/module.go Normal file
View File

@@ -0,0 +1,76 @@
package glossary
import (
"context"
"fmt"
"gitea.maximumdirect.net/eric/audita/internal/framework/contracts"
"gitea.maximumdirect.net/eric/audita/internal/framework/proposal_generation"
"gitea.maximumdirect.net/eric/audita/internal/framework/proposals"
"gitea.maximumdirect.net/eric/audita/internal/framework/validators"
)
type Module struct {
validators []contracts.Validator
}
func New() (*Module, error) {
spokenForm, err := validators.NewLLMBackedValidator("spoken_form_plausibility_review", validators.LLMValidatorTypeSpokenFormPlausibility, "")
if err != nil {
return nil, err
}
meaningReversal, err := validators.NewLLMBackedValidator("meaning_reversal_review", validators.LLMValidatorTypeMeaningReversal, "")
if err != nil {
return nil, err
}
return &Module{
validators: []contracts.Validator{
validators.NoEffectValidator{},
validators.OriginalTextPresenceValidator{},
validators.ConfidenceThresholdValidator{},
validators.GlossaryStageProtectedGlossaryTermValidator{},
validators.NonEmptyCorrectionValidator{},
spokenForm,
meaningReversal,
},
}, nil
}
func (m *Module) Key() string { return "glossary" }
func (m *Module) ReplacementPolicy() proposals.ReplacementPolicy {
// Python glossary module uses replace_all to update repeated term occurrences.
return proposals.ReplacementPolicyReplaceAll
}
func (m *Module) Validators() []contracts.Validator {
return append([]contracts.Validator(nil), m.validators...)
}
func (m *Module) Propose(ctx context.Context, req contracts.ProposalRequest) ([]proposals.CorrectionProposal, error) {
messages, err := BuildProposalMessages(req.WorkingTranscript, req.Glossary)
if err != nil {
return nil, err
}
generated, err := proposal_generation.GenerateCandidates(ctx, proposal_generation.Request{
ModuleKey: req.RunSpec.ModuleKey,
ModuleInstance: req.RunSpec.InstanceName,
ReplacementPolicy: req.RunSpec.ReplacementPolicy,
WorkingTranscript: req.WorkingTranscript,
Section: req.Section,
Glossary: req.Glossary,
Config: req.Config,
Messages: messages,
StageName: fmt.Sprintf("%s:proposal", req.RunSpec.InstanceName),
StartIndex: 0,
LLMClient: req.LLMClient,
Scheduler: req.LLMScheduler,
DiagnosticsDir: req.DiagnosticsDir,
})
if err != nil {
return nil, err
}
return generated.Corrections, nil
}

188
internal/modules/glossary/module_test.go Normal file
View File

@@ -0,0 +1,188 @@
package glossary
import (
"context"
"errors"
"os"
"path/filepath"
"strings"
"testing"
"gitea.maximumdirect.net/eric/audita/internal/core/config"
"gitea.maximumdirect.net/eric/audita/internal/core/schema"
"gitea.maximumdirect.net/eric/audita/internal/framework/contracts"
"gitea.maximumdirect.net/eric/audita/internal/framework/proposal_generation"
"gitea.maximumdirect.net/eric/audita/internal/framework/proposals"
)
type fakeLLMClient struct {
responses []proposal_generation.StructuredCorrectionSet
err error
calls []contracts.StructuredCompletionRequest
}
func (f *fakeLLMClient) CompleteStructured(ctx context.Context, req contracts.StructuredCompletionRequest, out any) (contracts.StructuredCompletionResponse, error) {
_ = ctx
f.calls = append(f.calls, req)
if f.err != nil {
return contracts.StructuredCompletionResponse{}, f.err
}
target, ok := out.(*proposal_generation.StructuredCorrectionSet)
if !ok {
return contracts.StructuredCompletionResponse{}, errors.New("unexpected output type")
}
if len(f.responses) == 0 {
return contracts.StructuredCompletionResponse{}, errors.New("unexpected call")
}
*target = f.responses[0]
f.responses = f.responses[1:]
return contracts.StructuredCompletionResponse{}, nil
}
type countingScheduler struct {
runs int
}
func (s *countingScheduler) Run(ctx context.Context, fn func(context.Context) error) error {
s.runs++
return fn(ctx)
}
func tinyTranscript() *schema.Transcript {
return &schema.Transcript{Segments: []schema.Segment{
{ID: 1, Speaker: "A", Start: 0, End: 1, Text: "There were gestures in the hall.", Categories: []string{"session"}},
}}
}
func tinyGlossary() *schema.Glossary {
return &schema.Glossary{Entries: []schema.GlossaryEntry{
{Name: "Jesters", Aliases: []string{"Jester"}, Category: "faction", Summary: "Guild members", Plural: "Jesters"},
}}
}
func TestBuildProposalMessagesContainsGlossaryContextAndConstraints(t *testing.T) {
msgs, err := BuildProposalMessages(tinyTranscript(), tinyGlossary())
if err != nil {
t.Fatalf("BuildProposalMessages error: %v", err)
}
if len(msgs) != 2 {
t.Fatalf("expected 2 messages, got %d", len(msgs))
}
combined := msgs[0].Content + "\n" + msgs[1].Content
for _, want := range []string{
"Glossary:",
"Transcript section:",
`"Aliases":`,
`"Jester"`,
`"Category": "faction"`,
`"Summary": "Guild members"`,
`"Plural": "Jesters"`,
"glossary-supported corrections",
"acoustically similar",
"Do not make generic grammar, spelling, capitalization, style, or filler-word edits",
} {
if !strings.Contains(combined, want) {
t.Fatalf("expected prompt to contain %q", want)
}
}
if strings.Contains(strings.ToLower(combined), "summarize the transcript") {
t.Fatalf("prompt should not invite summarization")
}
}
func TestGlossaryModuleReplacementPolicy(t *testing.T) {
m, err := New()
if err != nil {
t.Fatalf("New error: %v", err)
}
if m.ReplacementPolicy() != proposals.ReplacementPolicyReplaceAll {
t.Fatalf("unexpected replacement policy: %q", m.ReplacementPolicy())
}
}
func TestGlossaryModuleValidatorChain(t *testing.T) {
m, err := New()
if err != nil {
t.Fatalf("New error: %v", err)
}
got := make([]string, 0, len(m.Validators()))
for _, v := range m.Validators() {
got = append(got, v.Name())
}
want := []string{
"no_effect",
"original_text_presence",
"confidence_threshold",
"glossary_stage_protected_glossary_terms",
"non_empty_correction",
"spoken_form_plausibility_review",
"meaning_reversal_review",
}
if strings.Join(got, ",") != strings.Join(want, ",") {
t.Fatalf("unexpected validator chain\n got: %v\nwant: %v", got, want)
}
}
func TestGlossaryModuleProposeMapsCorrectionsAndWritesDiagnostics(t *testing.T) {
secret := "glossary-secret"
client := &fakeLLMClient{
responses: []proposal_generation.StructuredCorrectionSet{
{
Corrections: []proposal_generation.StructuredCorrectionProposal{
{TargetSegmentID: 1, OriginalText: "gestures", CorrectedText: "Jesters", Confidence: 0.94},
},
},
},
}
scheduler := &countingScheduler{}
cfg := config.Default()
cfg.PrimaryLLM.APIKey = secret
m, err := New()
if err != nil {
t.Fatalf("New error: %v", err)
}
diagDir := t.TempDir()
out, err := m.Propose(context.Background(), contracts.ProposalRequest{
ExecutionContext: contracts.ExecutionContext{
Config: &cfg,
WorkingTranscript: tinyTranscript(),
Glossary: tinyGlossary(),
DiagnosticsDir: diagDir,
},
RunSpec: contracts.ModuleRunSpec{
ModuleKey: "glossary",
InstanceName: "glossary",
ReplacementPolicy: proposals.ReplacementPolicyReplaceAll,
},
LLMClient: client,
LLMScheduler: scheduler,
})
if err != nil {
t.Fatalf("Propose error: %v", err)
}
if scheduler.runs != 1 {
t.Fatalf("expected scheduler run count 1, got %d", scheduler.runs)
}
if len(client.calls) != 1 || client.calls[0].StageName != "glossary:proposal" {
t.Fatalf("expected one glossary:proposal call, got %+v", client.calls)
}
if len(out) != 1 || out[0].CorrectedText != "Jesters" {
t.Fatalf("unexpected proposals: %+v", out)
}
diagFiles, globErr := filepath.Glob(filepath.Join(diagDir, "glossary", "*proposal*response-payload.json"))
if globErr != nil {
t.Fatalf("glob error: %v", globErr)
}
if len(diagFiles) == 0 {
t.Fatalf("expected diagnostics response payload under %s", filepath.Join(diagDir, "glossary"))
}
for _, f := range diagFiles {
raw, readErr := os.ReadFile(f)
if readErr != nil {
t.Fatalf("read diag %q: %v", f, readErr)
}
if strings.Contains(string(raw), secret) {
t.Fatalf("secret leaked in diagnostics: %s", string(raw))
}
}
}

80
internal/modules/glossary/prompt.go Normal file
View File

@@ -0,0 +1,80 @@
package glossary
import (
"encoding/json"
"fmt"
"gitea.maximumdirect.net/eric/audita/internal/core/schema"
"gitea.maximumdirect.net/eric/audita/internal/framework/contracts"
)
type promptSegment struct {
ID int `json:"id"`
Speaker string `json:"speaker"`
Start float64 `json:"start"`
End float64 `json:"end"`
Text string `json:"text"`
Categories []string `json:"categories,omitempty"`
}
type promptTranscriptSection struct {
SectionIndex int `json:"section_index"`
Segments []promptSegment `json:"segments"`
}
func BuildProposalMessages(transcript *schema.Transcript, glossary *schema.Glossary) ([]contracts.LLMMessage, error) {
glossaryJSON, err := json.MarshalIndent(glossary, "", " ")
if err != nil {
return nil, fmt.Errorf("marshal glossary prompt context: %w", err)
}
sectionPayload := promptTranscriptSection{
SectionIndex: 0,
Segments: make([]promptSegment, 0),
}
if transcript != nil {
for _, s := range transcript.Segments {
sectionPayload.Segments = append(sectionPayload.Segments, promptSegment{
ID: s.ID,
Speaker: s.Speaker,
Start: s.Start,
End: s.End,
Text: s.Text,
Categories: append([]string(nil), s.Categories...),
})
}
}
sectionJSON, err := json.MarshalIndent(sectionPayload, "", " ")
if err != nil {
return nil, fmt.Errorf("marshal transcript prompt context: %w", err)
}
system := "You are Audita, a careful transcript correction assistant. Identify only transcription errors that are strongly supported by the glossary. A valid correction must be acoustically plausible: the original transcript text should sound similar to the proposed correction when spoken aloud. Do not make generic grammar, spelling, capitalization, style, or filler-word edits. Do not substitute an unrelated glossary term just because it could fit the topic. Preserve speaker names, timestamps, and meaning."
user := "Review this transcript section and return only glossary-supported corrections that should be applied.\n\n" +
"Rules:\n" +
"- Correct domain-specific names, aliases, jargon, deities, locations, NPCs, players, factions, and similar terms only when both the glossary and surrounding transcript context support the correction.\n" +
"- The correction must plausibly fix a transcription error: the original words should be phonetically or acoustically similar to the corrected words in spoken English.\n" +
"- Appropriate example: correcting \"gestures\" to \"Jesters\" can be valid if \"Jesters\" appears in the glossary and nearby context supports that inference.\n" +
"- Inappropriate example: correcting \"Lyra\" to \"Jesters\" should be omitted because those words are not similar in spoken English, even if \"Jesters\" appears in the glossary.\n" +
"- Do not replace one clear glossary term, character name, location, or ordinary word with a different glossary term unless it is a plausible mishearing.\n" +
"- Treat glossary names and aliases already present in the transcript as protected spellings.\n" +
"- Do not replace, Anglicize, normalize, lowercase, or otherwise alter protected glossary names or aliases away from their glossary spelling.\n" +
"- Preserve canonical glossary capitalization for protected names and aliases, even if they look unusual.\n" +
"- If a segment includes categories, treat them as additional transcript context.\n" +
"- Plural forms of glossary names and aliases are allowed targets when spoken similarity and context support them, even if the plural is not explicitly listed in the glossary.\n" +
"- Use the exact id from the input segment.\n" +
"- For returned corrections, original_text must be only the exact text span that needs replacement, not the full segment text unless the whole segment is the replacement span.\n" +
"- corrected_text must be only the replacement text for that span, not the full corrected segment text unless the whole segment is the replacement span.\n" +
"- Each returned correction must contain only id, original_text, corrected_text, and confidence.\n" +
"- Do not return corrections where original_text and corrected_text are identical.\n" +
"- Do not return speaker, start, or end fields.\n" +
"- Return only changed segments; do not return entries for unchanged segments.\n" +
"- confidence must be between 0.0 and 1.0.\n" +
"- If no corrections are needed, return an empty corrections list.\n\n" +
fmt.Sprintf("Glossary:\n%s\n\nTranscript section:\n%s", string(glossaryJSON), string(sectionJSON))
return []contracts.LLMMessage{
{Role: "system", Content: system},
{Role: "user", Content: user},
}, nil
}