Complete Phase 15 spoken-word module

This commit is contained in:
2026-05-12 11:56:28 +00:00
parent dbf3605712
commit a9f7fa27ff
10 changed files with 945 additions and 78 deletions

View File

@@ -36,14 +36,18 @@ Implemented today:
- Production `grammar` module implementation in `internal/modules/grammar`.
- Production `glossary` module implementation in `internal/modules/glossary`.
- Production `homophones` module implementation in `internal/modules/homophones`.
- Production `spoken_word` module implementation in `internal/modules/spoken_word`.
- Explicit runtime support for `--modules grammar` through the production runner path.
- Explicit runtime support for `--modules glossary`, including repeated stages such as `--modules glossary,glossary`.
- Explicit runtime support for `--modules homophones` through the production runner path.
- Explicit runtime support for `--modules spoken_word` through the production runner path.
Not implemented in CLI runtime path today:
- Real module execution pipeline for `spoken_word`.
- End-to-end transcript polishing with the full default module sequence.
Current reality:
- all production modules exist and are explicitly runnable by `--modules`, but default full-sequence integration remains a later phase.
Phase sequencing note:
- Phase 9 LLM infrastructure is complete (structured client, scheduler, effective config resolution, diagnostics primitives);
- Phase 10 LLM-backed validator runtime integration is complete;
@@ -51,7 +55,8 @@ Phase sequencing note:
- Phase 12 grammar module implementation and explicit runtime wiring are complete;
- Phase 13 glossary module and protected-term behavior are complete;
- Phase 14 homophones module implementation and explicit runtime wiring are complete;
- next recommended phase is Phase 15 (spoken-word module).
- Phase 15 spoken-word module implementation and explicit runtime wiring are complete;
- next recommended phase is Phase 16 (default full pipeline integration).
## Actual Go package layout
@@ -122,6 +127,10 @@ internal/modules/homophones/
module.go
prompt.go
internal/modules/spoken_word/
module.go
prompt.go
internal/framework/validators/
models.go
deterministic.go
@@ -157,17 +166,17 @@ Current runtime flow (`internal/cli/run.go`):
10. Chunk normalized transcript and compute chunk summaries.
11. Write chunking summary artifact.
12. Execute runner modules sequentially when:
- `--modules` is explicitly provided (production grammar/glossary/homophones paths); or
- `--modules` is explicitly provided (production grammar/glossary/homophones/spoken_word paths); or
- a test/injected module factory is provided.
13. Output working transcript to `--output` file or stdout.
14. Build process report (`phase` currently set to `phase14-homophones-module`).
14. Build process report (`phase` currently set to `phase15-spoken-word-module`).
15. Optionally write `--report-json`; always write run-dir `report.json`.
16. Apply work-dir retention.
Important behavior details:
- Glossary is validated and is used for explicit glossary/grammar/homophones module correction paths.
- Glossary is validated and is used for explicit glossary/grammar/homophones/spoken_word module correction paths.
- Default production CLI behavior remains deterministic normalization/chunking/reporting unless modules are explicitly selected with `--modules`.
- Explicit `--modules grammar`, `--modules glossary`, and `--modules homophones` run production module paths with LLM-backed proposal generation and validator-chain execution.
- Explicit `--modules grammar`, `--modules glossary`, `--modules homophones`, and `--modules spoken_word` run production module paths with LLM-backed proposal generation and validator-chain execution.
- Default runs (without explicit module selection) do not perform LLM calls.
- Success path is generally quiet on stderr.
- Source IDs are preserved into a canonical transcript before normalization; normalization then reassigns output IDs sequentially from `1`.
@@ -216,7 +225,7 @@ Implemented config surfaces include:
- work-dir and retention mode
Current caveat:
- LLM/module-related settings are active for explicit grammar/glossary/homophones runs; the default non-explicit path remains deterministic.
- LLM/module-related settings are active for explicit grammar/glossary/homophones/spoken_word runs; the default non-explicit path remains deterministic.
## Implemented structured LLM infrastructure
`internal/framework/contracts` now defines a typed structured-completion contract:
@@ -233,7 +242,7 @@ Current caveat:
Current runtime boundary:
- the default CLI runtime path (without explicit module selection) still does not instantiate the full production module sequence.
- LLM calls are exercised in production when `--modules grammar`, `--modules glossary`, or `--modules homophones` is explicitly requested and in tests when fake/injected clients are used.
- LLM calls are exercised in production when `--modules grammar`, `--modules glossary`, `--modules homophones`, or `--modules spoken_word` is explicitly requested and in tests when fake/injected clients are used.
`internal/framework/llm` also provides:
- a bounded `Scheduler` for controlled concurrent LLM calls with reliable permit release;
@@ -275,7 +284,7 @@ Current behavior details:
`internal/framework/contracts` provides interfaces and run-spec metadata scaffolding, including deterministic repeated module instance naming (`ResolveModuleRunSpecs`).
These primitives are wired into the production runner and report model. The grammar and glossary modules are implemented; other production modules remain pending.
These primitives are wired into the production runner and report model. The grammar, glossary, homophones, and spoken_word modules are implemented.
## Implemented validator runtime infrastructure
`internal/framework/validators` provides deterministic validator infrastructure:
@@ -340,7 +349,7 @@ This helper only produces candidate proposals; validator-chain execution and pro
- diagnostics directory context
- returns explicit errors for unknown keys (`unsupported_module`) and recognized-but-unimplemented keys (`unimplemented_module`).
The `grammar`, `glossary`, and `homophones` module keys are now registered and constructible. `spoken_word` remains recognized-but-unimplemented.
The `grammar`, `glossary`, `homophones`, and `spoken_word` module keys are now registered and constructible.
## Implemented grammar production module
`internal/modules/grammar` now provides the first production module:
@@ -387,6 +396,19 @@ This vocabulary is used by deterministic validators for both glossary-stage and
- protected-term guardrails for non-glossary modules remain active and are exercised through the homophones path;
- module-level reporting and diagnostics capture through existing runner/reporting paths.
## Implemented spoken_word production module
`internal/modules/spoken_word` now provides the fourth production module:
- prompt builder aligned to Python spoken_word-module intent, constrained to conservative dysfluency cleanup;
- strong prompt guardrails preserving meaning/intent/voice/named entities/domain terms and substantive content;
- explicit guardrails against summarization, style rewriting, grammar-only cleanup, punctuation-only cleanup, invention, and meaning-changing rewrites;
- proposal generation through `internal/framework/proposal_generation` and `contracts.StructuredLLMClient`;
- scheduler-aware proposal calls through existing `contracts.LLMScheduler` hooks;
- replacement policy `require_unique` (matching Python spoken_word behavior);
- validator chain integration using existing deterministic + LLM-backed validators, including strong semantic guardrails (`spoken_word_review`, `meaning_reversal_review`);
- spoken_word confidence threshold enforcement through existing validator/config infrastructure;
- protected-term guardrails for non-glossary modules remain active and are exercised through the spoken_word path;
- module-level reporting and diagnostics capture through existing runner/reporting paths.
## Reports and diagnostics (implemented)
Current per-run artifacts include:
- `source-transcript.json`
@@ -426,7 +448,7 @@ Retention modes implemented in `ApplyRetention`:
Current runtime note:
- default non-explicit runs usually have no module-level skipped corrections, so `auto` commonly removes clean successful run directories.
- explicit grammar/glossary/homophones runs can produce validator rejections and application skips, which are reflected in reports and retention input.
- explicit grammar/glossary/homophones/spoken_word runs can produce validator rejections and application skips, which are reflected in reports and retention input.
Intentionally deferred to module/LLM phases:
- real domain proposal prompts and production module implementations remain tied to later module phases.
@@ -450,14 +472,15 @@ Implemented tests currently cover:
- production grammar module prompt constraints, proposal mapping, validator-chain behavior, confidence-threshold enforcement, diagnostics redaction, and explicit CLI/runtime integration (`internal/modules/grammar/*_test.go`, `internal/cli/run_test.go`, `internal/framework/runner/*_test.go`)
- production glossary module prompt constraints, proposal mapping, validator-chain behavior, confidence-threshold enforcement, diagnostics redaction, repeated-stage behavior, and explicit CLI/runtime integration (`internal/modules/glossary/*_test.go`, `internal/cli/run_test.go`, `internal/framework/runner/*_test.go`)
- production homophones module prompt constraints, proposal mapping, validator-chain behavior, confidence-threshold enforcement, diagnostics redaction, protected-term behavior, and explicit CLI/runtime integration (`internal/modules/homophones/*_test.go`, `internal/cli/run_test.go`, `internal/framework/runner/*_test.go`)
- production spoken_word module prompt constraints, proposal mapping, validator-chain behavior, semantic guardrail behavior, confidence-threshold enforcement, diagnostics redaction, protected-term behavior, and explicit CLI/runtime integration (`internal/modules/spoken_word/*_test.go`, `internal/cli/run_test.go`, `internal/framework/runner/*_test.go`)
- glossary-derived protected-term extraction and stable behavior (`internal/framework/validators/protected_terms_test.go`)
Not covered yet (because not implemented): production `spoken_word` module plus full default-sequence transcript-polishing runtime behavior.
Not covered yet (because not implemented): full default-sequence transcript-polishing runtime behavior as a single default path.
## Intended final architecture (not yet implemented)
The intended end-state still matches the rewrite plan:
- sequential module pipeline over a mutable working transcript
- real module implementation (`spoken_word`)
- default full module-sequence integration in the standard runtime path
- structured LLM proposal generation
- deterministic and LLM validators
- validator cardinality enforcement in pipeline execution

View File

@@ -68,6 +68,10 @@ Implemented:
- Production homophones module package with Python-aligned prompt intent and guardrails.
- Explicit `--modules homophones` runtime path through runner, shared proposal generation, validators, application, reporting, and diagnostics.
- Focused multi-module runtime tests for already-implemented interoperability (for example `glossary,homophones`) without claiming full default-pipeline completion.
- Production spoken_word module package with Python-aligned prompt intent and guardrails.
- Explicit `--modules spoken_word` runtime path through runner, shared proposal generation, validators, application, reporting, and diagnostics.
- Focused multi-module runtime tests for already-implemented interoperability (for example `spoken_word,grammar`) without claiming full default-pipeline completion.
- All production modules now exist (`glossary`, `homophones`, `spoken_word`, `grammar`), but default full-sequence integration remains Phase 16 work.
- Broad deterministic and CLI/subprocess test coverage for implemented phases through `go test ./...`.
- Internal typed structured LLM contract (`StructuredLLMClient.CompleteStructured(ctx, req, out)`).
- `internal/framework/llm` instructor-go-backed adapter with:
@@ -83,8 +87,6 @@ Implemented:
- Generic JSON diagnostics primitives for LLM interactions (request metadata, request payload, response payload, optional error payload) with secret redaction.
Not yet implemented in runtime pipeline:
- Real correction module for `spoken_word`.
- Domain proposal prompts for remaining real module.
- End-to-end transcript polishing behavior with the full default module sequence.
## Completed phases
@@ -223,7 +225,7 @@ Not implemented in Phase 8 (by design):
## Remaining work plan
Next recommended phase: **Phase 15 (spoken-word module)**.
Next recommended phase: **Phase 16 (default full pipeline integration)**.
## Phase 9: Structured LLM client and scheduler infrastructure
@@ -418,41 +420,29 @@ Not implemented in Phase 14 (by design):
## Phase 15: Spoken-word module
### Purpose
Completed.
Implement conservative dysfluency cleanup while preserving substantive meaning.
Implemented:
- Production spoken_word module package in `internal/modules/spoken_word`.
- Spoken_word prompt builder aligned to Python intent and constrained to conservative dysfluency cleanup.
- Prompt context using glossary/protected-term information (names, aliases, plurals where present) to avoid damaging known domain terms.
- Strong prompt guardrails preserving meaning, intent, speaker voice, named entities, game/domain terms, and substantive content.
- Explicit prompt guardrails against summarization, style rewriting, grammar-only cleanup, punctuation-only cleanup, invention, event reordering, and certainty inflation.
- Spoken_word proposal generation through shared `internal/framework/proposal_generation` using `contracts.StructuredLLMClient`.
- Scheduler-aware spoken_word proposal generation through existing scheduler hooks.
- Spoken_word replacement policy `require_unique` (matching Python behavior).
- Spoken_word validator chain using existing deterministic and LLM-backed validators.
- Strong semantic guardrails in runtime validator chain through existing LLM-backed validators (`spoken_word_review`, `meaning_reversal_review`).
- Spoken-word confidence threshold enforcement through existing config + confidence-threshold validator behavior.
- Protected-term guardrails remaining active for spoken_word via existing deterministic validators.
- Explicit runtime support for `--modules spoken_word` through normalization, chunking, runner, proposal generation, validation, application, and reporting.
- Prompt/response diagnostics artifacts for spoken_word proposal + validator interactions with secret redaction.
- Module-level reports for spoken_word including generated proposals, validator decisions/rejections, applied changes, and application skips.
- CLI/runtime fake-client tests for approved cleanup, validator rejection, meaning-changing rejection, application skips, diagnostics, protected-term rejection behavior, failure/error.log behavior, and report outputs (`--report-json` and run-dir `report.json`).
- Focused interoperability tests for already-implemented module combinations (for example `spoken_word,grammar`) to verify working-transcript handoff and guardrails without claiming full default-sequence parity.
### Scope
Implement:
- `spoken_word` module package.
- Spoken-word prompt builder ported from Python.
- Spoken-word structured response model.
- Spoken-word replacement policy.
- Spoken-word confidence threshold handling.
- Spoken-word validator chain.
- Strong semantic guardrails using LLM-backed validators.
- Prompt/response diagnostics.
- CLI support for `--modules spoken_word`.
- Fake LLM tests.
- Tests for rejection of meaning-changing cleanup.
Do not implement:
- Prompt redesign beyond faithful porting.
- New stylistic rewriting behavior not present in the Python implementation.
### Expected behavior at end of phase
Running `audita process ... --modules spoken_word` should perform real conservative dysfluency cleanup, with guardrails against semantic changes.
### Definition of done
- Spoken-word module runs in the production runner.
- Spoken-word proposals are generated through structured LLM calls.
- Semantic validators reject meaning-changing proposals.
- Applied/skipped spoken-word changes appear in reports.
- Prompt/response diagnostics are written.
- `go test ./...` passes without requiring external LLM credentials.
Not implemented in Phase 15 (by design):
- Full default module sequence execution as a feature-complete claim.
## Phase 16: Default full pipeline integration

View File

@@ -460,7 +460,7 @@ func extractErrorPhase(err error) (phase string, message string) {
func buildProcessReport(status string, inv processInvocation, runDir *diagnostics.RunDirectory, startedAt, completedAt time.Time, errorMessage string, errorPhase string, normalizationSummary *normalization.NormalizationSummary, chunkingSummary *chunking.Summary, runOutput *runner.RunOutput) reporting.ProcessReport {
report := reporting.ProcessReport{
Phase: "phase14-homophones-module",
Phase: "phase15-spoken-word-module",
Status: status,
Operation: "process",
TranscriptPath: inv.TranscriptPath,

View File

@@ -616,8 +616,8 @@ func TestRunProcessReportJSONIncludesChunkingSummary(t *testing.T) {
if report.Chunking.MaxSectionTokens == 0 {
t.Errorf("expected max_section_tokens in report")
}
if report.Phase != "phase14-homophones-module" {
t.Errorf("expected phase 'phase14-homophones-module', got %q", report.Phase)
if report.Phase != "phase15-spoken-word-module" {
t.Errorf("expected phase 'phase15-spoken-word-module', got %q", report.Phase)
}
}
@@ -909,7 +909,7 @@ func TestRunProcessInjectedFactoryFailureWritesFailedReport(t *testing.T) {
}
}
func TestRunProcessProductionRegistryUnimplementedModuleFailsCleanly(t *testing.T) {
func TestRunProcessProductionRegistryUnsupportedModuleFailsCleanly(t *testing.T) {
cfg := modules.Dependencies{}
processModuleFactory = modules.NewFactory(cfg)
t.Cleanup(func() { processModuleFactory = nil })
@@ -924,7 +924,7 @@ func TestRunProcessProductionRegistryUnimplementedModuleFailsCleanly(t *testing.
exitCode := Run([]string{
"process", transcriptPath,
"--glossary", fixturePath("tiny_glossary.yaml"),
"--modules", "spoken_word",
"--modules", "made_up",
"--work-dir", workDir,
"--work-dir-retention", "always",
"--report-json", reportPath,
@@ -938,8 +938,8 @@ func TestRunProcessProductionRegistryUnimplementedModuleFailsCleanly(t *testing.
if !strings.Contains(stderr.String(), "runner_execution") {
t.Fatalf("expected runner_execution failure on stderr, got %q", stderr.String())
}
if !strings.Contains(stderr.String(), "recognized but not implemented") {
t.Fatalf("expected explicit unimplemented module message, got %q", stderr.String())
if !strings.Contains(stderr.String(), "unsupported module key") {
t.Fatalf("expected explicit unsupported module message, got %q", stderr.String())
}
report := readProcessReport(t, reportPath)
@@ -949,13 +949,13 @@ func TestRunProcessProductionRegistryUnimplementedModuleFailsCleanly(t *testing.
if report.ErrorPhase != "runner_execution" {
t.Fatalf("expected runner_execution phase, got %q", report.ErrorPhase)
}
if !strings.Contains(report.ErrorMessage, "recognized but not implemented") {
t.Fatalf("expected report error message to mention unimplemented module, got %q", report.ErrorMessage)
if !strings.Contains(report.ErrorMessage, "unsupported module key") {
t.Fatalf("expected report error message to mention unsupported module, got %q", report.ErrorMessage)
}
}
func TestRunProcessExplicitUnimplementedModulesFailClearly(t *testing.T) {
for _, moduleKey := range []string{"spoken_word"} {
func TestRunProcessExplicitUnsupportedModulesFailClearly(t *testing.T) {
for _, moduleKey := range []string{"made_up"} {
t.Run(moduleKey, func(t *testing.T) {
var stdout, stderr bytes.Buffer
transcriptPath := writeFile(t, "transcript.json", `[
@@ -972,8 +972,8 @@ func TestRunProcessExplicitUnimplementedModulesFailClearly(t *testing.T) {
if stdout.Len() != 0 {
t.Fatalf("expected empty stdout on failure, got %q", stdout.String())
}
if !strings.Contains(stderr.String(), "recognized but not implemented") {
t.Fatalf("expected unimplemented message, got %q", stderr.String())
if !strings.Contains(stderr.String(), "unsupported module key") {
t.Fatalf("expected unsupported-module message, got %q", stderr.String())
}
})
}
@@ -1790,6 +1790,385 @@ func TestRunProcessExplicitGlossaryThenHomophonesSeesWorkingTranscriptChanges(t
}
}
func TestRunProcessExplicitSpokenWordAppliesCleanupAndReportsDiagnostics(t *testing.T) {
secret := "phase15-secret"
proposalClient := &fakeStructuredLLMClient{
proposalResponses: []proposal_generation.StructuredCorrectionSet{
{
Corrections: []proposal_generation.StructuredCorrectionProposal{
{TargetSegmentID: 1, OriginalText: "I I think", CorrectedText: "I think", Confidence: 0.95},
},
},
},
}
validationClient := &fakeStructuredLLMClient{
validationResponses: []validators.LLMValidationResponse{
{Validations: []validators.LLMValidationDecision{{CorrectionIndex: 0, Approved: true, Confidence: 0.99, Reason: "spoken cleanup acceptable"}}},
{Validations: []validators.LLMValidationDecision{{CorrectionIndex: 0, Approved: true, Confidence: 0.99, Reason: secret}}},
},
}
processProposalLLMClient = proposalClient
processValidationLLMClient = validationClient
t.Setenv("AUDITA_LLM_API_KEY", secret)
t.Setenv("AUDITA_VALIDATION_LLM_API_KEY", secret)
t.Cleanup(func() {
processProposalLLMClient = nil
processValidationLLMClient = nil
})
var stdout, stderr bytes.Buffer
workDir := t.TempDir()
reportPath := filepath.Join(t.TempDir(), "report.json")
outputPath := filepath.Join(t.TempDir(), "out.json")
transcriptPath := writeFile(t, "transcript.json", `[
{"id":1,"speaker":"Alice","start":0.0,"end":1.0,"text":"I I think we should go."}
]`)
exitCode := Run([]string{
"process", transcriptPath,
"--glossary", fixturePath("tiny_glossary.yaml"),
"--modules", "spoken_word",
"--output", outputPath,
"--report-json", reportPath,
"--work-dir", workDir,
"--work-dir-retention", "always",
}, &stdout, &stderr)
if exitCode != 0 {
t.Fatalf("expected success, got %d stderr=%q", exitCode, stderr.String())
}
if stdout.Len() != 0 {
t.Fatalf("expected empty stdout with --output, got %q", stdout.String())
}
parsed, err := schema.ParseTranscriptJSON(readFile(t, outputPath))
if err != nil {
t.Fatalf("parse output: %v", err)
}
if parsed.Segments[0].Text != "I think we should go." {
t.Fatalf("expected spoken_word cleanup applied, got %q", parsed.Segments[0].Text)
}
report := readProcessReport(t, reportPath)
if len(report.ModuleResults) != 1 || report.ModuleResults[0].ModuleKey != "spoken_word" {
t.Fatalf("expected one spoken_word module result, got %+v", report.ModuleResults)
}
if len(report.ModuleResults[0].AppliedChanges) != 1 {
t.Fatalf("expected one applied spoken_word change, got %+v", report.ModuleResults[0].AppliedChanges)
}
if len(report.ModuleResults[0].ValidatorDecisions) == 0 {
t.Fatalf("expected validator decisions in report")
}
runDir := onlyRunDir(t, workDir)
runReport := readProcessReport(t, filepath.Join(runDir, "report.json"))
if len(runReport.ModuleResults) != 1 || runReport.ModuleResults[0].ModuleKey != "spoken_word" {
t.Fatalf("expected spoken_word module results in run-dir report, got %+v", runReport.ModuleResults)
}
diagFiles, globErr := filepath.Glob(filepath.Join(runDir, "spoken_word", "*response-payload.json"))
if globErr != nil {
t.Fatalf("glob diagnostics: %v", globErr)
}
if len(diagFiles) == 0 {
t.Fatalf("expected spoken_word diagnostics payload files in %s", filepath.Join(runDir, "spoken_word"))
}
for _, f := range diagFiles {
raw := string(readFile(t, f))
if strings.Contains(raw, secret) {
t.Fatalf("secret leaked in diagnostics %q: %s", f, raw)
}
}
}
func TestRunProcessExplicitSpokenWordRejectedAndApplicationSkipAreDistinct(t *testing.T) {
processProposalLLMClient = &fakeStructuredLLMClient{
proposalResponses: []proposal_generation.StructuredCorrectionSet{
{
Corrections: []proposal_generation.StructuredCorrectionProposal{
{TargetSegmentID: 1, OriginalText: "hall", CorrectedText: "temple", Confidence: 0.99},
{TargetSegmentID: 1, OriginalText: "I I think in the hall.", CorrectedText: "I think in the temple.", Confidence: 0.99},
{TargetSegmentID: 1, OriginalText: "I I think", CorrectedText: "I suppose", Confidence: 0.99},
},
},
},
}
processValidationLLMClient = &fakeStructuredLLMClient{
validationResponses: []validators.LLMValidationResponse{
{
Validations: []validators.LLMValidationDecision{
{CorrectionIndex: 0, Approved: true, Confidence: 0.9, Reason: "ok"},
{CorrectionIndex: 1, Approved: true, Confidence: 0.9, Reason: "ok"},
{CorrectionIndex: 2, Approved: false, Confidence: 0.9, Reason: "reject"},
},
},
{
Validations: []validators.LLMValidationDecision{
{CorrectionIndex: 0, Approved: true, Confidence: 0.9, Reason: "ok"},
{CorrectionIndex: 1, Approved: true, Confidence: 0.9, Reason: "ok"},
},
},
},
}
t.Cleanup(func() {
processProposalLLMClient = nil
processValidationLLMClient = nil
})
var stdout, stderr bytes.Buffer
reportPath := filepath.Join(t.TempDir(), "report.json")
outputPath := filepath.Join(t.TempDir(), "out.json")
transcriptPath := writeFile(t, "transcript.json", `[
{"id":1,"speaker":"Alice","start":0.0,"end":1.0,"text":"I I think in the hall."}
]`)
exitCode := Run([]string{
"process", transcriptPath,
"--glossary", fixturePath("tiny_glossary.yaml"),
"--modules", "spoken_word",
"--output", outputPath,
"--report-json", reportPath,
}, &stdout, &stderr)
if exitCode != 0 {
t.Fatalf("expected success, got %d stderr=%q", exitCode, stderr.String())
}
report := readProcessReport(t, reportPath)
if len(report.ModuleResults) != 1 {
t.Fatalf("expected one module result")
}
module := report.ModuleResults[0]
if len(module.ValidatorRejected) != 1 {
t.Fatalf("expected one validator rejection, got %+v", module.ValidatorRejected)
}
if len(module.SkippedChanges) != 1 {
t.Fatalf("expected one application skip, got %+v", module.SkippedChanges)
}
if module.ValidatorRejected[0].ReasonCode == string(module.SkippedChanges[0].SkipReason) {
t.Fatalf("validator rejection and application skip should remain distinct")
}
}
func TestRunProcessExplicitSpokenWordMeaningChangingCleanupRejected(t *testing.T) {
processProposalLLMClient = &fakeStructuredLLMClient{
proposalResponses: []proposal_generation.StructuredCorrectionSet{
{
Corrections: []proposal_generation.StructuredCorrectionProposal{
{TargetSegmentID: 1, OriginalText: "I I think", CorrectedText: "I know", Confidence: 0.99},
},
},
},
}
processValidationLLMClient = &fakeStructuredLLMClient{
validationResponses: []validators.LLMValidationResponse{
{Validations: []validators.LLMValidationDecision{{CorrectionIndex: 0, Approved: false, Confidence: 0.99, Reason: "meaning changed"}}},
},
}
t.Cleanup(func() {
processProposalLLMClient = nil
processValidationLLMClient = nil
})
var stdout, stderr bytes.Buffer
reportPath := filepath.Join(t.TempDir(), "report.json")
outputPath := filepath.Join(t.TempDir(), "out.json")
transcriptPath := writeFile(t, "transcript.json", `[
{"id":1,"speaker":"Alice","start":0.0,"end":1.0,"text":"I I think we should go."}
]`)
exitCode := Run([]string{
"process", transcriptPath,
"--glossary", fixturePath("tiny_glossary.yaml"),
"--modules", "spoken_word",
"--output", outputPath,
"--report-json", reportPath,
}, &stdout, &stderr)
if exitCode != 0 {
t.Fatalf("expected success, got %d stderr=%q", exitCode, stderr.String())
}
parsed, err := schema.ParseTranscriptJSON(readFile(t, outputPath))
if err != nil {
t.Fatalf("parse output: %v", err)
}
if parsed.Segments[0].Text != "I I think we should go." {
t.Fatalf("expected meaning-changing cleanup to be rejected, got %q", parsed.Segments[0].Text)
}
report := readProcessReport(t, reportPath)
if len(report.ModuleResults) != 1 {
t.Fatalf("expected one module result, got %+v", report.ModuleResults)
}
module := report.ModuleResults[0]
if len(module.AppliedChanges) != 0 {
t.Fatalf("expected no applied changes, got %+v", module.AppliedChanges)
}
if len(module.ValidatorRejected) != 1 {
t.Fatalf("expected one semantic guardrail rejection, got %+v", module.ValidatorRejected)
}
if module.ValidatorRejected[0].ValidatorName != "spoken_word_review" {
t.Fatalf("expected spoken_word_review rejection, got %+v", module.ValidatorRejected[0])
}
}
func TestRunProcessExplicitSpokenWordProtectedGlossaryTermRejected(t *testing.T) {
processProposalLLMClient = &fakeStructuredLLMClient{
proposalResponses: []proposal_generation.StructuredCorrectionSet{
{
Corrections: []proposal_generation.StructuredCorrectionProposal{
{TargetSegmentID: 1, OriginalText: "Audita", CorrectedText: "audita", Confidence: 0.99},
},
},
},
}
t.Cleanup(func() {
processProposalLLMClient = nil
processValidationLLMClient = nil
})
var stdout, stderr bytes.Buffer
reportPath := filepath.Join(t.TempDir(), "report.json")
outputPath := filepath.Join(t.TempDir(), "out.json")
transcriptPath := writeFile(t, "transcript.json", `[
{"id":1,"speaker":"Alice","start":0.0,"end":1.0,"text":"Audita held the line."}
]`)
exitCode := Run([]string{
"process", transcriptPath,
"--glossary", fixturePath("tiny_glossary.yaml"),
"--modules", "spoken_word",
"--output", outputPath,
"--report-json", reportPath,
}, &stdout, &stderr)
if exitCode != 0 {
t.Fatalf("expected success, got %d stderr=%q", exitCode, stderr.String())
}
parsed, err := schema.ParseTranscriptJSON(readFile(t, outputPath))
if err != nil {
t.Fatalf("parse output: %v", err)
}
if parsed.Segments[0].Text != "Audita held the line." {
t.Fatalf("expected protected glossary term to remain unchanged, got %q", parsed.Segments[0].Text)
}
report := readProcessReport(t, reportPath)
if len(report.ModuleResults) != 1 {
t.Fatalf("expected one module result, got %+v", report.ModuleResults)
}
module := report.ModuleResults[0]
if len(module.AppliedChanges) != 0 {
t.Fatalf("expected no applied changes, got %+v", module.AppliedChanges)
}
if len(module.ValidatorRejected) != 1 {
t.Fatalf("expected one validator rejection, got %+v", module.ValidatorRejected)
}
if module.ValidatorRejected[0].ReasonCode != validators.ReasonProtectedGlossaryTerm {
t.Fatalf("expected protected glossary term rejection, got %+v", module.ValidatorRejected[0])
}
}
func TestRunProcessExplicitSpokenWordMalformedLLMOutputFailsWithErrorLog(t *testing.T) {
processProposalLLMClient = &fakeStructuredLLMClient{err: errors.New("malformed structured output")}
t.Cleanup(func() { processProposalLLMClient = nil })
var stdout, stderr bytes.Buffer
workDir := t.TempDir()
reportPath := filepath.Join(t.TempDir(), "report.json")
transcriptPath := writeFile(t, "transcript.json", `[
{"id":1,"speaker":"Alice","start":0.0,"end":1.0,"text":"hello"}
]`)
exitCode := Run([]string{
"process", transcriptPath,
"--glossary", fixturePath("tiny_glossary.yaml"),
"--modules", "spoken_word",
"--work-dir", workDir,
"--work-dir-retention", "always",
"--report-json", reportPath,
}, &stdout, &stderr)
if exitCode == 0 {
t.Fatal("expected failure")
}
if stdout.Len() != 0 {
t.Fatalf("expected empty stdout on failure, got %q", stdout.String())
}
if !strings.Contains(stderr.String(), "runner_execution") {
t.Fatalf("expected runner_execution error, got %q", stderr.String())
}
runDir := onlyRunDir(t, workDir)
if _, err := os.Stat(filepath.Join(runDir, "error.log")); err != nil {
t.Fatalf("expected error.log on failed spoken_word run: %v", err)
}
report := readProcessReport(t, reportPath)
if report.Status != "failed" || report.ErrorPhase != "runner_execution" {
t.Fatalf("expected failed runner_execution report, got %+v", report)
}
}
func TestRunProcessExplicitSpokenWordThenGrammarSeesWorkingTranscriptChanges(t *testing.T) {
proposalClient := &fakeStructuredLLMClient{
proposalResponses: []proposal_generation.StructuredCorrectionSet{
{
Corrections: []proposal_generation.StructuredCorrectionProposal{
{TargetSegmentID: 1, OriginalText: "i i think", CorrectedText: "i think", Confidence: 0.99},
},
},
{
Corrections: []proposal_generation.StructuredCorrectionProposal{
{TargetSegmentID: 1, OriginalText: "i think we should go.", CorrectedText: "I think we should go.", Confidence: 0.99},
},
},
},
}
validationClient := &fakeStructuredLLMClient{
validationResponses: []validators.LLMValidationResponse{
{Validations: []validators.LLMValidationDecision{{CorrectionIndex: 0, Approved: true, Confidence: 0.99, Reason: "ok"}}},
{Validations: []validators.LLMValidationDecision{{CorrectionIndex: 0, Approved: true, Confidence: 0.99, Reason: "ok"}}},
{Validations: []validators.LLMValidationDecision{{CorrectionIndex: 0, Approved: true, Confidence: 0.99, Reason: "ok"}}},
{Validations: []validators.LLMValidationDecision{{CorrectionIndex: 0, Approved: true, Confidence: 0.99, Reason: "ok"}}},
},
}
processProposalLLMClient = proposalClient
processValidationLLMClient = validationClient
t.Cleanup(func() {
processProposalLLMClient = nil
processValidationLLMClient = nil
})
var stdout, stderr bytes.Buffer
reportPath := filepath.Join(t.TempDir(), "report.json")
outputPath := filepath.Join(t.TempDir(), "out.json")
transcriptPath := writeFile(t, "transcript.json", `[
{"id":1,"speaker":"Alice","start":0.0,"end":1.0,"text":"i i think we should go."}
]`)
exitCode := Run([]string{
"process", transcriptPath,
"--glossary", fixturePath("tiny_glossary.yaml"),
"--modules", "spoken_word,grammar",
"--output", outputPath,
"--report-json", reportPath,
}, &stdout, &stderr)
if exitCode != 0 {
t.Fatalf("expected success, got %d stderr=%q", exitCode, stderr.String())
}
parsed, err := schema.ParseTranscriptJSON(readFile(t, outputPath))
if err != nil {
t.Fatalf("parse output: %v", err)
}
if parsed.Segments[0].Text != "I think we should go." {
t.Fatalf("expected grammar stage to see spoken_word output, got %q", parsed.Segments[0].Text)
}
report := readProcessReport(t, reportPath)
if len(report.ModuleResults) != 2 {
t.Fatalf("expected two module results, got %+v", report.ModuleResults)
}
if report.ModuleResults[0].ModuleKey != "spoken_word" || report.ModuleResults[1].ModuleKey != "grammar" {
t.Fatalf("expected spoken_word then grammar results, got %+v", report.ModuleResults)
}
if len(report.ModuleResults[0].AppliedChanges) != 1 || len(report.ModuleResults[1].AppliedChanges) != 1 {
t.Fatalf("expected one applied change per module, got %+v", report.ModuleResults)
}
}
func TestRunProcessChunkingSummaryArtifactWritten(t *testing.T) {
var stdout bytes.Buffer
var stderr bytes.Buffer

View File

@@ -11,7 +11,7 @@ import (
func TestProcessReportModuleResultsJSONSuccessAndSkipped(t *testing.T) {
now := time.Now().UTC()
report := ProcessReport{
Phase: "phase14-homophones-module",
Phase: "phase15-spoken-word-module",
Status: "success",
ModuleResults: []ModuleReport{
{
@@ -80,7 +80,7 @@ func TestProcessReportModuleResultsJSONSuccessAndSkipped(t *testing.T) {
func TestProcessReportModuleResultsJSONFailedModule(t *testing.T) {
now := time.Now().UTC()
report := ProcessReport{
Phase: "phase14-homophones-module",
Phase: "phase15-spoken-word-module",
Status: "failed",
ModuleResults: []ModuleReport{
{

View File

@@ -11,6 +11,7 @@ import (
glossarymodule "gitea.maximumdirect.net/eric/audita/internal/modules/glossary"
grammarmodule "gitea.maximumdirect.net/eric/audita/internal/modules/grammar"
homophonesmodule "gitea.maximumdirect.net/eric/audita/internal/modules/homophones"
spokenwordmodule "gitea.maximumdirect.net/eric/audita/internal/modules/spoken_word"
)
const (
@@ -74,6 +75,7 @@ func NewFactory(deps Dependencies) *Factory {
}
_ = factory.RegisterConstructor(ModuleKeyGlossary, constructGlossaryModule)
_ = factory.RegisterConstructor(ModuleKeyHomophones, constructHomophonesModule)
_ = factory.RegisterConstructor(ModuleKeySpokenWord, constructSpokenWordModule)
_ = factory.RegisterConstructor(ModuleKeyGrammar, constructGrammarModule)
return factory
}
@@ -112,6 +114,12 @@ func constructHomophonesModule(ctx context.Context, req ConstructRequest) (contr
return homophonesmodule.New()
}
func constructSpokenWordModule(ctx context.Context, req ConstructRequest) (contracts.TranscriptModule, error) {
_ = ctx
_ = req
return spokenwordmodule.New()
}
// ModuleForSpec resolves one configured run spec into a module instance.
func (f *Factory) ModuleForSpec(spec contracts.ModuleRunSpec) (contracts.TranscriptModule, error) {
if f == nil {

View File

@@ -66,21 +66,20 @@ func TestUnsupportedUnknownModuleKeyFailsCleanly(t *testing.T) {
func TestRecognizedButUnimplementedModuleKeyFailsCleanly(t *testing.T) {
factory := NewFactory(Dependencies{})
for _, key := range []string{ModuleKeySpokenWord} {
t.Run(key, func(t *testing.T) {
_, err := factory.ModuleForSpec(contracts.ModuleRunSpec{ModuleKey: key, InstanceName: key})
if err == nil {
t.Fatal("expected unimplemented-module error")
}
// Force an unimplemented state for a known key to keep reason-code behavior tested.
factory.constructors[ModuleKeySpokenWord] = nil
var unimplemented *UnimplementedModuleError
if !errors.As(err, &unimplemented) {
t.Fatalf("expected UnimplementedModuleError, got %T (%v)", err, err)
}
if unimplemented.ReasonCode() != ReasonUnimplementedModule {
t.Fatalf("unexpected reason code: %q", unimplemented.ReasonCode())
}
})
_, err := factory.ModuleForSpec(contracts.ModuleRunSpec{ModuleKey: ModuleKeySpokenWord, InstanceName: ModuleKeySpokenWord})
if err == nil {
t.Fatal("expected unimplemented-module error")
}
var unimplemented *UnimplementedModuleError
if !errors.As(err, &unimplemented) {
t.Fatalf("expected UnimplementedModuleError, got %T (%v)", err, err)
}
if unimplemented.ReasonCode() != ReasonUnimplementedModule {
t.Fatalf("unexpected reason code: %q", unimplemented.ReasonCode())
}
}
@@ -117,6 +116,17 @@ func TestHomophonesIsRegisteredAndConstructibleByDefault(t *testing.T) {
}
}
func TestSpokenWordIsRegisteredAndConstructibleByDefault(t *testing.T) {
factory := NewFactory(Dependencies{})
module, err := factory.ModuleForSpec(contracts.ModuleRunSpec{ModuleKey: ModuleKeySpokenWord, InstanceName: ModuleKeySpokenWord})
if err != nil {
t.Fatalf("ModuleForSpec error: %v", err)
}
if module.Key() != ModuleKeySpokenWord {
t.Fatalf("expected spoken_word module key, got %q", module.Key())
}
}
func TestRegisterConstructorAndConstruct(t *testing.T) {
factory := NewFactory(Dependencies{})
if err := factory.RegisterConstructor(ModuleKeyGlossary, func(ctx context.Context, req ConstructRequest) (contracts.TranscriptModule, error) {

View File

@@ -0,0 +1,76 @@
package spoken_word
import (
"context"
"fmt"
"gitea.maximumdirect.net/eric/audita/internal/framework/contracts"
"gitea.maximumdirect.net/eric/audita/internal/framework/proposal_generation"
"gitea.maximumdirect.net/eric/audita/internal/framework/proposals"
"gitea.maximumdirect.net/eric/audita/internal/framework/validators"
)
type Module struct {
validators []contracts.Validator
}
func New() (*Module, error) {
spokenWordReview, err := validators.NewLLMBackedValidator("spoken_word_review", validators.LLMValidatorTypeSpokenWordReview, "")
if err != nil {
return nil, err
}
meaningReversal, err := validators.NewLLMBackedValidator("meaning_reversal_review", validators.LLMValidatorTypeMeaningReversal, "")
if err != nil {
return nil, err
}
return &Module{
validators: []contracts.Validator{
validators.NoEffectValidator{},
validators.OriginalTextPresenceValidator{},
validators.ConfidenceThresholdValidator{},
validators.ProtectedGlossaryTermValidator{},
validators.NonEmptyCorrectionValidator{},
spokenWordReview,
meaningReversal,
},
}, nil
}
func (m *Module) Key() string { return "spoken_word" }
func (m *Module) ReplacementPolicy() proposals.ReplacementPolicy {
// Python spoken_word module uses require_unique for conservative single-span replacement.
return proposals.ReplacementPolicyRequireUnique
}
func (m *Module) Validators() []contracts.Validator {
return append([]contracts.Validator(nil), m.validators...)
}
func (m *Module) Propose(ctx context.Context, req contracts.ProposalRequest) ([]proposals.CorrectionProposal, error) {
messages, err := BuildProposalMessages(req.WorkingTranscript, req.Glossary)
if err != nil {
return nil, err
}
generated, err := proposal_generation.GenerateCandidates(ctx, proposal_generation.Request{
ModuleKey: req.RunSpec.ModuleKey,
ModuleInstance: req.RunSpec.InstanceName,
ReplacementPolicy: req.RunSpec.ReplacementPolicy,
WorkingTranscript: req.WorkingTranscript,
Section: req.Section,
Glossary: req.Glossary,
Config: req.Config,
Messages: messages,
StageName: fmt.Sprintf("%s:proposal", req.RunSpec.InstanceName),
StartIndex: 0,
LLMClient: req.LLMClient,
Scheduler: req.LLMScheduler,
DiagnosticsDir: req.DiagnosticsDir,
})
if err != nil {
return nil, err
}
return generated.Corrections, nil
}

View File

@@ -0,0 +1,295 @@
package spoken_word
import (
"context"
"errors"
"os"
"path/filepath"
"strings"
"testing"
"gitea.maximumdirect.net/eric/audita/internal/core/config"
"gitea.maximumdirect.net/eric/audita/internal/core/schema"
"gitea.maximumdirect.net/eric/audita/internal/framework/contracts"
"gitea.maximumdirect.net/eric/audita/internal/framework/proposal_generation"
"gitea.maximumdirect.net/eric/audita/internal/framework/proposals"
"gitea.maximumdirect.net/eric/audita/internal/framework/validators"
)
type fakeLLMClient struct {
responses []proposal_generation.StructuredCorrectionSet
err error
calls []contracts.StructuredCompletionRequest
}
func (f *fakeLLMClient) CompleteStructured(ctx context.Context, req contracts.StructuredCompletionRequest, out any) (contracts.StructuredCompletionResponse, error) {
_ = ctx
f.calls = append(f.calls, req)
if f.err != nil {
return contracts.StructuredCompletionResponse{}, f.err
}
target, ok := out.(*proposal_generation.StructuredCorrectionSet)
if !ok {
return contracts.StructuredCompletionResponse{}, errors.New("unexpected output type")
}
if len(f.responses) == 0 {
return contracts.StructuredCompletionResponse{}, errors.New("unexpected call")
}
*target = f.responses[0]
f.responses = f.responses[1:]
return contracts.StructuredCompletionResponse{}, nil
}
type countingScheduler struct {
runs int
}
func (s *countingScheduler) Run(ctx context.Context, fn func(context.Context) error) error {
s.runs++
return fn(ctx)
}
func tinyTranscript() *schema.Transcript {
return &schema.Transcript{Segments: []schema.Segment{
{ID: 1, Speaker: "A", Start: 0, End: 1, Text: "I I think, you know, we should go.", Categories: []string{"session"}},
}}
}
func tinyGlossary() *schema.Glossary {
return &schema.Glossary{Entries: []schema.GlossaryEntry{
{Name: "Jesters", Aliases: []string{"Jester"}, Category: "faction", Summary: "Guild members", Plural: "Jesters"},
}}
}
func TestBuildProposalMessagesContainsContextAndMeaningGuardrails(t *testing.T) {
msgs, err := BuildProposalMessages(tinyTranscript(), tinyGlossary())
if err != nil {
t.Fatalf("BuildProposalMessages error: %v", err)
}
if len(msgs) != 2 {
t.Fatalf("expected 2 messages, got %d", len(msgs))
}
combined := msgs[0].Content + "\n" + msgs[1].Content
for _, want := range []string{
"Protected glossary/context:",
"Transcript section:",
`"Aliases":`,
`"Category": "faction"`,
`"Summary": "Guild members"`,
`"Plural": "Jesters"`,
"conservative spoken-word cleanup assistant",
"Preserve substantive meaning",
"Do not paraphrase, summarize, reorder ideas",
`"id": 1`,
} {
if !strings.Contains(combined, want) {
t.Fatalf("expected prompt to contain %q", want)
}
}
for _, forbidden := range []string{
"style rewriting",
"invent",
"grammar-only cleanup",
"punctuation-only cleanup",
} {
if strings.Contains(strings.ToLower(combined), forbidden) {
t.Fatalf("prompt should not invite %q", forbidden)
}
}
}
func TestSpokenWordModuleReplacementPolicy(t *testing.T) {
m, err := New()
if err != nil {
t.Fatalf("New error: %v", err)
}
if m.ReplacementPolicy() != proposals.ReplacementPolicyRequireUnique {
t.Fatalf("unexpected replacement policy: %q", m.ReplacementPolicy())
}
}
func TestSpokenWordModuleValidatorChain(t *testing.T) {
m, err := New()
if err != nil {
t.Fatalf("New error: %v", err)
}
got := make([]string, 0, len(m.Validators()))
for _, v := range m.Validators() {
got = append(got, v.Name())
}
want := []string{
"no_effect",
"original_text_presence",
"confidence_threshold",
"protected_glossary_terms",
"non_empty_correction",
"spoken_word_review",
"meaning_reversal_review",
}
if strings.Join(got, ",") != strings.Join(want, ",") {
t.Fatalf("unexpected validator chain\n got: %v\nwant: %v", got, want)
}
}
func TestSpokenWordModuleProposeMapsCorrectionsAndWritesDiagnostics(t *testing.T) {
secret := "spoken-word-secret"
client := &fakeLLMClient{
responses: []proposal_generation.StructuredCorrectionSet{
{
Corrections: []proposal_generation.StructuredCorrectionProposal{
{TargetSegmentID: 1, OriginalText: "I I think", CorrectedText: "I think", Confidence: 0.94},
},
},
},
}
scheduler := &countingScheduler{}
cfg := config.Default()
cfg.PrimaryLLM.APIKey = secret
m, err := New()
if err != nil {
t.Fatalf("New error: %v", err)
}
diagDir := t.TempDir()
out, err := m.Propose(context.Background(), contracts.ProposalRequest{
ExecutionContext: contracts.ExecutionContext{
Config: &cfg,
WorkingTranscript: tinyTranscript(),
Glossary: tinyGlossary(),
DiagnosticsDir: diagDir,
},
RunSpec: contracts.ModuleRunSpec{
ModuleKey: "spoken_word",
InstanceName: "spoken_word",
ReplacementPolicy: proposals.ReplacementPolicyRequireUnique,
},
LLMClient: client,
LLMScheduler: scheduler,
})
if err != nil {
t.Fatalf("Propose error: %v", err)
}
if scheduler.runs != 1 {
t.Fatalf("expected scheduler run count 1, got %d", scheduler.runs)
}
if len(client.calls) != 1 || client.calls[0].StageName != "spoken_word:proposal" {
t.Fatalf("expected one spoken_word:proposal call, got %+v", client.calls)
}
if len(out) != 1 || out[0].CorrectedText != "I think" {
t.Fatalf("unexpected proposals: %+v", out)
}
diagFiles, globErr := filepath.Glob(filepath.Join(diagDir, "spoken_word", "*proposal*response-payload.json"))
if globErr != nil {
t.Fatalf("glob error: %v", globErr)
}
if len(diagFiles) == 0 {
t.Fatalf("expected diagnostics response payload under %s", filepath.Join(diagDir, "spoken_word"))
}
for _, f := range diagFiles {
raw, readErr := os.ReadFile(f)
if readErr != nil {
t.Fatalf("read diag %q: %v", f, readErr)
}
if strings.Contains(string(raw), secret) {
t.Fatalf("secret leaked in diagnostics: %s", string(raw))
}
}
}
func TestSpokenWordConfidenceThresholdIsUsed(t *testing.T) {
module, err := New()
if err != nil {
t.Fatalf("New error: %v", err)
}
var thresholdValidator contracts.Validator
for _, v := range module.Validators() {
if v.Name() == "confidence_threshold" {
thresholdValidator = v
break
}
}
if thresholdValidator == nil {
t.Fatal("expected confidence_threshold validator")
}
cfg := config.Default()
cfg.Thresholds.SpokenWord = 0.95
result, err := thresholdValidator.Validate(context.Background(), validators.Request{
Config: &cfg,
ModuleKey: "spoken_word",
CandidateProposal: []proposals.EnrichedCorrectionProposal{
{
CorrectionProposal: proposals.CorrectionProposal{
TargetSegmentID: 1,
OriginalText: "I I think",
CorrectedText: "I think",
Confidence: 0.94,
},
ProposalMetadata: proposals.ProposalMetadata{ProposalIndex: 0, ModuleKey: "spoken_word", ModuleInstance: "spoken_word"},
},
},
})
if err != nil {
t.Fatalf("validate error: %v", err)
}
if len(result.Decisions) != 1 {
t.Fatalf("expected one decision, got %+v", result.Decisions)
}
if result.Decisions[0].Approved {
t.Fatalf("expected rejection under spoken_word threshold, got %+v", result.Decisions[0])
}
if result.Decisions[0].ReasonCode != validators.ReasonLowConfidence {
t.Fatalf("expected low confidence reason, got %+v", result.Decisions[0])
}
}
func TestSpokenWordProtectedGlossaryTermBehaviorRejectsUnsafeCorrection(t *testing.T) {
module, err := New()
if err != nil {
t.Fatalf("New error: %v", err)
}
var protectedValidator contracts.Validator
for _, v := range module.Validators() {
if v.Name() == "protected_glossary_terms" {
protectedValidator = v
break
}
}
if protectedValidator == nil {
t.Fatal("expected protected_glossary_terms validator")
}
working := &schema.Transcript{Segments: []schema.Segment{
{ID: 1, Speaker: "A", Start: 0, End: 1, Text: "Jesters are here."},
}}
glossary := &schema.Glossary{Entries: []schema.GlossaryEntry{
{Name: "Jesters", Category: "faction", Summary: "Faction"},
}}
result, err := protectedValidator.Validate(context.Background(), validators.Request{
WorkingTranscript: working,
Glossary: glossary,
ModuleKey: "spoken_word",
CandidateProposal: []proposals.EnrichedCorrectionProposal{
{
CorrectionProposal: proposals.CorrectionProposal{
TargetSegmentID: 1,
OriginalText: "Jesters",
CorrectedText: "gestures",
Confidence: 0.99,
},
ProposalMetadata: proposals.ProposalMetadata{ProposalIndex: 0, ModuleKey: "spoken_word", ModuleInstance: "spoken_word"},
},
},
})
if err != nil {
t.Fatalf("validate error: %v", err)
}
if len(result.Decisions) != 1 {
t.Fatalf("expected one decision, got %+v", result.Decisions)
}
if result.Decisions[0].Approved {
t.Fatalf("expected protected-term rejection, got %+v", result.Decisions[0])
}
if result.Decisions[0].ReasonCode != validators.ReasonProtectedGlossaryTerm {
t.Fatalf("expected protected glossary-term reason, got %+v", result.Decisions[0])
}
}

View File

@@ -0,0 +1,86 @@
package spoken_word
import (
"encoding/json"
"fmt"
"gitea.maximumdirect.net/eric/audita/internal/core/schema"
"gitea.maximumdirect.net/eric/audita/internal/framework/contracts"
)
type promptSegment struct {
ID int `json:"id"`
Speaker string `json:"speaker"`
Start float64 `json:"start"`
End float64 `json:"end"`
Text string `json:"text"`
Categories []string `json:"categories,omitempty"`
}
type promptTranscriptSection struct {
SectionIndex int `json:"section_index"`
Segments []promptSegment `json:"segments"`
}
// BuildProposalMessages mirrors the Python spoken_word-module prompt intent:
// conservative dysfluency cleanup with strict semantic preservation.
func BuildProposalMessages(transcript *schema.Transcript, glossary *schema.Glossary) ([]contracts.LLMMessage, error) {
glossaryJSON, err := json.MarshalIndent(glossary, "", " ")
if err != nil {
return nil, fmt.Errorf("marshal glossary prompt context: %w", err)
}
sectionPayload := promptTranscriptSection{
SectionIndex: 0,
Segments: make([]promptSegment, 0),
}
if transcript != nil {
for _, s := range transcript.Segments {
sectionPayload.Segments = append(sectionPayload.Segments, promptSegment{
ID: s.ID,
Speaker: s.Speaker,
Start: s.Start,
End: s.End,
Text: s.Text,
Categories: append([]string(nil), s.Categories...),
})
}
}
sectionJSON, err := json.MarshalIndent(sectionPayload, "", " ")
if err != nil {
return nil, fmt.Errorf("marshal transcript prompt context: %w", err)
}
system := "You are Audita, a conservative spoken-word cleanup assistant. Identify only low-risk cleanup of repeated words or short phrases, filler words, hesitation artifacts, and similar dysfluencies that commonly appear in spoken English transcripts. Preserve substantive meaning, named entities, and transcript content."
user := "Review this transcript section and return only spoken-word cleanup corrections that should be applied.\n\n" +
"Rules:\n" +
"- Approve only conservative cleanup of repeated words, repeated short phrases, filler words, hesitation artifacts, and similar spoken dysfluencies.\n" +
"- You may collapse adjacent repetition such as \"I I think\" to \"I think\" or remove filler spans such as \"you know\" or \"uh\" when local context supports that cleanup.\n" +
"- Do not collapse repeated words or short phrases when the repetition plausibly expresses urgency, excitement, insistence, or deliberate rhetorical emphasis rather than dysfluency.\n" +
"- Phrases such as \"Help! Help! Help!\", \"Stop! Stop! Stop!\", \"No! No! No!\", \"Yes! Yes! Yes!\", and \"Go! Go! Go!\" are often intentional emphasis and should usually be preserved.\n" +
"- Only collapse repetition when local context supports it as accidental spoken repetition, hesitation, or verbal restart.\n" +
"- You may include low-risk punctuation, spacing, or capitalization cleanup when it is part of removing a dysfluency, such as removing ellipses or hesitation punctuation that no longer belongs after the cleanup.\n" +
"- Do not paraphrase, summarize, reorder ideas, replace content with different wording, or make substantive semantic edits.\n" +
"- Do not change clear content words just because a different phrasing reads better.\n" +
"- Do not convert uncertain statements into certain statements.\n" +
"- Treat glossary names and aliases as protected spellings and context.\n" +
"- Do not replace, Anglicize, normalize, lowercase, or otherwise alter protected glossary names or aliases that already appear correctly in the transcript.\n" +
"- Preserve canonical glossary capitalization for protected names and aliases, even if they look unusual.\n" +
"- If a segment includes categories, treat them as additional transcript context.\n" +
"- Use the exact id from the input segment.\n" +
"- For returned corrections, original_text must be only the exact text span that needs replacement, not the full segment text unless the whole segment is the replacement span.\n" +
"- Choose an original_text span that appears exactly once in the current segment text.\n" +
"- corrected_text must be only the replacement text for that span, not the full corrected segment text unless the whole segment is the replacement span.\n" +
"- Each returned correction must contain only id, original_text, corrected_text, and confidence.\n" +
"- Do not return corrections where original_text and corrected_text are identical.\n" +
"- Do not return speaker, start, or end fields.\n" +
"- Return only changed segments; do not return entries for unchanged segments.\n" +
"- confidence must be between 0.0 and 1.0.\n" +
"- If no corrections are needed, return an empty corrections list.\n\n" +
fmt.Sprintf("Protected glossary/context:\n%s\n\nTranscript section:\n%s", string(glossaryJSON), string(sectionJSON))
return []contracts.LLMMessage{
{Role: "system", Content: system},
{Role: "user", Content: user},
}, nil
}