Complete Phase 15 spoken-word module
This commit is contained in:
@@ -36,14 +36,18 @@ Implemented today:
|
|||||||
- Production `grammar` module implementation in `internal/modules/grammar`.
|
- Production `grammar` module implementation in `internal/modules/grammar`.
|
||||||
- Production `glossary` module implementation in `internal/modules/glossary`.
|
- Production `glossary` module implementation in `internal/modules/glossary`.
|
||||||
- Production `homophones` module implementation in `internal/modules/homophones`.
|
- Production `homophones` module implementation in `internal/modules/homophones`.
|
||||||
|
- Production `spoken_word` module implementation in `internal/modules/spoken_word`.
|
||||||
- Explicit runtime support for `--modules grammar` through the production runner path.
|
- Explicit runtime support for `--modules grammar` through the production runner path.
|
||||||
- Explicit runtime support for `--modules glossary`, including repeated stages such as `--modules glossary,glossary`.
|
- Explicit runtime support for `--modules glossary`, including repeated stages such as `--modules glossary,glossary`.
|
||||||
- Explicit runtime support for `--modules homophones` through the production runner path.
|
- Explicit runtime support for `--modules homophones` through the production runner path.
|
||||||
|
- Explicit runtime support for `--modules spoken_word` through the production runner path.
|
||||||
|
|
||||||
Not implemented in CLI runtime path today:
|
Not implemented in CLI runtime path today:
|
||||||
- Real module execution pipeline for `spoken_word`.
|
|
||||||
- End-to-end transcript polishing with the full default module sequence.
|
- End-to-end transcript polishing with the full default module sequence.
|
||||||
|
|
||||||
|
Current reality:
|
||||||
|
- all production modules exist and are explicitly runnable by `--modules`, but default full-sequence integration remains a later phase.
|
||||||
|
|
||||||
Phase sequencing note:
|
Phase sequencing note:
|
||||||
- Phase 9 LLM infrastructure is complete (structured client, scheduler, effective config resolution, diagnostics primitives);
|
- Phase 9 LLM infrastructure is complete (structured client, scheduler, effective config resolution, diagnostics primitives);
|
||||||
- Phase 10 LLM-backed validator runtime integration is complete;
|
- Phase 10 LLM-backed validator runtime integration is complete;
|
||||||
@@ -51,7 +55,8 @@ Phase sequencing note:
|
|||||||
- Phase 12 grammar module implementation and explicit runtime wiring are complete;
|
- Phase 12 grammar module implementation and explicit runtime wiring are complete;
|
||||||
- Phase 13 glossary module and protected-term behavior are complete;
|
- Phase 13 glossary module and protected-term behavior are complete;
|
||||||
- Phase 14 homophones module implementation and explicit runtime wiring are complete;
|
- Phase 14 homophones module implementation and explicit runtime wiring are complete;
|
||||||
- next recommended phase is Phase 15 (spoken-word module).
|
- Phase 15 spoken-word module implementation and explicit runtime wiring are complete;
|
||||||
|
- next recommended phase is Phase 16 (default full pipeline integration).
|
||||||
|
|
||||||
## Actual Go package layout
|
## Actual Go package layout
|
||||||
|
|
||||||
@@ -122,6 +127,10 @@ internal/modules/homophones/
|
|||||||
module.go
|
module.go
|
||||||
prompt.go
|
prompt.go
|
||||||
|
|
||||||
|
internal/modules/spoken_word/
|
||||||
|
module.go
|
||||||
|
prompt.go
|
||||||
|
|
||||||
internal/framework/validators/
|
internal/framework/validators/
|
||||||
models.go
|
models.go
|
||||||
deterministic.go
|
deterministic.go
|
||||||
@@ -157,17 +166,17 @@ Current runtime flow (`internal/cli/run.go`):
|
|||||||
10. Chunk normalized transcript and compute chunk summaries.
|
10. Chunk normalized transcript and compute chunk summaries.
|
||||||
11. Write chunking summary artifact.
|
11. Write chunking summary artifact.
|
||||||
12. Execute runner modules sequentially when:
|
12. Execute runner modules sequentially when:
|
||||||
- `--modules` is explicitly provided (production grammar/glossary/homophones paths); or
|
- `--modules` is explicitly provided (production grammar/glossary/homophones/spoken_word paths); or
|
||||||
- a test/injected module factory is provided.
|
- a test/injected module factory is provided.
|
||||||
13. Output working transcript to `--output` file or stdout.
|
13. Output working transcript to `--output` file or stdout.
|
||||||
14. Build process report (`phase` currently set to `phase14-homophones-module`).
|
14. Build process report (`phase` currently set to `phase15-spoken-word-module`).
|
||||||
15. Optionally write `--report-json`; always write run-dir `report.json`.
|
15. Optionally write `--report-json`; always write run-dir `report.json`.
|
||||||
16. Apply work-dir retention.
|
16. Apply work-dir retention.
|
||||||
|
|
||||||
Important behavior details:
|
Important behavior details:
|
||||||
- Glossary is validated and is used for explicit glossary/grammar/homophones module correction paths.
|
- Glossary is validated and is used for explicit glossary/grammar/homophones/spoken_word module correction paths.
|
||||||
- Default production CLI behavior remains deterministic normalization/chunking/reporting unless modules are explicitly selected with `--modules`.
|
- Default production CLI behavior remains deterministic normalization/chunking/reporting unless modules are explicitly selected with `--modules`.
|
||||||
- Explicit `--modules grammar`, `--modules glossary`, and `--modules homophones` run production module paths with LLM-backed proposal generation and validator-chain execution.
|
- Explicit `--modules grammar`, `--modules glossary`, `--modules homophones`, and `--modules spoken_word` run production module paths with LLM-backed proposal generation and validator-chain execution.
|
||||||
- Default runs (without explicit module selection) do not perform LLM calls.
|
- Default runs (without explicit module selection) do not perform LLM calls.
|
||||||
- Success path is generally quiet on stderr.
|
- Success path is generally quiet on stderr.
|
||||||
- Source IDs are preserved into a canonical transcript before normalization; normalization then reassigns output IDs sequentially from `1`.
|
- Source IDs are preserved into a canonical transcript before normalization; normalization then reassigns output IDs sequentially from `1`.
|
||||||
@@ -216,7 +225,7 @@ Implemented config surfaces include:
|
|||||||
- work-dir and retention mode
|
- work-dir and retention mode
|
||||||
|
|
||||||
Current caveat:
|
Current caveat:
|
||||||
- LLM/module-related settings are active for explicit grammar/glossary/homophones runs; the default non-explicit path remains deterministic.
|
- LLM/module-related settings are active for explicit grammar/glossary/homophones/spoken_word runs; the default non-explicit path remains deterministic.
|
||||||
|
|
||||||
## Implemented structured LLM infrastructure
|
## Implemented structured LLM infrastructure
|
||||||
`internal/framework/contracts` now defines a typed structured-completion contract:
|
`internal/framework/contracts` now defines a typed structured-completion contract:
|
||||||
@@ -233,7 +242,7 @@ Current caveat:
|
|||||||
|
|
||||||
Current runtime boundary:
|
Current runtime boundary:
|
||||||
- the default CLI runtime path (without explicit module selection) still does not instantiate the full production module sequence.
|
- the default CLI runtime path (without explicit module selection) still does not instantiate the full production module sequence.
|
||||||
- LLM calls are exercised in production when `--modules grammar`, `--modules glossary`, or `--modules homophones` is explicitly requested and in tests when fake/injected clients are used.
|
- LLM calls are exercised in production when `--modules grammar`, `--modules glossary`, `--modules homophones`, or `--modules spoken_word` is explicitly requested and in tests when fake/injected clients are used.
|
||||||
|
|
||||||
`internal/framework/llm` also provides:
|
`internal/framework/llm` also provides:
|
||||||
- a bounded `Scheduler` for controlled concurrent LLM calls with reliable permit release;
|
- a bounded `Scheduler` for controlled concurrent LLM calls with reliable permit release;
|
||||||
@@ -275,7 +284,7 @@ Current behavior details:
|
|||||||
|
|
||||||
`internal/framework/contracts` provides interfaces and run-spec metadata scaffolding, including deterministic repeated module instance naming (`ResolveModuleRunSpecs`).
|
`internal/framework/contracts` provides interfaces and run-spec metadata scaffolding, including deterministic repeated module instance naming (`ResolveModuleRunSpecs`).
|
||||||
|
|
||||||
These primitives are wired into the production runner and report model. The grammar and glossary modules are implemented; other production modules remain pending.
|
These primitives are wired into the production runner and report model. The grammar, glossary, homophones, and spoken_word modules are implemented.
|
||||||
|
|
||||||
## Implemented validator runtime infrastructure
|
## Implemented validator runtime infrastructure
|
||||||
`internal/framework/validators` provides deterministic validator infrastructure:
|
`internal/framework/validators` provides deterministic validator infrastructure:
|
||||||
@@ -340,7 +349,7 @@ This helper only produces candidate proposals; validator-chain execution and pro
|
|||||||
- diagnostics directory context
|
- diagnostics directory context
|
||||||
- returns explicit errors for unknown keys (`unsupported_module`) and recognized-but-unimplemented keys (`unimplemented_module`).
|
- returns explicit errors for unknown keys (`unsupported_module`) and recognized-but-unimplemented keys (`unimplemented_module`).
|
||||||
|
|
||||||
The `grammar`, `glossary`, and `homophones` module keys are now registered and constructible. `spoken_word` remains recognized-but-unimplemented.
|
The `grammar`, `glossary`, `homophones`, and `spoken_word` module keys are now registered and constructible.
|
||||||
|
|
||||||
## Implemented grammar production module
|
## Implemented grammar production module
|
||||||
`internal/modules/grammar` now provides the first production module:
|
`internal/modules/grammar` now provides the first production module:
|
||||||
@@ -387,6 +396,19 @@ This vocabulary is used by deterministic validators for both glossary-stage and
|
|||||||
- protected-term guardrails for non-glossary modules remain active and are exercised through the homophones path;
|
- protected-term guardrails for non-glossary modules remain active and are exercised through the homophones path;
|
||||||
- module-level reporting and diagnostics capture through existing runner/reporting paths.
|
- module-level reporting and diagnostics capture through existing runner/reporting paths.
|
||||||
|
|
||||||
|
## Implemented spoken_word production module
|
||||||
|
`internal/modules/spoken_word` now provides the fourth production module:
|
||||||
|
- prompt builder aligned to Python spoken_word-module intent, constrained to conservative dysfluency cleanup;
|
||||||
|
- strong prompt guardrails preserving meaning/intent/voice/named entities/domain terms and substantive content;
|
||||||
|
- explicit guardrails against summarization, style rewriting, grammar-only cleanup, punctuation-only cleanup, invention, and meaning-changing rewrites;
|
||||||
|
- proposal generation through `internal/framework/proposal_generation` and `contracts.StructuredLLMClient`;
|
||||||
|
- scheduler-aware proposal calls through existing `contracts.LLMScheduler` hooks;
|
||||||
|
- replacement policy `require_unique` (matching Python spoken_word behavior);
|
||||||
|
- validator chain integration using existing deterministic + LLM-backed validators, including strong semantic guardrails (`spoken_word_review`, `meaning_reversal_review`);
|
||||||
|
- spoken_word confidence threshold enforcement through existing validator/config infrastructure;
|
||||||
|
- protected-term guardrails for non-glossary modules remain active and are exercised through the spoken_word path;
|
||||||
|
- module-level reporting and diagnostics capture through existing runner/reporting paths.
|
||||||
|
|
||||||
## Reports and diagnostics (implemented)
|
## Reports and diagnostics (implemented)
|
||||||
Current per-run artifacts include:
|
Current per-run artifacts include:
|
||||||
- `source-transcript.json`
|
- `source-transcript.json`
|
||||||
@@ -426,7 +448,7 @@ Retention modes implemented in `ApplyRetention`:
|
|||||||
|
|
||||||
Current runtime note:
|
Current runtime note:
|
||||||
- default non-explicit runs usually have no module-level skipped corrections, so `auto` commonly removes clean successful run directories.
|
- default non-explicit runs usually have no module-level skipped corrections, so `auto` commonly removes clean successful run directories.
|
||||||
- explicit grammar/glossary/homophones runs can produce validator rejections and application skips, which are reflected in reports and retention input.
|
- explicit grammar/glossary/homophones/spoken_word runs can produce validator rejections and application skips, which are reflected in reports and retention input.
|
||||||
|
|
||||||
Intentionally deferred to module/LLM phases:
|
Intentionally deferred to module/LLM phases:
|
||||||
- real domain proposal prompts and production module implementations remain tied to later module phases.
|
- real domain proposal prompts and production module implementations remain tied to later module phases.
|
||||||
@@ -450,14 +472,15 @@ Implemented tests currently cover:
|
|||||||
- production grammar module prompt constraints, proposal mapping, validator-chain behavior, confidence-threshold enforcement, diagnostics redaction, and explicit CLI/runtime integration (`internal/modules/grammar/*_test.go`, `internal/cli/run_test.go`, `internal/framework/runner/*_test.go`)
|
- production grammar module prompt constraints, proposal mapping, validator-chain behavior, confidence-threshold enforcement, diagnostics redaction, and explicit CLI/runtime integration (`internal/modules/grammar/*_test.go`, `internal/cli/run_test.go`, `internal/framework/runner/*_test.go`)
|
||||||
- production glossary module prompt constraints, proposal mapping, validator-chain behavior, confidence-threshold enforcement, diagnostics redaction, repeated-stage behavior, and explicit CLI/runtime integration (`internal/modules/glossary/*_test.go`, `internal/cli/run_test.go`, `internal/framework/runner/*_test.go`)
|
- production glossary module prompt constraints, proposal mapping, validator-chain behavior, confidence-threshold enforcement, diagnostics redaction, repeated-stage behavior, and explicit CLI/runtime integration (`internal/modules/glossary/*_test.go`, `internal/cli/run_test.go`, `internal/framework/runner/*_test.go`)
|
||||||
- production homophones module prompt constraints, proposal mapping, validator-chain behavior, confidence-threshold enforcement, diagnostics redaction, protected-term behavior, and explicit CLI/runtime integration (`internal/modules/homophones/*_test.go`, `internal/cli/run_test.go`, `internal/framework/runner/*_test.go`)
|
- production homophones module prompt constraints, proposal mapping, validator-chain behavior, confidence-threshold enforcement, diagnostics redaction, protected-term behavior, and explicit CLI/runtime integration (`internal/modules/homophones/*_test.go`, `internal/cli/run_test.go`, `internal/framework/runner/*_test.go`)
|
||||||
|
- production spoken_word module prompt constraints, proposal mapping, validator-chain behavior, semantic guardrail behavior, confidence-threshold enforcement, diagnostics redaction, protected-term behavior, and explicit CLI/runtime integration (`internal/modules/spoken_word/*_test.go`, `internal/cli/run_test.go`, `internal/framework/runner/*_test.go`)
|
||||||
- glossary-derived protected-term extraction and stable behavior (`internal/framework/validators/protected_terms_test.go`)
|
- glossary-derived protected-term extraction and stable behavior (`internal/framework/validators/protected_terms_test.go`)
|
||||||
|
|
||||||
Not covered yet (because not implemented): production `spoken_word` module plus full default-sequence transcript-polishing runtime behavior.
|
Not covered yet (because not implemented): full default-sequence transcript-polishing runtime behavior as a single default path.
|
||||||
|
|
||||||
## Intended final architecture (not yet implemented)
|
## Intended final architecture (not yet implemented)
|
||||||
The intended end-state still matches the rewrite plan:
|
The intended end-state still matches the rewrite plan:
|
||||||
- sequential module pipeline over a mutable working transcript
|
- sequential module pipeline over a mutable working transcript
|
||||||
- real module implementation (`spoken_word`)
|
- default full module-sequence integration in the standard runtime path
|
||||||
- structured LLM proposal generation
|
- structured LLM proposal generation
|
||||||
- deterministic and LLM validators
|
- deterministic and LLM validators
|
||||||
- validator cardinality enforcement in pipeline execution
|
- validator cardinality enforcement in pipeline execution
|
||||||
|
|||||||
@@ -68,6 +68,10 @@ Implemented:
|
|||||||
- Production homophones module package with Python-aligned prompt intent and guardrails.
|
- Production homophones module package with Python-aligned prompt intent and guardrails.
|
||||||
- Explicit `--modules homophones` runtime path through runner, shared proposal generation, validators, application, reporting, and diagnostics.
|
- Explicit `--modules homophones` runtime path through runner, shared proposal generation, validators, application, reporting, and diagnostics.
|
||||||
- Focused multi-module runtime tests for already-implemented interoperability (for example `glossary,homophones`) without claiming full default-pipeline completion.
|
- Focused multi-module runtime tests for already-implemented interoperability (for example `glossary,homophones`) without claiming full default-pipeline completion.
|
||||||
|
- Production spoken_word module package with Python-aligned prompt intent and guardrails.
|
||||||
|
- Explicit `--modules spoken_word` runtime path through runner, shared proposal generation, validators, application, reporting, and diagnostics.
|
||||||
|
- Focused multi-module runtime tests for already-implemented interoperability (for example `spoken_word,grammar`) without claiming full default-pipeline completion.
|
||||||
|
- All production modules now exist (`glossary`, `homophones`, `spoken_word`, `grammar`), but default full-sequence integration remains Phase 16 work.
|
||||||
- Broad deterministic and CLI/subprocess test coverage for implemented phases through `go test ./...`.
|
- Broad deterministic and CLI/subprocess test coverage for implemented phases through `go test ./...`.
|
||||||
- Internal typed structured LLM contract (`StructuredLLMClient.CompleteStructured(ctx, req, out)`).
|
- Internal typed structured LLM contract (`StructuredLLMClient.CompleteStructured(ctx, req, out)`).
|
||||||
- `internal/framework/llm` instructor-go-backed adapter with:
|
- `internal/framework/llm` instructor-go-backed adapter with:
|
||||||
@@ -83,8 +87,6 @@ Implemented:
|
|||||||
- Generic JSON diagnostics primitives for LLM interactions (request metadata, request payload, response payload, optional error payload) with secret redaction.
|
- Generic JSON diagnostics primitives for LLM interactions (request metadata, request payload, response payload, optional error payload) with secret redaction.
|
||||||
|
|
||||||
Not yet implemented in runtime pipeline:
|
Not yet implemented in runtime pipeline:
|
||||||
- Real correction module for `spoken_word`.
|
|
||||||
- Domain proposal prompts for remaining real module.
|
|
||||||
- End-to-end transcript polishing behavior with the full default module sequence.
|
- End-to-end transcript polishing behavior with the full default module sequence.
|
||||||
|
|
||||||
## Completed phases
|
## Completed phases
|
||||||
@@ -223,7 +225,7 @@ Not implemented in Phase 8 (by design):
|
|||||||
|
|
||||||
## Remaining work plan
|
## Remaining work plan
|
||||||
|
|
||||||
Next recommended phase: **Phase 15 (spoken-word module)**.
|
Next recommended phase: **Phase 16 (default full pipeline integration)**.
|
||||||
|
|
||||||
## Phase 9: Structured LLM client and scheduler infrastructure
|
## Phase 9: Structured LLM client and scheduler infrastructure
|
||||||
|
|
||||||
@@ -418,41 +420,29 @@ Not implemented in Phase 14 (by design):
|
|||||||
|
|
||||||
## Phase 15: Spoken-word module
|
## Phase 15: Spoken-word module
|
||||||
|
|
||||||
### Purpose
|
Completed.
|
||||||
|
|
||||||
Implement conservative dysfluency cleanup while preserving substantive meaning.
|
Implemented:
|
||||||
|
- Production spoken_word module package in `internal/modules/spoken_word`.
|
||||||
|
- Spoken_word prompt builder aligned to Python intent and constrained to conservative dysfluency cleanup.
|
||||||
|
- Prompt context using glossary/protected-term information (names, aliases, plurals where present) to avoid damaging known domain terms.
|
||||||
|
- Strong prompt guardrails preserving meaning, intent, speaker voice, named entities, game/domain terms, and substantive content.
|
||||||
|
- Explicit prompt guardrails against summarization, style rewriting, grammar-only cleanup, punctuation-only cleanup, invention, event reordering, and certainty inflation.
|
||||||
|
- Spoken_word proposal generation through shared `internal/framework/proposal_generation` using `contracts.StructuredLLMClient`.
|
||||||
|
- Scheduler-aware spoken_word proposal generation through existing scheduler hooks.
|
||||||
|
- Spoken_word replacement policy `require_unique` (matching Python behavior).
|
||||||
|
- Spoken_word validator chain using existing deterministic and LLM-backed validators.
|
||||||
|
- Strong semantic guardrails in runtime validator chain through existing LLM-backed validators (`spoken_word_review`, `meaning_reversal_review`).
|
||||||
|
- Spoken-word confidence threshold enforcement through existing config + confidence-threshold validator behavior.
|
||||||
|
- Protected-term guardrails remaining active for spoken_word via existing deterministic validators.
|
||||||
|
- Explicit runtime support for `--modules spoken_word` through normalization, chunking, runner, proposal generation, validation, application, and reporting.
|
||||||
|
- Prompt/response diagnostics artifacts for spoken_word proposal + validator interactions with secret redaction.
|
||||||
|
- Module-level reports for spoken_word including generated proposals, validator decisions/rejections, applied changes, and application skips.
|
||||||
|
- CLI/runtime fake-client tests for approved cleanup, validator rejection, meaning-changing rejection, application skips, diagnostics, protected-term rejection behavior, failure/error.log behavior, and report outputs (`--report-json` and run-dir `report.json`).
|
||||||
|
- Focused interoperability tests for already-implemented module combinations (for example `spoken_word,grammar`) to verify working-transcript handoff and guardrails without claiming full default-sequence parity.
|
||||||
|
|
||||||
### Scope
|
Not implemented in Phase 15 (by design):
|
||||||
|
- Full default module sequence execution as a feature-complete claim.
|
||||||
Implement:
|
|
||||||
- `spoken_word` module package.
|
|
||||||
- Spoken-word prompt builder ported from Python.
|
|
||||||
- Spoken-word structured response model.
|
|
||||||
- Spoken-word replacement policy.
|
|
||||||
- Spoken-word confidence threshold handling.
|
|
||||||
- Spoken-word validator chain.
|
|
||||||
- Strong semantic guardrails using LLM-backed validators.
|
|
||||||
- Prompt/response diagnostics.
|
|
||||||
- CLI support for `--modules spoken_word`.
|
|
||||||
- Fake LLM tests.
|
|
||||||
- Tests for rejection of meaning-changing cleanup.
|
|
||||||
|
|
||||||
Do not implement:
|
|
||||||
- Prompt redesign beyond faithful porting.
|
|
||||||
- New stylistic rewriting behavior not present in the Python implementation.
|
|
||||||
|
|
||||||
### Expected behavior at end of phase
|
|
||||||
|
|
||||||
Running `audita process ... --modules spoken_word` should perform real conservative dysfluency cleanup, with guardrails against semantic changes.
|
|
||||||
|
|
||||||
### Definition of done
|
|
||||||
|
|
||||||
- Spoken-word module runs in the production runner.
|
|
||||||
- Spoken-word proposals are generated through structured LLM calls.
|
|
||||||
- Semantic validators reject meaning-changing proposals.
|
|
||||||
- Applied/skipped spoken-word changes appear in reports.
|
|
||||||
- Prompt/response diagnostics are written.
|
|
||||||
- `go test ./...` passes without requiring external LLM credentials.
|
|
||||||
|
|
||||||
## Phase 16: Default full pipeline integration
|
## Phase 16: Default full pipeline integration
|
||||||
|
|
||||||
|
|||||||
@@ -460,7 +460,7 @@ func extractErrorPhase(err error) (phase string, message string) {
|
|||||||
|
|
||||||
func buildProcessReport(status string, inv processInvocation, runDir *diagnostics.RunDirectory, startedAt, completedAt time.Time, errorMessage string, errorPhase string, normalizationSummary *normalization.NormalizationSummary, chunkingSummary *chunking.Summary, runOutput *runner.RunOutput) reporting.ProcessReport {
|
func buildProcessReport(status string, inv processInvocation, runDir *diagnostics.RunDirectory, startedAt, completedAt time.Time, errorMessage string, errorPhase string, normalizationSummary *normalization.NormalizationSummary, chunkingSummary *chunking.Summary, runOutput *runner.RunOutput) reporting.ProcessReport {
|
||||||
report := reporting.ProcessReport{
|
report := reporting.ProcessReport{
|
||||||
Phase: "phase14-homophones-module",
|
Phase: "phase15-spoken-word-module",
|
||||||
Status: status,
|
Status: status,
|
||||||
Operation: "process",
|
Operation: "process",
|
||||||
TranscriptPath: inv.TranscriptPath,
|
TranscriptPath: inv.TranscriptPath,
|
||||||
|
|||||||
@@ -616,8 +616,8 @@ func TestRunProcessReportJSONIncludesChunkingSummary(t *testing.T) {
|
|||||||
if report.Chunking.MaxSectionTokens == 0 {
|
if report.Chunking.MaxSectionTokens == 0 {
|
||||||
t.Errorf("expected max_section_tokens in report")
|
t.Errorf("expected max_section_tokens in report")
|
||||||
}
|
}
|
||||||
if report.Phase != "phase14-homophones-module" {
|
if report.Phase != "phase15-spoken-word-module" {
|
||||||
t.Errorf("expected phase 'phase14-homophones-module', got %q", report.Phase)
|
t.Errorf("expected phase 'phase15-spoken-word-module', got %q", report.Phase)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -909,7 +909,7 @@ func TestRunProcessInjectedFactoryFailureWritesFailedReport(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
func TestRunProcessProductionRegistryUnimplementedModuleFailsCleanly(t *testing.T) {
|
func TestRunProcessProductionRegistryUnsupportedModuleFailsCleanly(t *testing.T) {
|
||||||
cfg := modules.Dependencies{}
|
cfg := modules.Dependencies{}
|
||||||
processModuleFactory = modules.NewFactory(cfg)
|
processModuleFactory = modules.NewFactory(cfg)
|
||||||
t.Cleanup(func() { processModuleFactory = nil })
|
t.Cleanup(func() { processModuleFactory = nil })
|
||||||
@@ -924,7 +924,7 @@ func TestRunProcessProductionRegistryUnimplementedModuleFailsCleanly(t *testing.
|
|||||||
exitCode := Run([]string{
|
exitCode := Run([]string{
|
||||||
"process", transcriptPath,
|
"process", transcriptPath,
|
||||||
"--glossary", fixturePath("tiny_glossary.yaml"),
|
"--glossary", fixturePath("tiny_glossary.yaml"),
|
||||||
"--modules", "spoken_word",
|
"--modules", "made_up",
|
||||||
"--work-dir", workDir,
|
"--work-dir", workDir,
|
||||||
"--work-dir-retention", "always",
|
"--work-dir-retention", "always",
|
||||||
"--report-json", reportPath,
|
"--report-json", reportPath,
|
||||||
@@ -938,8 +938,8 @@ func TestRunProcessProductionRegistryUnimplementedModuleFailsCleanly(t *testing.
|
|||||||
if !strings.Contains(stderr.String(), "runner_execution") {
|
if !strings.Contains(stderr.String(), "runner_execution") {
|
||||||
t.Fatalf("expected runner_execution failure on stderr, got %q", stderr.String())
|
t.Fatalf("expected runner_execution failure on stderr, got %q", stderr.String())
|
||||||
}
|
}
|
||||||
if !strings.Contains(stderr.String(), "recognized but not implemented") {
|
if !strings.Contains(stderr.String(), "unsupported module key") {
|
||||||
t.Fatalf("expected explicit unimplemented module message, got %q", stderr.String())
|
t.Fatalf("expected explicit unsupported module message, got %q", stderr.String())
|
||||||
}
|
}
|
||||||
|
|
||||||
report := readProcessReport(t, reportPath)
|
report := readProcessReport(t, reportPath)
|
||||||
@@ -949,13 +949,13 @@ func TestRunProcessProductionRegistryUnimplementedModuleFailsCleanly(t *testing.
|
|||||||
if report.ErrorPhase != "runner_execution" {
|
if report.ErrorPhase != "runner_execution" {
|
||||||
t.Fatalf("expected runner_execution phase, got %q", report.ErrorPhase)
|
t.Fatalf("expected runner_execution phase, got %q", report.ErrorPhase)
|
||||||
}
|
}
|
||||||
if !strings.Contains(report.ErrorMessage, "recognized but not implemented") {
|
if !strings.Contains(report.ErrorMessage, "unsupported module key") {
|
||||||
t.Fatalf("expected report error message to mention unimplemented module, got %q", report.ErrorMessage)
|
t.Fatalf("expected report error message to mention unsupported module, got %q", report.ErrorMessage)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
func TestRunProcessExplicitUnimplementedModulesFailClearly(t *testing.T) {
|
func TestRunProcessExplicitUnsupportedModulesFailClearly(t *testing.T) {
|
||||||
for _, moduleKey := range []string{"spoken_word"} {
|
for _, moduleKey := range []string{"made_up"} {
|
||||||
t.Run(moduleKey, func(t *testing.T) {
|
t.Run(moduleKey, func(t *testing.T) {
|
||||||
var stdout, stderr bytes.Buffer
|
var stdout, stderr bytes.Buffer
|
||||||
transcriptPath := writeFile(t, "transcript.json", `[
|
transcriptPath := writeFile(t, "transcript.json", `[
|
||||||
@@ -972,8 +972,8 @@ func TestRunProcessExplicitUnimplementedModulesFailClearly(t *testing.T) {
|
|||||||
if stdout.Len() != 0 {
|
if stdout.Len() != 0 {
|
||||||
t.Fatalf("expected empty stdout on failure, got %q", stdout.String())
|
t.Fatalf("expected empty stdout on failure, got %q", stdout.String())
|
||||||
}
|
}
|
||||||
if !strings.Contains(stderr.String(), "recognized but not implemented") {
|
if !strings.Contains(stderr.String(), "unsupported module key") {
|
||||||
t.Fatalf("expected unimplemented message, got %q", stderr.String())
|
t.Fatalf("expected unsupported-module message, got %q", stderr.String())
|
||||||
}
|
}
|
||||||
})
|
})
|
||||||
}
|
}
|
||||||
@@ -1790,6 +1790,385 @@ func TestRunProcessExplicitGlossaryThenHomophonesSeesWorkingTranscriptChanges(t
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func TestRunProcessExplicitSpokenWordAppliesCleanupAndReportsDiagnostics(t *testing.T) {
|
||||||
|
secret := "phase15-secret"
|
||||||
|
proposalClient := &fakeStructuredLLMClient{
|
||||||
|
proposalResponses: []proposal_generation.StructuredCorrectionSet{
|
||||||
|
{
|
||||||
|
Corrections: []proposal_generation.StructuredCorrectionProposal{
|
||||||
|
{TargetSegmentID: 1, OriginalText: "I I think", CorrectedText: "I think", Confidence: 0.95},
|
||||||
|
},
|
||||||
|
},
|
||||||
|
},
|
||||||
|
}
|
||||||
|
validationClient := &fakeStructuredLLMClient{
|
||||||
|
validationResponses: []validators.LLMValidationResponse{
|
||||||
|
{Validations: []validators.LLMValidationDecision{{CorrectionIndex: 0, Approved: true, Confidence: 0.99, Reason: "spoken cleanup acceptable"}}},
|
||||||
|
{Validations: []validators.LLMValidationDecision{{CorrectionIndex: 0, Approved: true, Confidence: 0.99, Reason: secret}}},
|
||||||
|
},
|
||||||
|
}
|
||||||
|
processProposalLLMClient = proposalClient
|
||||||
|
processValidationLLMClient = validationClient
|
||||||
|
t.Setenv("AUDITA_LLM_API_KEY", secret)
|
||||||
|
t.Setenv("AUDITA_VALIDATION_LLM_API_KEY", secret)
|
||||||
|
t.Cleanup(func() {
|
||||||
|
processProposalLLMClient = nil
|
||||||
|
processValidationLLMClient = nil
|
||||||
|
})
|
||||||
|
|
||||||
|
var stdout, stderr bytes.Buffer
|
||||||
|
workDir := t.TempDir()
|
||||||
|
reportPath := filepath.Join(t.TempDir(), "report.json")
|
||||||
|
outputPath := filepath.Join(t.TempDir(), "out.json")
|
||||||
|
transcriptPath := writeFile(t, "transcript.json", `[
|
||||||
|
{"id":1,"speaker":"Alice","start":0.0,"end":1.0,"text":"I I think we should go."}
|
||||||
|
]`)
|
||||||
|
|
||||||
|
exitCode := Run([]string{
|
||||||
|
"process", transcriptPath,
|
||||||
|
"--glossary", fixturePath("tiny_glossary.yaml"),
|
||||||
|
"--modules", "spoken_word",
|
||||||
|
"--output", outputPath,
|
||||||
|
"--report-json", reportPath,
|
||||||
|
"--work-dir", workDir,
|
||||||
|
"--work-dir-retention", "always",
|
||||||
|
}, &stdout, &stderr)
|
||||||
|
if exitCode != 0 {
|
||||||
|
t.Fatalf("expected success, got %d stderr=%q", exitCode, stderr.String())
|
||||||
|
}
|
||||||
|
if stdout.Len() != 0 {
|
||||||
|
t.Fatalf("expected empty stdout with --output, got %q", stdout.String())
|
||||||
|
}
|
||||||
|
|
||||||
|
parsed, err := schema.ParseTranscriptJSON(readFile(t, outputPath))
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("parse output: %v", err)
|
||||||
|
}
|
||||||
|
if parsed.Segments[0].Text != "I think we should go." {
|
||||||
|
t.Fatalf("expected spoken_word cleanup applied, got %q", parsed.Segments[0].Text)
|
||||||
|
}
|
||||||
|
|
||||||
|
report := readProcessReport(t, reportPath)
|
||||||
|
if len(report.ModuleResults) != 1 || report.ModuleResults[0].ModuleKey != "spoken_word" {
|
||||||
|
t.Fatalf("expected one spoken_word module result, got %+v", report.ModuleResults)
|
||||||
|
}
|
||||||
|
if len(report.ModuleResults[0].AppliedChanges) != 1 {
|
||||||
|
t.Fatalf("expected one applied spoken_word change, got %+v", report.ModuleResults[0].AppliedChanges)
|
||||||
|
}
|
||||||
|
if len(report.ModuleResults[0].ValidatorDecisions) == 0 {
|
||||||
|
t.Fatalf("expected validator decisions in report")
|
||||||
|
}
|
||||||
|
|
||||||
|
runDir := onlyRunDir(t, workDir)
|
||||||
|
runReport := readProcessReport(t, filepath.Join(runDir, "report.json"))
|
||||||
|
if len(runReport.ModuleResults) != 1 || runReport.ModuleResults[0].ModuleKey != "spoken_word" {
|
||||||
|
t.Fatalf("expected spoken_word module results in run-dir report, got %+v", runReport.ModuleResults)
|
||||||
|
}
|
||||||
|
|
||||||
|
diagFiles, globErr := filepath.Glob(filepath.Join(runDir, "spoken_word", "*response-payload.json"))
|
||||||
|
if globErr != nil {
|
||||||
|
t.Fatalf("glob diagnostics: %v", globErr)
|
||||||
|
}
|
||||||
|
if len(diagFiles) == 0 {
|
||||||
|
t.Fatalf("expected spoken_word diagnostics payload files in %s", filepath.Join(runDir, "spoken_word"))
|
||||||
|
}
|
||||||
|
for _, f := range diagFiles {
|
||||||
|
raw := string(readFile(t, f))
|
||||||
|
if strings.Contains(raw, secret) {
|
||||||
|
t.Fatalf("secret leaked in diagnostics %q: %s", f, raw)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestRunProcessExplicitSpokenWordRejectedAndApplicationSkipAreDistinct(t *testing.T) {
|
||||||
|
processProposalLLMClient = &fakeStructuredLLMClient{
|
||||||
|
proposalResponses: []proposal_generation.StructuredCorrectionSet{
|
||||||
|
{
|
||||||
|
Corrections: []proposal_generation.StructuredCorrectionProposal{
|
||||||
|
{TargetSegmentID: 1, OriginalText: "hall", CorrectedText: "temple", Confidence: 0.99},
|
||||||
|
{TargetSegmentID: 1, OriginalText: "I I think in the hall.", CorrectedText: "I think in the temple.", Confidence: 0.99},
|
||||||
|
{TargetSegmentID: 1, OriginalText: "I I think", CorrectedText: "I suppose", Confidence: 0.99},
|
||||||
|
},
|
||||||
|
},
|
||||||
|
},
|
||||||
|
}
|
||||||
|
processValidationLLMClient = &fakeStructuredLLMClient{
|
||||||
|
validationResponses: []validators.LLMValidationResponse{
|
||||||
|
{
|
||||||
|
Validations: []validators.LLMValidationDecision{
|
||||||
|
{CorrectionIndex: 0, Approved: true, Confidence: 0.9, Reason: "ok"},
|
||||||
|
{CorrectionIndex: 1, Approved: true, Confidence: 0.9, Reason: "ok"},
|
||||||
|
{CorrectionIndex: 2, Approved: false, Confidence: 0.9, Reason: "reject"},
|
||||||
|
},
|
||||||
|
},
|
||||||
|
{
|
||||||
|
Validations: []validators.LLMValidationDecision{
|
||||||
|
{CorrectionIndex: 0, Approved: true, Confidence: 0.9, Reason: "ok"},
|
||||||
|
{CorrectionIndex: 1, Approved: true, Confidence: 0.9, Reason: "ok"},
|
||||||
|
},
|
||||||
|
},
|
||||||
|
},
|
||||||
|
}
|
||||||
|
t.Cleanup(func() {
|
||||||
|
processProposalLLMClient = nil
|
||||||
|
processValidationLLMClient = nil
|
||||||
|
})
|
||||||
|
|
||||||
|
var stdout, stderr bytes.Buffer
|
||||||
|
reportPath := filepath.Join(t.TempDir(), "report.json")
|
||||||
|
outputPath := filepath.Join(t.TempDir(), "out.json")
|
||||||
|
transcriptPath := writeFile(t, "transcript.json", `[
|
||||||
|
{"id":1,"speaker":"Alice","start":0.0,"end":1.0,"text":"I I think in the hall."}
|
||||||
|
]`)
|
||||||
|
exitCode := Run([]string{
|
||||||
|
"process", transcriptPath,
|
||||||
|
"--glossary", fixturePath("tiny_glossary.yaml"),
|
||||||
|
"--modules", "spoken_word",
|
||||||
|
"--output", outputPath,
|
||||||
|
"--report-json", reportPath,
|
||||||
|
}, &stdout, &stderr)
|
||||||
|
if exitCode != 0 {
|
||||||
|
t.Fatalf("expected success, got %d stderr=%q", exitCode, stderr.String())
|
||||||
|
}
|
||||||
|
|
||||||
|
report := readProcessReport(t, reportPath)
|
||||||
|
if len(report.ModuleResults) != 1 {
|
||||||
|
t.Fatalf("expected one module result")
|
||||||
|
}
|
||||||
|
module := report.ModuleResults[0]
|
||||||
|
if len(module.ValidatorRejected) != 1 {
|
||||||
|
t.Fatalf("expected one validator rejection, got %+v", module.ValidatorRejected)
|
||||||
|
}
|
||||||
|
if len(module.SkippedChanges) != 1 {
|
||||||
|
t.Fatalf("expected one application skip, got %+v", module.SkippedChanges)
|
||||||
|
}
|
||||||
|
if module.ValidatorRejected[0].ReasonCode == string(module.SkippedChanges[0].SkipReason) {
|
||||||
|
t.Fatalf("validator rejection and application skip should remain distinct")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestRunProcessExplicitSpokenWordMeaningChangingCleanupRejected(t *testing.T) {
|
||||||
|
processProposalLLMClient = &fakeStructuredLLMClient{
|
||||||
|
proposalResponses: []proposal_generation.StructuredCorrectionSet{
|
||||||
|
{
|
||||||
|
Corrections: []proposal_generation.StructuredCorrectionProposal{
|
||||||
|
{TargetSegmentID: 1, OriginalText: "I I think", CorrectedText: "I know", Confidence: 0.99},
|
||||||
|
},
|
||||||
|
},
|
||||||
|
},
|
||||||
|
}
|
||||||
|
processValidationLLMClient = &fakeStructuredLLMClient{
|
||||||
|
validationResponses: []validators.LLMValidationResponse{
|
||||||
|
{Validations: []validators.LLMValidationDecision{{CorrectionIndex: 0, Approved: false, Confidence: 0.99, Reason: "meaning changed"}}},
|
||||||
|
},
|
||||||
|
}
|
||||||
|
t.Cleanup(func() {
|
||||||
|
processProposalLLMClient = nil
|
||||||
|
processValidationLLMClient = nil
|
||||||
|
})
|
||||||
|
|
||||||
|
var stdout, stderr bytes.Buffer
|
||||||
|
reportPath := filepath.Join(t.TempDir(), "report.json")
|
||||||
|
outputPath := filepath.Join(t.TempDir(), "out.json")
|
||||||
|
transcriptPath := writeFile(t, "transcript.json", `[
|
||||||
|
{"id":1,"speaker":"Alice","start":0.0,"end":1.0,"text":"I I think we should go."}
|
||||||
|
]`)
|
||||||
|
exitCode := Run([]string{
|
||||||
|
"process", transcriptPath,
|
||||||
|
"--glossary", fixturePath("tiny_glossary.yaml"),
|
||||||
|
"--modules", "spoken_word",
|
||||||
|
"--output", outputPath,
|
||||||
|
"--report-json", reportPath,
|
||||||
|
}, &stdout, &stderr)
|
||||||
|
if exitCode != 0 {
|
||||||
|
t.Fatalf("expected success, got %d stderr=%q", exitCode, stderr.String())
|
||||||
|
}
|
||||||
|
|
||||||
|
parsed, err := schema.ParseTranscriptJSON(readFile(t, outputPath))
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("parse output: %v", err)
|
||||||
|
}
|
||||||
|
if parsed.Segments[0].Text != "I I think we should go." {
|
||||||
|
t.Fatalf("expected meaning-changing cleanup to be rejected, got %q", parsed.Segments[0].Text)
|
||||||
|
}
|
||||||
|
|
||||||
|
report := readProcessReport(t, reportPath)
|
||||||
|
if len(report.ModuleResults) != 1 {
|
||||||
|
t.Fatalf("expected one module result, got %+v", report.ModuleResults)
|
||||||
|
}
|
||||||
|
module := report.ModuleResults[0]
|
||||||
|
if len(module.AppliedChanges) != 0 {
|
||||||
|
t.Fatalf("expected no applied changes, got %+v", module.AppliedChanges)
|
||||||
|
}
|
||||||
|
if len(module.ValidatorRejected) != 1 {
|
||||||
|
t.Fatalf("expected one semantic guardrail rejection, got %+v", module.ValidatorRejected)
|
||||||
|
}
|
||||||
|
if module.ValidatorRejected[0].ValidatorName != "spoken_word_review" {
|
||||||
|
t.Fatalf("expected spoken_word_review rejection, got %+v", module.ValidatorRejected[0])
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestRunProcessExplicitSpokenWordProtectedGlossaryTermRejected(t *testing.T) {
|
||||||
|
processProposalLLMClient = &fakeStructuredLLMClient{
|
||||||
|
proposalResponses: []proposal_generation.StructuredCorrectionSet{
|
||||||
|
{
|
||||||
|
Corrections: []proposal_generation.StructuredCorrectionProposal{
|
||||||
|
{TargetSegmentID: 1, OriginalText: "Audita", CorrectedText: "audita", Confidence: 0.99},
|
||||||
|
},
|
||||||
|
},
|
||||||
|
},
|
||||||
|
}
|
||||||
|
t.Cleanup(func() {
|
||||||
|
processProposalLLMClient = nil
|
||||||
|
processValidationLLMClient = nil
|
||||||
|
})
|
||||||
|
|
||||||
|
var stdout, stderr bytes.Buffer
|
||||||
|
reportPath := filepath.Join(t.TempDir(), "report.json")
|
||||||
|
outputPath := filepath.Join(t.TempDir(), "out.json")
|
||||||
|
transcriptPath := writeFile(t, "transcript.json", `[
|
||||||
|
{"id":1,"speaker":"Alice","start":0.0,"end":1.0,"text":"Audita held the line."}
|
||||||
|
]`)
|
||||||
|
exitCode := Run([]string{
|
||||||
|
"process", transcriptPath,
|
||||||
|
"--glossary", fixturePath("tiny_glossary.yaml"),
|
||||||
|
"--modules", "spoken_word",
|
||||||
|
"--output", outputPath,
|
||||||
|
"--report-json", reportPath,
|
||||||
|
}, &stdout, &stderr)
|
||||||
|
if exitCode != 0 {
|
||||||
|
t.Fatalf("expected success, got %d stderr=%q", exitCode, stderr.String())
|
||||||
|
}
|
||||||
|
|
||||||
|
parsed, err := schema.ParseTranscriptJSON(readFile(t, outputPath))
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("parse output: %v", err)
|
||||||
|
}
|
||||||
|
if parsed.Segments[0].Text != "Audita held the line." {
|
||||||
|
t.Fatalf("expected protected glossary term to remain unchanged, got %q", parsed.Segments[0].Text)
|
||||||
|
}
|
||||||
|
|
||||||
|
report := readProcessReport(t, reportPath)
|
||||||
|
if len(report.ModuleResults) != 1 {
|
||||||
|
t.Fatalf("expected one module result, got %+v", report.ModuleResults)
|
||||||
|
}
|
||||||
|
module := report.ModuleResults[0]
|
||||||
|
if len(module.AppliedChanges) != 0 {
|
||||||
|
t.Fatalf("expected no applied changes, got %+v", module.AppliedChanges)
|
||||||
|
}
|
||||||
|
if len(module.ValidatorRejected) != 1 {
|
||||||
|
t.Fatalf("expected one validator rejection, got %+v", module.ValidatorRejected)
|
||||||
|
}
|
||||||
|
if module.ValidatorRejected[0].ReasonCode != validators.ReasonProtectedGlossaryTerm {
|
||||||
|
t.Fatalf("expected protected glossary term rejection, got %+v", module.ValidatorRejected[0])
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestRunProcessExplicitSpokenWordMalformedLLMOutputFailsWithErrorLog(t *testing.T) {
|
||||||
|
processProposalLLMClient = &fakeStructuredLLMClient{err: errors.New("malformed structured output")}
|
||||||
|
t.Cleanup(func() { processProposalLLMClient = nil })
|
||||||
|
|
||||||
|
var stdout, stderr bytes.Buffer
|
||||||
|
workDir := t.TempDir()
|
||||||
|
reportPath := filepath.Join(t.TempDir(), "report.json")
|
||||||
|
transcriptPath := writeFile(t, "transcript.json", `[
|
||||||
|
{"id":1,"speaker":"Alice","start":0.0,"end":1.0,"text":"hello"}
|
||||||
|
]`)
|
||||||
|
|
||||||
|
exitCode := Run([]string{
|
||||||
|
"process", transcriptPath,
|
||||||
|
"--glossary", fixturePath("tiny_glossary.yaml"),
|
||||||
|
"--modules", "spoken_word",
|
||||||
|
"--work-dir", workDir,
|
||||||
|
"--work-dir-retention", "always",
|
||||||
|
"--report-json", reportPath,
|
||||||
|
}, &stdout, &stderr)
|
||||||
|
if exitCode == 0 {
|
||||||
|
t.Fatal("expected failure")
|
||||||
|
}
|
||||||
|
if stdout.Len() != 0 {
|
||||||
|
t.Fatalf("expected empty stdout on failure, got %q", stdout.String())
|
||||||
|
}
|
||||||
|
if !strings.Contains(stderr.String(), "runner_execution") {
|
||||||
|
t.Fatalf("expected runner_execution error, got %q", stderr.String())
|
||||||
|
}
|
||||||
|
runDir := onlyRunDir(t, workDir)
|
||||||
|
if _, err := os.Stat(filepath.Join(runDir, "error.log")); err != nil {
|
||||||
|
t.Fatalf("expected error.log on failed spoken_word run: %v", err)
|
||||||
|
}
|
||||||
|
report := readProcessReport(t, reportPath)
|
||||||
|
if report.Status != "failed" || report.ErrorPhase != "runner_execution" {
|
||||||
|
t.Fatalf("expected failed runner_execution report, got %+v", report)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestRunProcessExplicitSpokenWordThenGrammarSeesWorkingTranscriptChanges(t *testing.T) {
|
||||||
|
proposalClient := &fakeStructuredLLMClient{
|
||||||
|
proposalResponses: []proposal_generation.StructuredCorrectionSet{
|
||||||
|
{
|
||||||
|
Corrections: []proposal_generation.StructuredCorrectionProposal{
|
||||||
|
{TargetSegmentID: 1, OriginalText: "i i think", CorrectedText: "i think", Confidence: 0.99},
|
||||||
|
},
|
||||||
|
},
|
||||||
|
{
|
||||||
|
Corrections: []proposal_generation.StructuredCorrectionProposal{
|
||||||
|
{TargetSegmentID: 1, OriginalText: "i think we should go.", CorrectedText: "I think we should go.", Confidence: 0.99},
|
||||||
|
},
|
||||||
|
},
|
||||||
|
},
|
||||||
|
}
|
||||||
|
validationClient := &fakeStructuredLLMClient{
|
||||||
|
validationResponses: []validators.LLMValidationResponse{
|
||||||
|
{Validations: []validators.LLMValidationDecision{{CorrectionIndex: 0, Approved: true, Confidence: 0.99, Reason: "ok"}}},
|
||||||
|
{Validations: []validators.LLMValidationDecision{{CorrectionIndex: 0, Approved: true, Confidence: 0.99, Reason: "ok"}}},
|
||||||
|
{Validations: []validators.LLMValidationDecision{{CorrectionIndex: 0, Approved: true, Confidence: 0.99, Reason: "ok"}}},
|
||||||
|
{Validations: []validators.LLMValidationDecision{{CorrectionIndex: 0, Approved: true, Confidence: 0.99, Reason: "ok"}}},
|
||||||
|
},
|
||||||
|
}
|
||||||
|
processProposalLLMClient = proposalClient
|
||||||
|
processValidationLLMClient = validationClient
|
||||||
|
t.Cleanup(func() {
|
||||||
|
processProposalLLMClient = nil
|
||||||
|
processValidationLLMClient = nil
|
||||||
|
})
|
||||||
|
|
||||||
|
var stdout, stderr bytes.Buffer
|
||||||
|
reportPath := filepath.Join(t.TempDir(), "report.json")
|
||||||
|
outputPath := filepath.Join(t.TempDir(), "out.json")
|
||||||
|
transcriptPath := writeFile(t, "transcript.json", `[
|
||||||
|
{"id":1,"speaker":"Alice","start":0.0,"end":1.0,"text":"i i think we should go."}
|
||||||
|
]`)
|
||||||
|
exitCode := Run([]string{
|
||||||
|
"process", transcriptPath,
|
||||||
|
"--glossary", fixturePath("tiny_glossary.yaml"),
|
||||||
|
"--modules", "spoken_word,grammar",
|
||||||
|
"--output", outputPath,
|
||||||
|
"--report-json", reportPath,
|
||||||
|
}, &stdout, &stderr)
|
||||||
|
if exitCode != 0 {
|
||||||
|
t.Fatalf("expected success, got %d stderr=%q", exitCode, stderr.String())
|
||||||
|
}
|
||||||
|
|
||||||
|
parsed, err := schema.ParseTranscriptJSON(readFile(t, outputPath))
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("parse output: %v", err)
|
||||||
|
}
|
||||||
|
if parsed.Segments[0].Text != "I think we should go." {
|
||||||
|
t.Fatalf("expected grammar stage to see spoken_word output, got %q", parsed.Segments[0].Text)
|
||||||
|
}
|
||||||
|
|
||||||
|
report := readProcessReport(t, reportPath)
|
||||||
|
if len(report.ModuleResults) != 2 {
|
||||||
|
t.Fatalf("expected two module results, got %+v", report.ModuleResults)
|
||||||
|
}
|
||||||
|
if report.ModuleResults[0].ModuleKey != "spoken_word" || report.ModuleResults[1].ModuleKey != "grammar" {
|
||||||
|
t.Fatalf("expected spoken_word then grammar results, got %+v", report.ModuleResults)
|
||||||
|
}
|
||||||
|
if len(report.ModuleResults[0].AppliedChanges) != 1 || len(report.ModuleResults[1].AppliedChanges) != 1 {
|
||||||
|
t.Fatalf("expected one applied change per module, got %+v", report.ModuleResults)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
func TestRunProcessChunkingSummaryArtifactWritten(t *testing.T) {
|
func TestRunProcessChunkingSummaryArtifactWritten(t *testing.T) {
|
||||||
var stdout bytes.Buffer
|
var stdout bytes.Buffer
|
||||||
var stderr bytes.Buffer
|
var stderr bytes.Buffer
|
||||||
|
|||||||
@@ -11,7 +11,7 @@ import (
|
|||||||
func TestProcessReportModuleResultsJSONSuccessAndSkipped(t *testing.T) {
|
func TestProcessReportModuleResultsJSONSuccessAndSkipped(t *testing.T) {
|
||||||
now := time.Now().UTC()
|
now := time.Now().UTC()
|
||||||
report := ProcessReport{
|
report := ProcessReport{
|
||||||
Phase: "phase14-homophones-module",
|
Phase: "phase15-spoken-word-module",
|
||||||
Status: "success",
|
Status: "success",
|
||||||
ModuleResults: []ModuleReport{
|
ModuleResults: []ModuleReport{
|
||||||
{
|
{
|
||||||
@@ -80,7 +80,7 @@ func TestProcessReportModuleResultsJSONSuccessAndSkipped(t *testing.T) {
|
|||||||
func TestProcessReportModuleResultsJSONFailedModule(t *testing.T) {
|
func TestProcessReportModuleResultsJSONFailedModule(t *testing.T) {
|
||||||
now := time.Now().UTC()
|
now := time.Now().UTC()
|
||||||
report := ProcessReport{
|
report := ProcessReport{
|
||||||
Phase: "phase14-homophones-module",
|
Phase: "phase15-spoken-word-module",
|
||||||
Status: "failed",
|
Status: "failed",
|
||||||
ModuleResults: []ModuleReport{
|
ModuleResults: []ModuleReport{
|
||||||
{
|
{
|
||||||
|
|||||||
@@ -11,6 +11,7 @@ import (
|
|||||||
glossarymodule "gitea.maximumdirect.net/eric/audita/internal/modules/glossary"
|
glossarymodule "gitea.maximumdirect.net/eric/audita/internal/modules/glossary"
|
||||||
grammarmodule "gitea.maximumdirect.net/eric/audita/internal/modules/grammar"
|
grammarmodule "gitea.maximumdirect.net/eric/audita/internal/modules/grammar"
|
||||||
homophonesmodule "gitea.maximumdirect.net/eric/audita/internal/modules/homophones"
|
homophonesmodule "gitea.maximumdirect.net/eric/audita/internal/modules/homophones"
|
||||||
|
spokenwordmodule "gitea.maximumdirect.net/eric/audita/internal/modules/spoken_word"
|
||||||
)
|
)
|
||||||
|
|
||||||
const (
|
const (
|
||||||
@@ -74,6 +75,7 @@ func NewFactory(deps Dependencies) *Factory {
|
|||||||
}
|
}
|
||||||
_ = factory.RegisterConstructor(ModuleKeyGlossary, constructGlossaryModule)
|
_ = factory.RegisterConstructor(ModuleKeyGlossary, constructGlossaryModule)
|
||||||
_ = factory.RegisterConstructor(ModuleKeyHomophones, constructHomophonesModule)
|
_ = factory.RegisterConstructor(ModuleKeyHomophones, constructHomophonesModule)
|
||||||
|
_ = factory.RegisterConstructor(ModuleKeySpokenWord, constructSpokenWordModule)
|
||||||
_ = factory.RegisterConstructor(ModuleKeyGrammar, constructGrammarModule)
|
_ = factory.RegisterConstructor(ModuleKeyGrammar, constructGrammarModule)
|
||||||
return factory
|
return factory
|
||||||
}
|
}
|
||||||
@@ -112,6 +114,12 @@ func constructHomophonesModule(ctx context.Context, req ConstructRequest) (contr
|
|||||||
return homophonesmodule.New()
|
return homophonesmodule.New()
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func constructSpokenWordModule(ctx context.Context, req ConstructRequest) (contracts.TranscriptModule, error) {
|
||||||
|
_ = ctx
|
||||||
|
_ = req
|
||||||
|
return spokenwordmodule.New()
|
||||||
|
}
|
||||||
|
|
||||||
// ModuleForSpec resolves one configured run spec into a module instance.
|
// ModuleForSpec resolves one configured run spec into a module instance.
|
||||||
func (f *Factory) ModuleForSpec(spec contracts.ModuleRunSpec) (contracts.TranscriptModule, error) {
|
func (f *Factory) ModuleForSpec(spec contracts.ModuleRunSpec) (contracts.TranscriptModule, error) {
|
||||||
if f == nil {
|
if f == nil {
|
||||||
|
|||||||
@@ -66,9 +66,10 @@ func TestUnsupportedUnknownModuleKeyFailsCleanly(t *testing.T) {
|
|||||||
|
|
||||||
func TestRecognizedButUnimplementedModuleKeyFailsCleanly(t *testing.T) {
|
func TestRecognizedButUnimplementedModuleKeyFailsCleanly(t *testing.T) {
|
||||||
factory := NewFactory(Dependencies{})
|
factory := NewFactory(Dependencies{})
|
||||||
for _, key := range []string{ModuleKeySpokenWord} {
|
// Force an unimplemented state for a known key to keep reason-code behavior tested.
|
||||||
t.Run(key, func(t *testing.T) {
|
factory.constructors[ModuleKeySpokenWord] = nil
|
||||||
_, err := factory.ModuleForSpec(contracts.ModuleRunSpec{ModuleKey: key, InstanceName: key})
|
|
||||||
|
_, err := factory.ModuleForSpec(contracts.ModuleRunSpec{ModuleKey: ModuleKeySpokenWord, InstanceName: ModuleKeySpokenWord})
|
||||||
if err == nil {
|
if err == nil {
|
||||||
t.Fatal("expected unimplemented-module error")
|
t.Fatal("expected unimplemented-module error")
|
||||||
}
|
}
|
||||||
@@ -80,8 +81,6 @@ func TestRecognizedButUnimplementedModuleKeyFailsCleanly(t *testing.T) {
|
|||||||
if unimplemented.ReasonCode() != ReasonUnimplementedModule {
|
if unimplemented.ReasonCode() != ReasonUnimplementedModule {
|
||||||
t.Fatalf("unexpected reason code: %q", unimplemented.ReasonCode())
|
t.Fatalf("unexpected reason code: %q", unimplemented.ReasonCode())
|
||||||
}
|
}
|
||||||
})
|
|
||||||
}
|
|
||||||
}
|
}
|
||||||
|
|
||||||
func TestGrammarIsRegisteredAndConstructibleByDefault(t *testing.T) {
|
func TestGrammarIsRegisteredAndConstructibleByDefault(t *testing.T) {
|
||||||
@@ -117,6 +116,17 @@ func TestHomophonesIsRegisteredAndConstructibleByDefault(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func TestSpokenWordIsRegisteredAndConstructibleByDefault(t *testing.T) {
|
||||||
|
factory := NewFactory(Dependencies{})
|
||||||
|
module, err := factory.ModuleForSpec(contracts.ModuleRunSpec{ModuleKey: ModuleKeySpokenWord, InstanceName: ModuleKeySpokenWord})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("ModuleForSpec error: %v", err)
|
||||||
|
}
|
||||||
|
if module.Key() != ModuleKeySpokenWord {
|
||||||
|
t.Fatalf("expected spoken_word module key, got %q", module.Key())
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
func TestRegisterConstructorAndConstruct(t *testing.T) {
|
func TestRegisterConstructorAndConstruct(t *testing.T) {
|
||||||
factory := NewFactory(Dependencies{})
|
factory := NewFactory(Dependencies{})
|
||||||
if err := factory.RegisterConstructor(ModuleKeyGlossary, func(ctx context.Context, req ConstructRequest) (contracts.TranscriptModule, error) {
|
if err := factory.RegisterConstructor(ModuleKeyGlossary, func(ctx context.Context, req ConstructRequest) (contracts.TranscriptModule, error) {
|
||||||
|
|||||||
76
internal/modules/spoken_word/module.go
Normal file
76
internal/modules/spoken_word/module.go
Normal file
@@ -0,0 +1,76 @@
|
|||||||
|
package spoken_word
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"fmt"
|
||||||
|
|
||||||
|
"gitea.maximumdirect.net/eric/audita/internal/framework/contracts"
|
||||||
|
"gitea.maximumdirect.net/eric/audita/internal/framework/proposal_generation"
|
||||||
|
"gitea.maximumdirect.net/eric/audita/internal/framework/proposals"
|
||||||
|
"gitea.maximumdirect.net/eric/audita/internal/framework/validators"
|
||||||
|
)
|
||||||
|
|
||||||
|
type Module struct {
|
||||||
|
validators []contracts.Validator
|
||||||
|
}
|
||||||
|
|
||||||
|
func New() (*Module, error) {
|
||||||
|
spokenWordReview, err := validators.NewLLMBackedValidator("spoken_word_review", validators.LLMValidatorTypeSpokenWordReview, "")
|
||||||
|
if err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
meaningReversal, err := validators.NewLLMBackedValidator("meaning_reversal_review", validators.LLMValidatorTypeMeaningReversal, "")
|
||||||
|
if err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
|
||||||
|
return &Module{
|
||||||
|
validators: []contracts.Validator{
|
||||||
|
validators.NoEffectValidator{},
|
||||||
|
validators.OriginalTextPresenceValidator{},
|
||||||
|
validators.ConfidenceThresholdValidator{},
|
||||||
|
validators.ProtectedGlossaryTermValidator{},
|
||||||
|
validators.NonEmptyCorrectionValidator{},
|
||||||
|
spokenWordReview,
|
||||||
|
meaningReversal,
|
||||||
|
},
|
||||||
|
}, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func (m *Module) Key() string { return "spoken_word" }
|
||||||
|
|
||||||
|
func (m *Module) ReplacementPolicy() proposals.ReplacementPolicy {
|
||||||
|
// Python spoken_word module uses require_unique for conservative single-span replacement.
|
||||||
|
return proposals.ReplacementPolicyRequireUnique
|
||||||
|
}
|
||||||
|
|
||||||
|
func (m *Module) Validators() []contracts.Validator {
|
||||||
|
return append([]contracts.Validator(nil), m.validators...)
|
||||||
|
}
|
||||||
|
|
||||||
|
func (m *Module) Propose(ctx context.Context, req contracts.ProposalRequest) ([]proposals.CorrectionProposal, error) {
|
||||||
|
messages, err := BuildProposalMessages(req.WorkingTranscript, req.Glossary)
|
||||||
|
if err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
|
||||||
|
generated, err := proposal_generation.GenerateCandidates(ctx, proposal_generation.Request{
|
||||||
|
ModuleKey: req.RunSpec.ModuleKey,
|
||||||
|
ModuleInstance: req.RunSpec.InstanceName,
|
||||||
|
ReplacementPolicy: req.RunSpec.ReplacementPolicy,
|
||||||
|
WorkingTranscript: req.WorkingTranscript,
|
||||||
|
Section: req.Section,
|
||||||
|
Glossary: req.Glossary,
|
||||||
|
Config: req.Config,
|
||||||
|
Messages: messages,
|
||||||
|
StageName: fmt.Sprintf("%s:proposal", req.RunSpec.InstanceName),
|
||||||
|
StartIndex: 0,
|
||||||
|
LLMClient: req.LLMClient,
|
||||||
|
Scheduler: req.LLMScheduler,
|
||||||
|
DiagnosticsDir: req.DiagnosticsDir,
|
||||||
|
})
|
||||||
|
if err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
return generated.Corrections, nil
|
||||||
|
}
|
||||||
295
internal/modules/spoken_word/module_test.go
Normal file
295
internal/modules/spoken_word/module_test.go
Normal file
@@ -0,0 +1,295 @@
|
|||||||
|
package spoken_word
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"errors"
|
||||||
|
"os"
|
||||||
|
"path/filepath"
|
||||||
|
"strings"
|
||||||
|
"testing"
|
||||||
|
|
||||||
|
"gitea.maximumdirect.net/eric/audita/internal/core/config"
|
||||||
|
"gitea.maximumdirect.net/eric/audita/internal/core/schema"
|
||||||
|
"gitea.maximumdirect.net/eric/audita/internal/framework/contracts"
|
||||||
|
"gitea.maximumdirect.net/eric/audita/internal/framework/proposal_generation"
|
||||||
|
"gitea.maximumdirect.net/eric/audita/internal/framework/proposals"
|
||||||
|
"gitea.maximumdirect.net/eric/audita/internal/framework/validators"
|
||||||
|
)
|
||||||
|
|
||||||
|
type fakeLLMClient struct {
|
||||||
|
responses []proposal_generation.StructuredCorrectionSet
|
||||||
|
err error
|
||||||
|
calls []contracts.StructuredCompletionRequest
|
||||||
|
}
|
||||||
|
|
||||||
|
func (f *fakeLLMClient) CompleteStructured(ctx context.Context, req contracts.StructuredCompletionRequest, out any) (contracts.StructuredCompletionResponse, error) {
|
||||||
|
_ = ctx
|
||||||
|
f.calls = append(f.calls, req)
|
||||||
|
if f.err != nil {
|
||||||
|
return contracts.StructuredCompletionResponse{}, f.err
|
||||||
|
}
|
||||||
|
target, ok := out.(*proposal_generation.StructuredCorrectionSet)
|
||||||
|
if !ok {
|
||||||
|
return contracts.StructuredCompletionResponse{}, errors.New("unexpected output type")
|
||||||
|
}
|
||||||
|
if len(f.responses) == 0 {
|
||||||
|
return contracts.StructuredCompletionResponse{}, errors.New("unexpected call")
|
||||||
|
}
|
||||||
|
*target = f.responses[0]
|
||||||
|
f.responses = f.responses[1:]
|
||||||
|
return contracts.StructuredCompletionResponse{}, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
type countingScheduler struct {
|
||||||
|
runs int
|
||||||
|
}
|
||||||
|
|
||||||
|
func (s *countingScheduler) Run(ctx context.Context, fn func(context.Context) error) error {
|
||||||
|
s.runs++
|
||||||
|
return fn(ctx)
|
||||||
|
}
|
||||||
|
|
||||||
|
func tinyTranscript() *schema.Transcript {
|
||||||
|
return &schema.Transcript{Segments: []schema.Segment{
|
||||||
|
{ID: 1, Speaker: "A", Start: 0, End: 1, Text: "I I think, you know, we should go.", Categories: []string{"session"}},
|
||||||
|
}}
|
||||||
|
}
|
||||||
|
|
||||||
|
func tinyGlossary() *schema.Glossary {
|
||||||
|
return &schema.Glossary{Entries: []schema.GlossaryEntry{
|
||||||
|
{Name: "Jesters", Aliases: []string{"Jester"}, Category: "faction", Summary: "Guild members", Plural: "Jesters"},
|
||||||
|
}}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestBuildProposalMessagesContainsContextAndMeaningGuardrails(t *testing.T) {
|
||||||
|
msgs, err := BuildProposalMessages(tinyTranscript(), tinyGlossary())
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("BuildProposalMessages error: %v", err)
|
||||||
|
}
|
||||||
|
if len(msgs) != 2 {
|
||||||
|
t.Fatalf("expected 2 messages, got %d", len(msgs))
|
||||||
|
}
|
||||||
|
combined := msgs[0].Content + "\n" + msgs[1].Content
|
||||||
|
for _, want := range []string{
|
||||||
|
"Protected glossary/context:",
|
||||||
|
"Transcript section:",
|
||||||
|
`"Aliases":`,
|
||||||
|
`"Category": "faction"`,
|
||||||
|
`"Summary": "Guild members"`,
|
||||||
|
`"Plural": "Jesters"`,
|
||||||
|
"conservative spoken-word cleanup assistant",
|
||||||
|
"Preserve substantive meaning",
|
||||||
|
"Do not paraphrase, summarize, reorder ideas",
|
||||||
|
`"id": 1`,
|
||||||
|
} {
|
||||||
|
if !strings.Contains(combined, want) {
|
||||||
|
t.Fatalf("expected prompt to contain %q", want)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
for _, forbidden := range []string{
|
||||||
|
"style rewriting",
|
||||||
|
"invent",
|
||||||
|
"grammar-only cleanup",
|
||||||
|
"punctuation-only cleanup",
|
||||||
|
} {
|
||||||
|
if strings.Contains(strings.ToLower(combined), forbidden) {
|
||||||
|
t.Fatalf("prompt should not invite %q", forbidden)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestSpokenWordModuleReplacementPolicy(t *testing.T) {
|
||||||
|
m, err := New()
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("New error: %v", err)
|
||||||
|
}
|
||||||
|
if m.ReplacementPolicy() != proposals.ReplacementPolicyRequireUnique {
|
||||||
|
t.Fatalf("unexpected replacement policy: %q", m.ReplacementPolicy())
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestSpokenWordModuleValidatorChain(t *testing.T) {
|
||||||
|
m, err := New()
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("New error: %v", err)
|
||||||
|
}
|
||||||
|
got := make([]string, 0, len(m.Validators()))
|
||||||
|
for _, v := range m.Validators() {
|
||||||
|
got = append(got, v.Name())
|
||||||
|
}
|
||||||
|
want := []string{
|
||||||
|
"no_effect",
|
||||||
|
"original_text_presence",
|
||||||
|
"confidence_threshold",
|
||||||
|
"protected_glossary_terms",
|
||||||
|
"non_empty_correction",
|
||||||
|
"spoken_word_review",
|
||||||
|
"meaning_reversal_review",
|
||||||
|
}
|
||||||
|
if strings.Join(got, ",") != strings.Join(want, ",") {
|
||||||
|
t.Fatalf("unexpected validator chain\n got: %v\nwant: %v", got, want)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestSpokenWordModuleProposeMapsCorrectionsAndWritesDiagnostics(t *testing.T) {
|
||||||
|
secret := "spoken-word-secret"
|
||||||
|
client := &fakeLLMClient{
|
||||||
|
responses: []proposal_generation.StructuredCorrectionSet{
|
||||||
|
{
|
||||||
|
Corrections: []proposal_generation.StructuredCorrectionProposal{
|
||||||
|
{TargetSegmentID: 1, OriginalText: "I I think", CorrectedText: "I think", Confidence: 0.94},
|
||||||
|
},
|
||||||
|
},
|
||||||
|
},
|
||||||
|
}
|
||||||
|
scheduler := &countingScheduler{}
|
||||||
|
cfg := config.Default()
|
||||||
|
cfg.PrimaryLLM.APIKey = secret
|
||||||
|
m, err := New()
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("New error: %v", err)
|
||||||
|
}
|
||||||
|
diagDir := t.TempDir()
|
||||||
|
out, err := m.Propose(context.Background(), contracts.ProposalRequest{
|
||||||
|
ExecutionContext: contracts.ExecutionContext{
|
||||||
|
Config: &cfg,
|
||||||
|
WorkingTranscript: tinyTranscript(),
|
||||||
|
Glossary: tinyGlossary(),
|
||||||
|
DiagnosticsDir: diagDir,
|
||||||
|
},
|
||||||
|
RunSpec: contracts.ModuleRunSpec{
|
||||||
|
ModuleKey: "spoken_word",
|
||||||
|
InstanceName: "spoken_word",
|
||||||
|
ReplacementPolicy: proposals.ReplacementPolicyRequireUnique,
|
||||||
|
},
|
||||||
|
LLMClient: client,
|
||||||
|
LLMScheduler: scheduler,
|
||||||
|
})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("Propose error: %v", err)
|
||||||
|
}
|
||||||
|
if scheduler.runs != 1 {
|
||||||
|
t.Fatalf("expected scheduler run count 1, got %d", scheduler.runs)
|
||||||
|
}
|
||||||
|
if len(client.calls) != 1 || client.calls[0].StageName != "spoken_word:proposal" {
|
||||||
|
t.Fatalf("expected one spoken_word:proposal call, got %+v", client.calls)
|
||||||
|
}
|
||||||
|
if len(out) != 1 || out[0].CorrectedText != "I think" {
|
||||||
|
t.Fatalf("unexpected proposals: %+v", out)
|
||||||
|
}
|
||||||
|
diagFiles, globErr := filepath.Glob(filepath.Join(diagDir, "spoken_word", "*proposal*response-payload.json"))
|
||||||
|
if globErr != nil {
|
||||||
|
t.Fatalf("glob error: %v", globErr)
|
||||||
|
}
|
||||||
|
if len(diagFiles) == 0 {
|
||||||
|
t.Fatalf("expected diagnostics response payload under %s", filepath.Join(diagDir, "spoken_word"))
|
||||||
|
}
|
||||||
|
for _, f := range diagFiles {
|
||||||
|
raw, readErr := os.ReadFile(f)
|
||||||
|
if readErr != nil {
|
||||||
|
t.Fatalf("read diag %q: %v", f, readErr)
|
||||||
|
}
|
||||||
|
if strings.Contains(string(raw), secret) {
|
||||||
|
t.Fatalf("secret leaked in diagnostics: %s", string(raw))
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestSpokenWordConfidenceThresholdIsUsed(t *testing.T) {
|
||||||
|
module, err := New()
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("New error: %v", err)
|
||||||
|
}
|
||||||
|
var thresholdValidator contracts.Validator
|
||||||
|
for _, v := range module.Validators() {
|
||||||
|
if v.Name() == "confidence_threshold" {
|
||||||
|
thresholdValidator = v
|
||||||
|
break
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if thresholdValidator == nil {
|
||||||
|
t.Fatal("expected confidence_threshold validator")
|
||||||
|
}
|
||||||
|
|
||||||
|
cfg := config.Default()
|
||||||
|
cfg.Thresholds.SpokenWord = 0.95
|
||||||
|
result, err := thresholdValidator.Validate(context.Background(), validators.Request{
|
||||||
|
Config: &cfg,
|
||||||
|
ModuleKey: "spoken_word",
|
||||||
|
CandidateProposal: []proposals.EnrichedCorrectionProposal{
|
||||||
|
{
|
||||||
|
CorrectionProposal: proposals.CorrectionProposal{
|
||||||
|
TargetSegmentID: 1,
|
||||||
|
OriginalText: "I I think",
|
||||||
|
CorrectedText: "I think",
|
||||||
|
Confidence: 0.94,
|
||||||
|
},
|
||||||
|
ProposalMetadata: proposals.ProposalMetadata{ProposalIndex: 0, ModuleKey: "spoken_word", ModuleInstance: "spoken_word"},
|
||||||
|
},
|
||||||
|
},
|
||||||
|
})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("validate error: %v", err)
|
||||||
|
}
|
||||||
|
if len(result.Decisions) != 1 {
|
||||||
|
t.Fatalf("expected one decision, got %+v", result.Decisions)
|
||||||
|
}
|
||||||
|
if result.Decisions[0].Approved {
|
||||||
|
t.Fatalf("expected rejection under spoken_word threshold, got %+v", result.Decisions[0])
|
||||||
|
}
|
||||||
|
if result.Decisions[0].ReasonCode != validators.ReasonLowConfidence {
|
||||||
|
t.Fatalf("expected low confidence reason, got %+v", result.Decisions[0])
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestSpokenWordProtectedGlossaryTermBehaviorRejectsUnsafeCorrection(t *testing.T) {
|
||||||
|
module, err := New()
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("New error: %v", err)
|
||||||
|
}
|
||||||
|
var protectedValidator contracts.Validator
|
||||||
|
for _, v := range module.Validators() {
|
||||||
|
if v.Name() == "protected_glossary_terms" {
|
||||||
|
protectedValidator = v
|
||||||
|
break
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if protectedValidator == nil {
|
||||||
|
t.Fatal("expected protected_glossary_terms validator")
|
||||||
|
}
|
||||||
|
|
||||||
|
working := &schema.Transcript{Segments: []schema.Segment{
|
||||||
|
{ID: 1, Speaker: "A", Start: 0, End: 1, Text: "Jesters are here."},
|
||||||
|
}}
|
||||||
|
glossary := &schema.Glossary{Entries: []schema.GlossaryEntry{
|
||||||
|
{Name: "Jesters", Category: "faction", Summary: "Faction"},
|
||||||
|
}}
|
||||||
|
result, err := protectedValidator.Validate(context.Background(), validators.Request{
|
||||||
|
WorkingTranscript: working,
|
||||||
|
Glossary: glossary,
|
||||||
|
ModuleKey: "spoken_word",
|
||||||
|
CandidateProposal: []proposals.EnrichedCorrectionProposal{
|
||||||
|
{
|
||||||
|
CorrectionProposal: proposals.CorrectionProposal{
|
||||||
|
TargetSegmentID: 1,
|
||||||
|
OriginalText: "Jesters",
|
||||||
|
CorrectedText: "gestures",
|
||||||
|
Confidence: 0.99,
|
||||||
|
},
|
||||||
|
ProposalMetadata: proposals.ProposalMetadata{ProposalIndex: 0, ModuleKey: "spoken_word", ModuleInstance: "spoken_word"},
|
||||||
|
},
|
||||||
|
},
|
||||||
|
})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("validate error: %v", err)
|
||||||
|
}
|
||||||
|
if len(result.Decisions) != 1 {
|
||||||
|
t.Fatalf("expected one decision, got %+v", result.Decisions)
|
||||||
|
}
|
||||||
|
if result.Decisions[0].Approved {
|
||||||
|
t.Fatalf("expected protected-term rejection, got %+v", result.Decisions[0])
|
||||||
|
}
|
||||||
|
if result.Decisions[0].ReasonCode != validators.ReasonProtectedGlossaryTerm {
|
||||||
|
t.Fatalf("expected protected glossary-term reason, got %+v", result.Decisions[0])
|
||||||
|
}
|
||||||
|
}
|
||||||
86
internal/modules/spoken_word/prompt.go
Normal file
86
internal/modules/spoken_word/prompt.go
Normal file
@@ -0,0 +1,86 @@
|
|||||||
|
package spoken_word
|
||||||
|
|
||||||
|
import (
|
||||||
|
"encoding/json"
|
||||||
|
"fmt"
|
||||||
|
|
||||||
|
"gitea.maximumdirect.net/eric/audita/internal/core/schema"
|
||||||
|
"gitea.maximumdirect.net/eric/audita/internal/framework/contracts"
|
||||||
|
)
|
||||||
|
|
||||||
|
type promptSegment struct {
|
||||||
|
ID int `json:"id"`
|
||||||
|
Speaker string `json:"speaker"`
|
||||||
|
Start float64 `json:"start"`
|
||||||
|
End float64 `json:"end"`
|
||||||
|
Text string `json:"text"`
|
||||||
|
Categories []string `json:"categories,omitempty"`
|
||||||
|
}
|
||||||
|
|
||||||
|
type promptTranscriptSection struct {
|
||||||
|
SectionIndex int `json:"section_index"`
|
||||||
|
Segments []promptSegment `json:"segments"`
|
||||||
|
}
|
||||||
|
|
||||||
|
// BuildProposalMessages mirrors the Python spoken_word-module prompt intent:
|
||||||
|
// conservative dysfluency cleanup with strict semantic preservation.
|
||||||
|
func BuildProposalMessages(transcript *schema.Transcript, glossary *schema.Glossary) ([]contracts.LLMMessage, error) {
|
||||||
|
glossaryJSON, err := json.MarshalIndent(glossary, "", " ")
|
||||||
|
if err != nil {
|
||||||
|
return nil, fmt.Errorf("marshal glossary prompt context: %w", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
sectionPayload := promptTranscriptSection{
|
||||||
|
SectionIndex: 0,
|
||||||
|
Segments: make([]promptSegment, 0),
|
||||||
|
}
|
||||||
|
if transcript != nil {
|
||||||
|
for _, s := range transcript.Segments {
|
||||||
|
sectionPayload.Segments = append(sectionPayload.Segments, promptSegment{
|
||||||
|
ID: s.ID,
|
||||||
|
Speaker: s.Speaker,
|
||||||
|
Start: s.Start,
|
||||||
|
End: s.End,
|
||||||
|
Text: s.Text,
|
||||||
|
Categories: append([]string(nil), s.Categories...),
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
sectionJSON, err := json.MarshalIndent(sectionPayload, "", " ")
|
||||||
|
if err != nil {
|
||||||
|
return nil, fmt.Errorf("marshal transcript prompt context: %w", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
system := "You are Audita, a conservative spoken-word cleanup assistant. Identify only low-risk cleanup of repeated words or short phrases, filler words, hesitation artifacts, and similar dysfluencies that commonly appear in spoken English transcripts. Preserve substantive meaning, named entities, and transcript content."
|
||||||
|
user := "Review this transcript section and return only spoken-word cleanup corrections that should be applied.\n\n" +
|
||||||
|
"Rules:\n" +
|
||||||
|
"- Approve only conservative cleanup of repeated words, repeated short phrases, filler words, hesitation artifacts, and similar spoken dysfluencies.\n" +
|
||||||
|
"- You may collapse adjacent repetition such as \"I I think\" to \"I think\" or remove filler spans such as \"you know\" or \"uh\" when local context supports that cleanup.\n" +
|
||||||
|
"- Do not collapse repeated words or short phrases when the repetition plausibly expresses urgency, excitement, insistence, or deliberate rhetorical emphasis rather than dysfluency.\n" +
|
||||||
|
"- Phrases such as \"Help! Help! Help!\", \"Stop! Stop! Stop!\", \"No! No! No!\", \"Yes! Yes! Yes!\", and \"Go! Go! Go!\" are often intentional emphasis and should usually be preserved.\n" +
|
||||||
|
"- Only collapse repetition when local context supports it as accidental spoken repetition, hesitation, or verbal restart.\n" +
|
||||||
|
"- You may include low-risk punctuation, spacing, or capitalization cleanup when it is part of removing a dysfluency, such as removing ellipses or hesitation punctuation that no longer belongs after the cleanup.\n" +
|
||||||
|
"- Do not paraphrase, summarize, reorder ideas, replace content with different wording, or make substantive semantic edits.\n" +
|
||||||
|
"- Do not change clear content words just because a different phrasing reads better.\n" +
|
||||||
|
"- Do not convert uncertain statements into certain statements.\n" +
|
||||||
|
"- Treat glossary names and aliases as protected spellings and context.\n" +
|
||||||
|
"- Do not replace, Anglicize, normalize, lowercase, or otherwise alter protected glossary names or aliases that already appear correctly in the transcript.\n" +
|
||||||
|
"- Preserve canonical glossary capitalization for protected names and aliases, even if they look unusual.\n" +
|
||||||
|
"- If a segment includes categories, treat them as additional transcript context.\n" +
|
||||||
|
"- Use the exact id from the input segment.\n" +
|
||||||
|
"- For returned corrections, original_text must be only the exact text span that needs replacement, not the full segment text unless the whole segment is the replacement span.\n" +
|
||||||
|
"- Choose an original_text span that appears exactly once in the current segment text.\n" +
|
||||||
|
"- corrected_text must be only the replacement text for that span, not the full corrected segment text unless the whole segment is the replacement span.\n" +
|
||||||
|
"- Each returned correction must contain only id, original_text, corrected_text, and confidence.\n" +
|
||||||
|
"- Do not return corrections where original_text and corrected_text are identical.\n" +
|
||||||
|
"- Do not return speaker, start, or end fields.\n" +
|
||||||
|
"- Return only changed segments; do not return entries for unchanged segments.\n" +
|
||||||
|
"- confidence must be between 0.0 and 1.0.\n" +
|
||||||
|
"- If no corrections are needed, return an empty corrections list.\n\n" +
|
||||||
|
fmt.Sprintf("Protected glossary/context:\n%s\n\nTranscript section:\n%s", string(glossaryJSON), string(sectionJSON))
|
||||||
|
|
||||||
|
return []contracts.LLMMessage{
|
||||||
|
{Role: "system", Content: system},
|
||||||
|
{Role: "user", Content: user},
|
||||||
|
}, nil
|
||||||
|
}
|
||||||
Reference in New Issue
Block a user