Complete Phase 12 grammar module

This commit is contained in:
2026-05-12 02:57:06 +00:00
parent b360493cdc
commit fc3a7b7a67
12 changed files with 816 additions and 88 deletions

View File

@@ -33,9 +33,11 @@ Implemented today:
- Deterministic proposal-index assignment and enriched proposal mapping for shared generation. - Deterministic proposal-index assignment and enriched proposal mapping for shared generation.
- Proposal-generation diagnostics artifacts with secret redaction. - Proposal-generation diagnostics artifacts with secret redaction.
- Production module registry scaffolding with known-key recognition and explicit unsupported/unimplemented errors. - Production module registry scaffolding with known-key recognition and explicit unsupported/unimplemented errors.
- Production `grammar` module implementation in `internal/modules/grammar`.
- Explicit runtime support for `--modules grammar` through the production runner path.
Not implemented in CLI runtime path today: Not implemented in CLI runtime path today:
- Real module execution pipeline (`glossary`, `homophones`, `spoken_word`, `grammar`). - Real module execution pipeline for `glossary`, `homophones`, and `spoken_word`.
- Real domain proposal prompts for production modules. - Real domain proposal prompts for production modules.
- End-to-end transcript polishing with real module behavior. - End-to-end transcript polishing with real module behavior.
@@ -43,7 +45,8 @@ Phase sequencing note:
- Phase 9 LLM infrastructure is complete (structured client, scheduler, effective config resolution, diagnostics primitives); - Phase 9 LLM infrastructure is complete (structured client, scheduler, effective config resolution, diagnostics primitives);
- Phase 10 LLM-backed validator runtime integration is complete; - Phase 10 LLM-backed validator runtime integration is complete;
- Phase 11 shared proposal-generation framework and module-registry scaffolding are complete; - Phase 11 shared proposal-generation framework and module-registry scaffolding are complete;
- next recommended phase is Phase 12 (grammar module). - Phase 12 grammar module implementation and explicit runtime wiring are complete;
- next recommended phase is Phase 13 (glossary module and protected-term behavior).
## Actual Go package layout ## Actual Go package layout
@@ -102,6 +105,10 @@ internal/framework/proposal_generation/
internal/framework/modules/ internal/framework/modules/
registry.go registry.go
internal/modules/grammar/
module.go
prompt.go
internal/framework/validators/ internal/framework/validators/
models.go models.go
deterministic.go deterministic.go
@@ -136,16 +143,19 @@ Current runtime flow (`internal/cli/run.go`):
9. Write normalized transcript and normalization summary artifacts. 9. Write normalized transcript and normalization summary artifacts.
10. Chunk normalized transcript and compute chunk summaries. 10. Chunk normalized transcript and compute chunk summaries.
11. Write chunking summary artifact. 11. Write chunking summary artifact.
12. Optionally execute runner modules sequentially when a module factory is injected (tests currently use this path; production defaults still avoid real module execution). 12. Execute runner modules sequentially when:
- `--modules` is explicitly provided (production grammar path); or
- a test/injected module factory is provided.
13. Output working transcript to `--output` file or stdout. 13. Output working transcript to `--output` file or stdout.
14. Build process report (`phase` currently set to `phase11-proposal-generation-framework`). 14. Build process report (`phase` currently set to `phase12-grammar-module`).
15. Optionally write `--report-json`; always write run-dir `report.json`. 15. Optionally write `--report-json`; always write run-dir `report.json`.
16. Apply work-dir retention. 16. Apply work-dir retention.
Important behavior details: Important behavior details:
- Glossary is validated but not yet used for real correction module logic. - Glossary is validated but not yet used for real correction module logic.
- Default production CLI behavior remains deterministic normalization/chunking output because no real module implementations are registered yet. - Default production CLI behavior remains deterministic normalization/chunking/reporting unless modules are explicitly selected with `--modules`.
- No real LLM calls occur in the default production runtime path because no real modules are registered yet. - Explicit `--modules grammar` runs the production grammar module path with LLM-backed proposal generation and validator-chain execution.
- Default runs (without explicit module selection) do not perform LLM calls.
- Success path is generally quiet on stderr. - Success path is generally quiet on stderr.
- Source IDs are preserved into a canonical transcript before normalization; normalization then reassigns output IDs sequentially from `1`. - Source IDs are preserved into a canonical transcript before normalization; normalization then reassigns output IDs sequentially from `1`.
@@ -193,7 +203,7 @@ Implemented config surfaces include:
- work-dir and retention mode - work-dir and retention mode
Current caveat: Current caveat:
- LLM/module-related settings are mostly infrastructure-only today; default runtime path does not execute real modules. - LLM/module-related settings are active for explicit grammar runs; the default non-explicit path remains deterministic.
## Implemented structured LLM infrastructure ## Implemented structured LLM infrastructure
`internal/framework/contracts` now defines a typed structured-completion contract: `internal/framework/contracts` now defines a typed structured-completion contract:
@@ -209,8 +219,8 @@ Current caveat:
- API-key redaction in adapter-returned errors. - API-key redaction in adapter-returned errors.
Current runtime boundary: Current runtime boundary:
- the default CLI runtime path still does not instantiate real production modules, so no default end-to-end LLM polishing occurs. - the default CLI runtime path (without explicit module selection) still does not instantiate the full production module sequence.
- LLM calls are exercised only when test/injected modules and validators are provided. - LLM calls are exercised in production when `--modules grammar` is explicitly requested and in tests when fake/injected clients are used.
`internal/framework/llm` also provides: `internal/framework/llm` also provides:
- a bounded `Scheduler` for controlled concurrent LLM calls with reliable permit release; - a bounded `Scheduler` for controlled concurrent LLM calls with reliable permit release;
@@ -252,7 +262,7 @@ Current behavior details:
`internal/framework/contracts` provides interfaces and run-spec metadata scaffolding, including deterministic repeated module instance naming (`ResolveModuleRunSpecs`). `internal/framework/contracts` provides interfaces and run-spec metadata scaffolding, including deterministic repeated module instance naming (`ResolveModuleRunSpecs`).
These primitives are wired into the production runner and report model. Real module implementations are still pending. These primitives are wired into the production runner and report model. The grammar module is implemented; other production modules remain pending.
## Implemented validator runtime infrastructure ## Implemented validator runtime infrastructure
`internal/framework/validators` provides deterministic validator infrastructure: `internal/framework/validators` provides deterministic validator infrastructure:
@@ -317,7 +327,18 @@ This helper only produces candidate proposals; validator-chain execution and pro
- diagnostics directory context - diagnostics directory context
- returns explicit errors for unknown keys (`unsupported_module`) and recognized-but-unimplemented keys (`unimplemented_module`). - returns explicit errors for unknown keys (`unsupported_module`) and recognized-but-unimplemented keys (`unimplemented_module`).
No real production correction modules are registered yet. The `grammar` module key is now registered and constructible. `glossary`, `homophones`, and `spoken_word` remain recognized-but-unimplemented.
## Implemented grammar production module
`internal/modules/grammar` now provides the first production module:
- prompt builder faithfully constrained to punctuation/capitalization/spacing/article cleanup;
- explicit guardrails against meaning-changing rewrites, style rewrites, summarization, and invention;
- proposal generation through `internal/framework/proposal_generation` and `contracts.StructuredLLMClient`;
- scheduler-aware proposal calls through existing `contracts.LLMScheduler` hooks;
- replacement policy `require_unique` (matching Python implementation);
- validator chain integration using existing deterministic + LLM-backed validators;
- grammar confidence threshold enforcement through existing validator/config infrastructure;
- module-level reporting and diagnostics capture through existing runner/reporting paths.
## Reports and diagnostics (implemented) ## Reports and diagnostics (implemented)
Current per-run artifacts include: Current per-run artifacts include:
@@ -357,7 +378,8 @@ Retention modes implemented in `ApplyRetention`:
- failed runs are always retained. - failed runs are always retained.
Current runtime note: Current runtime note:
- real module execution is not implemented yet, so normal successful runs generally have no skipped corrections and `auto` typically removes clean successful run directories. - default non-explicit runs usually have no module-level skipped corrections, so `auto` commonly removes clean successful run directories.
- explicit grammar runs can produce validator rejections and application skips, which are reflected in reports and retention input.
Intentionally deferred to module/LLM phases: Intentionally deferred to module/LLM phases:
- real domain proposal prompts and production module implementations remain tied to later module phases. - real domain proposal prompts and production module implementations remain tied to later module phases.
@@ -378,8 +400,9 @@ Implemented tests currently cover:
- LLM-backed validator batching, prompt builders, structured-response safety, scheduler hooks, and diagnostics redaction (`internal/framework/validators/*_test.go`, `internal/framework/runner/*_test.go`) - LLM-backed validator batching, prompt builders, structured-response safety, scheduler hooks, and diagnostics redaction (`internal/framework/validators/*_test.go`, `internal/framework/runner/*_test.go`)
- shared proposal-generation request/response parsing, deterministic indexing, scheduler hooks, and diagnostics redaction (`internal/framework/proposal_generation/*_test.go`, `internal/framework/runner/*_test.go`) - shared proposal-generation request/response parsing, deterministic indexing, scheduler hooks, and diagnostics redaction (`internal/framework/proposal_generation/*_test.go`, `internal/framework/runner/*_test.go`)
- production module-registry known-key recognition and unsupported/unimplemented error behavior (`internal/framework/modules/*_test.go`, `internal/cli/run_test.go`) - production module-registry known-key recognition and unsupported/unimplemented error behavior (`internal/framework/modules/*_test.go`, `internal/cli/run_test.go`)
- production grammar module prompt constraints, proposal mapping, validator-chain behavior, confidence-threshold enforcement, diagnostics redaction, and explicit CLI/runtime integration (`internal/modules/grammar/*_test.go`, `internal/cli/run_test.go`, `internal/framework/runner/*_test.go`)
Not covered yet (because not implemented): real production module implementations and full transcript-polishing runtime behavior. Not covered yet (because not implemented): production `glossary`, `homophones`, and `spoken_word` modules plus full default-sequence transcript-polishing runtime behavior.
## Intended final architecture (not yet implemented) ## Intended final architecture (not yet implemented)
The intended end-state still matches the rewrite plan: The intended end-state still matches the rewrite plan:

View File

@@ -59,6 +59,8 @@ Implemented:
- Deterministic proposal-index assignment for shared proposal generation. - Deterministic proposal-index assignment for shared proposal generation.
- Proposal-generation prompt/response diagnostics artifact wiring with secret redaction. - Proposal-generation prompt/response diagnostics artifact wiring with secret redaction.
- Production module-registry scaffolding with known key recognition and explicit unsupported/unimplemented errors. - Production module-registry scaffolding with known key recognition and explicit unsupported/unimplemented errors.
- Production grammar module package with Python-aligned prompt intent and guardrails.
- Explicit `--modules grammar` runtime path through runner, shared proposal generation, validators, application, reporting, and diagnostics.
- Broad deterministic and CLI/subprocess test coverage for implemented phases through `go test ./...`. - Broad deterministic and CLI/subprocess test coverage for implemented phases through `go test ./...`.
- Internal typed structured LLM contract (`StructuredLLMClient.CompleteStructured(ctx, req, out)`). - Internal typed structured LLM contract (`StructuredLLMClient.CompleteStructured(ctx, req, out)`).
- `internal/framework/llm` instructor-go-backed adapter with: - `internal/framework/llm` instructor-go-backed adapter with:
@@ -74,8 +76,8 @@ Implemented:
- Generic JSON diagnostics primitives for LLM interactions (request metadata, request payload, response payload, optional error payload) with secret redaction. - Generic JSON diagnostics primitives for LLM interactions (request metadata, request payload, response payload, optional error payload) with secret redaction.
Not yet implemented in runtime pipeline: Not yet implemented in runtime pipeline:
- Real correction modules. - Real correction modules for `glossary`, `homophones`, and `spoken_word`.
- Domain proposal prompts for real modules. - Domain proposal prompts for remaining real modules.
- End-to-end transcript polishing behavior. - End-to-end transcript polishing behavior.
## Completed phases ## Completed phases
@@ -214,7 +216,7 @@ Not implemented in Phase 8 (by design):
## Remaining work plan ## Remaining work plan
Next recommended phase: **Phase 12 (grammar module)**. Next recommended phase: **Phase 13 (glossary module and protected-term behavior)**.
## Phase 9: Structured LLM client and scheduler infrastructure ## Phase 9: Structured LLM client and scheduler infrastructure
@@ -333,52 +335,30 @@ Implemented:
- Runner/CLI injection-path tests showing shared proposal generation can flow through runner validation/application semantics using fake modules/clients. - Runner/CLI injection-path tests showing shared proposal generation can flow through runner validation/application semantics using fake modules/clients.
Not implemented in Phase 11 (by design): Not implemented in Phase 11 (by design):
- Real production `glossary`, `homophones`, `spoken_word`, and `grammar` modules. - Real production `glossary`, `homophones`, and `spoken_word` modules.
- Domain proposal prompts for production modules. - Domain proposal prompts for production modules.
- Default CLI end-to-end transcript polishing behavior. - Default CLI end-to-end transcript polishing behavior.
## Phase 12: Grammar module ## Phase 12: Grammar module
### Purpose Completed.
Implement the first production module in the Go runtime path. Grammar is a good first real module because it exercises LLM proposal generation and validator chains while remaining constrained to punctuation, capitalization, and spacing cleanup. Implemented:
- Production grammar module package in `internal/modules/grammar`.
- Grammar prompt builder aligned to Python intent and constrained to punctuation/capitalization/spacing/article cleanup.
- Grammar proposal generation through shared `internal/framework/proposal_generation` using `contracts.StructuredLLMClient`.
- Scheduler-aware grammar proposal generation through existing scheduler hooks.
- Grammar replacement policy `require_unique` (matching Python behavior).
- Grammar validator chain using existing deterministic and LLM-backed validator infrastructure.
- Grammar confidence threshold enforcement through existing config + confidence-threshold validator behavior.
- Explicit runtime support for `--modules grammar` through normalization, chunking, runner, proposal generation, validation, application, and reporting.
- Prompt/response diagnostics artifacts for grammar proposal + validator interactions with secret redaction.
- Module-level reports for grammar including validator decisions/rejections, applied changes, and skipped changes.
- CLI/runtime fake-client tests for approved proposals, validator rejection, application skips, diagnostics, failure/error.log behavior, and report outputs (`--report-json` and run-dir `report.json`).
### Scope Not implemented in Phase 12 (by design):
- Production `glossary`, `homophones`, and `spoken_word` modules.
Implement: - Full default module sequence execution as a feature-complete claim.
- `grammar` module package.
- Grammar prompt builder ported from the Python implementation.
- Grammar structured response model.
- Grammar replacement policy.
- Grammar validator chain.
- Grammar confidence threshold handling.
- Prompt/response diagnostics.
- Module-level report integration.
- CLI support for `--modules grammar`.
- Fake LLM tests.
- Optional real LLM smoke test gated so normal `go test ./...` does not require credentials.
Do not implement:
- Glossary module.
- Homophones module.
- Spoken-word module.
- Default full module sequence as active parity claim.
### Expected behavior at end of phase
Running `audita process ... --modules grammar` should perform real grammar-stage transcript polishing using the configured LLM endpoint.
### Definition of done
- Grammar module runs in the production runner.
- Grammar module generates structured proposals through the LLM client.
- Grammar validator chain runs.
- Approved grammar proposals are applied.
- Applied/skipped grammar changes appear in reports.
- Prompt/response diagnostics are written.
- `--modules grammar` works end-to-end.
- Default pipeline is not yet claimed complete.
- `go test ./...` passes without requiring external LLM credentials.
## Phase 13: Glossary module and protected-term behavior ## Phase 13: Glossary module and protected-term behavior

View File

@@ -18,15 +18,18 @@ import (
"gitea.maximumdirect.net/eric/audita/internal/core/reporting" "gitea.maximumdirect.net/eric/audita/internal/core/reporting"
"gitea.maximumdirect.net/eric/audita/internal/core/schema" "gitea.maximumdirect.net/eric/audita/internal/core/schema"
"gitea.maximumdirect.net/eric/audita/internal/framework/contracts" "gitea.maximumdirect.net/eric/audita/internal/framework/contracts"
"gitea.maximumdirect.net/eric/audita/internal/framework/llm"
"gitea.maximumdirect.net/eric/audita/internal/framework/modules"
"gitea.maximumdirect.net/eric/audita/internal/framework/runner" "gitea.maximumdirect.net/eric/audita/internal/framework/runner"
) )
type processInvocation struct { type processInvocation struct {
TranscriptPath string TranscriptPath string
GlossaryPath string GlossaryPath string
OutputPath string OutputPath string
ReportJSONPath string ReportJSONPath string
Config config.Config Config config.Config
ExplicitModules bool
} }
var processModuleFactory runner.ModuleFactory var processModuleFactory runner.ModuleFactory
@@ -127,22 +130,71 @@ var processRunner = func(inv processInvocation, stdout io.Writer) (*normalizatio
workingTranscript := normalizedTranscript workingTranscript := normalizedTranscript
var runOutput *runner.RunOutput var runOutput *runner.RunOutput
if processModuleFactory != nil { moduleFactory := processModuleFactory
if moduleFactory == nil && inv.ExplicitModules {
moduleFactory = modules.NewFactory(modules.Dependencies{
Config: &inv.Config,
Glossary: glossary,
DiagnosticsDir: runDir.Path(),
})
}
if moduleFactory != nil {
proposalLLMClient := processProposalLLMClient
validationLLMClient := processValidationLLMClient
proposalScheduler := processProposalLLMScheduler
validationScheduler := processValidationLLMScheduler
if processModuleFactory == nil {
// Production runtime path: construct clients/schedulers from config.
if proposalLLMClient == nil {
primaryCfg := llm.ResolvePrimaryConfig(inv.Config)
client, clientErr := llm.NewInstructorClient(primaryCfg.ToInstructorClientConfig(llm.ModeJSON, nil))
if clientErr != nil {
return fail("runner_setup", clientErr, nil)
}
proposalLLMClient = client
}
if validationLLMClient == nil {
validationCfg := llm.ResolveValidationConfig(inv.Config)
client, clientErr := llm.NewInstructorClient(validationCfg.ToInstructorClientConfig(llm.ModeJSON, nil))
if clientErr != nil {
return fail("runner_setup", clientErr, nil)
}
validationLLMClient = client
}
if proposalScheduler == nil {
primaryCfg := llm.ResolvePrimaryConfig(inv.Config)
s, sErr := llm.NewScheduler(primaryCfg.Concurrency)
if sErr != nil {
return fail("runner_setup", sErr, nil)
}
proposalScheduler = s
}
if validationScheduler == nil {
validationCfg := llm.ResolveValidationConfig(inv.Config)
s, sErr := llm.NewScheduler(validationCfg.Concurrency)
if sErr != nil {
return fail("runner_setup", sErr, nil)
}
validationScheduler = s
}
}
moduleSpecs, err := contracts.ResolveModuleRunSpecs(inv.Config.Modules) moduleSpecs, err := contracts.ResolveModuleRunSpecs(inv.Config.Modules)
if err != nil { if err != nil {
return fail("runner_setup", err, nil) return fail("runner_setup", err, nil)
} }
runnerResult, runErr := runner.New(processModuleFactory).Run(context.Background(), runner.RunInput{ runnerResult, runErr := runner.New(moduleFactory).Run(context.Background(), runner.RunInput{
Config: &inv.Config, Config: &inv.Config,
Transcript: normalizedTranscript, Transcript: normalizedTranscript,
Glossary: glossary, Glossary: glossary,
ModuleSpecs: moduleSpecs, ModuleSpecs: moduleSpecs,
ProposalLLMClient: processProposalLLMClient, ProposalLLMClient: proposalLLMClient,
ProposalLLMScheduler: processProposalLLMScheduler, ProposalLLMScheduler: proposalScheduler,
ProposalDiagnosticsDir: runDir.Path(), ProposalDiagnosticsDir: runDir.Path(),
ValidationLLMClient: processValidationLLMClient, ValidationLLMClient: validationLLMClient,
ValidationLLMScheduler: processValidationLLMScheduler, ValidationLLMScheduler: validationScheduler,
ValidationDiagnosticsDir: runDir.Path(), ValidationDiagnosticsDir: runDir.Path(),
}) })
runOutput = &runnerResult runOutput = &runnerResult
@@ -243,9 +295,11 @@ func runProcess(args []string, stdout, stderr io.Writer) int {
} }
overrides := config.CLIOverrides{} overrides := config.CLIOverrides{}
explicitModules := false
fs.Visit(func(f *flag.Flag) { fs.Visit(func(f *flag.Flag) {
switch f.Name { switch f.Name {
case "modules": case "modules":
explicitModules = true
overrides.ModulesCSV = pFlags.modules overrides.ModulesCSV = pFlags.modules
case "llm-api-key": case "llm-api-key":
overrides.PrimaryLLMAPIKey = pFlags.llmAPIKey overrides.PrimaryLLMAPIKey = pFlags.llmAPIKey
@@ -322,11 +376,12 @@ func runProcess(args []string, stdout, stderr io.Writer) int {
} }
inv := processInvocation{ inv := processInvocation{
TranscriptPath: positional[0], TranscriptPath: positional[0],
GlossaryPath: *pFlags.glossaryPath, GlossaryPath: *pFlags.glossaryPath,
OutputPath: *pFlags.outputPath, OutputPath: *pFlags.outputPath,
ReportJSONPath: *pFlags.reportJSONPath, ReportJSONPath: *pFlags.reportJSONPath,
Config: cfg, Config: cfg,
ExplicitModules: explicitModules,
} }
normSummary, chunkSummary, runOutput, runDir, runErr := processRunner(inv, stdout) normSummary, chunkSummary, runOutput, runDir, runErr := processRunner(inv, stdout)
@@ -405,7 +460,7 @@ func extractErrorPhase(err error) (phase string, message string) {
func buildProcessReport(status string, inv processInvocation, runDir *diagnostics.RunDirectory, startedAt, completedAt time.Time, errorMessage string, errorPhase string, normalizationSummary *normalization.NormalizationSummary, chunkingSummary *chunking.Summary, runOutput *runner.RunOutput) reporting.ProcessReport { func buildProcessReport(status string, inv processInvocation, runDir *diagnostics.RunDirectory, startedAt, completedAt time.Time, errorMessage string, errorPhase string, normalizationSummary *normalization.NormalizationSummary, chunkingSummary *chunking.Summary, runOutput *runner.RunOutput) reporting.ProcessReport {
report := reporting.ProcessReport{ report := reporting.ProcessReport{
Phase: "phase11-proposal-generation-framework", Phase: "phase12-grammar-module",
Status: status, Status: status,
Operation: "process", Operation: "process",
TranscriptPath: inv.TranscriptPath, TranscriptPath: inv.TranscriptPath,

View File

@@ -15,6 +15,7 @@ import (
"gitea.maximumdirect.net/eric/audita/internal/core/schema" "gitea.maximumdirect.net/eric/audita/internal/core/schema"
"gitea.maximumdirect.net/eric/audita/internal/framework/contracts" "gitea.maximumdirect.net/eric/audita/internal/framework/contracts"
"gitea.maximumdirect.net/eric/audita/internal/framework/modules" "gitea.maximumdirect.net/eric/audita/internal/framework/modules"
"gitea.maximumdirect.net/eric/audita/internal/framework/proposal_generation"
"gitea.maximumdirect.net/eric/audita/internal/framework/proposals" "gitea.maximumdirect.net/eric/audita/internal/framework/proposals"
"gitea.maximumdirect.net/eric/audita/internal/framework/validators" "gitea.maximumdirect.net/eric/audita/internal/framework/validators"
) )
@@ -615,8 +616,8 @@ func TestRunProcessReportJSONIncludesChunkingSummary(t *testing.T) {
if report.Chunking.MaxSectionTokens == 0 { if report.Chunking.MaxSectionTokens == 0 {
t.Errorf("expected max_section_tokens in report") t.Errorf("expected max_section_tokens in report")
} }
if report.Phase != "phase11-proposal-generation-framework" { if report.Phase != "phase12-grammar-module" {
t.Errorf("expected phase 'phase11-proposal-generation-framework', got %q", report.Phase) t.Errorf("expected phase 'phase12-grammar-module', got %q", report.Phase)
} }
} }
@@ -661,8 +662,9 @@ func (v fakeValidator) Validate(ctx context.Context, req contracts.ValidationReq
} }
type fakeStructuredLLMClient struct { type fakeStructuredLLMClient struct {
responses []validators.LLMValidationResponse validationResponses []validators.LLMValidationResponse
err error proposalResponses []proposal_generation.StructuredCorrectionSet
err error
} }
func (f *fakeStructuredLLMClient) CompleteStructured(ctx context.Context, req contracts.StructuredCompletionRequest, out any) (contracts.StructuredCompletionResponse, error) { func (f *fakeStructuredLLMClient) CompleteStructured(ctx context.Context, req contracts.StructuredCompletionRequest, out any) (contracts.StructuredCompletionResponse, error) {
@@ -671,13 +673,24 @@ func (f *fakeStructuredLLMClient) CompleteStructured(ctx context.Context, req co
if f.err != nil { if f.err != nil {
return contracts.StructuredCompletionResponse{}, f.err return contracts.StructuredCompletionResponse{}, f.err
} }
if len(f.responses) == 0 { switch target := out.(type) {
return contracts.StructuredCompletionResponse{}, errors.New("unexpected llm call") case *validators.LLMValidationResponse:
if len(f.validationResponses) == 0 {
return contracts.StructuredCompletionResponse{}, errors.New("unexpected validation llm call")
}
*target = f.validationResponses[0]
f.validationResponses = f.validationResponses[1:]
return contracts.StructuredCompletionResponse{}, nil
case *proposal_generation.StructuredCorrectionSet:
if len(f.proposalResponses) == 0 {
return contracts.StructuredCompletionResponse{}, errors.New("unexpected proposal llm call")
}
*target = f.proposalResponses[0]
f.proposalResponses = f.proposalResponses[1:]
return contracts.StructuredCompletionResponse{}, nil
default:
return contracts.StructuredCompletionResponse{}, errors.New("unexpected llm output type")
} }
target := out.(*validators.LLMValidationResponse)
*target = f.responses[0]
f.responses = f.responses[1:]
return contracts.StructuredCompletionResponse{}, nil
} }
func TestRunProcessInjectedFactoryExecutesRunnerAndReportsModules(t *testing.T) { func TestRunProcessInjectedFactoryExecutesRunnerAndReportsModules(t *testing.T) {
@@ -771,7 +784,7 @@ func TestRunProcessInjectedFactoryLLMValidatorResultsInReports(t *testing.T) {
t.Fatalf("NewLLMBackedValidator: %v", err) t.Fatalf("NewLLMBackedValidator: %v", err)
} }
processValidationLLMClient = &fakeStructuredLLMClient{ processValidationLLMClient = &fakeStructuredLLMClient{
responses: []validators.LLMValidationResponse{{ validationResponses: []validators.LLMValidationResponse{{
Validations: []validators.LLMValidationDecision{{CorrectionIndex: 0, Approved: false, Confidence: 0.99, Reason: "reject"}}, Validations: []validators.LLMValidationDecision{{CorrectionIndex: 0, Approved: false, Confidence: 0.99, Reason: "reject"}},
}}, }},
} }
@@ -940,6 +953,249 @@ func TestRunProcessProductionRegistryUnimplementedModuleFailsCleanly(t *testing.
} }
} }
func TestRunProcessExplicitUnimplementedModulesFailClearly(t *testing.T) {
for _, moduleKey := range []string{"glossary", "homophones", "spoken_word"} {
t.Run(moduleKey, func(t *testing.T) {
var stdout, stderr bytes.Buffer
transcriptPath := writeFile(t, "transcript.json", `[
{"id":1,"speaker":"Alice","start":0.0,"end":1.0,"text":"Hello"}
]`)
exitCode := Run([]string{
"process", transcriptPath,
"--glossary", fixturePath("tiny_glossary.yaml"),
"--modules", moduleKey,
}, &stdout, &stderr)
if exitCode == 0 {
t.Fatalf("expected failure for %s", moduleKey)
}
if stdout.Len() != 0 {
t.Fatalf("expected empty stdout on failure, got %q", stdout.String())
}
if !strings.Contains(stderr.String(), "recognized but not implemented") {
t.Fatalf("expected unimplemented message, got %q", stderr.String())
}
})
}
}
func TestRunProcessExplicitGrammarAppliesCorrectionAndReportsDiagnostics(t *testing.T) {
secret := "phase12-secret"
processProposalLLMClient = &fakeStructuredLLMClient{
proposalResponses: []proposal_generation.StructuredCorrectionSet{
{
Corrections: []proposal_generation.StructuredCorrectionProposal{
{TargetSegmentID: 1, OriginalText: "hello ,world", CorrectedText: "Hello, world", Confidence: 0.95},
},
},
},
}
processValidationLLMClient = &fakeStructuredLLMClient{
validationResponses: []validators.LLMValidationResponse{
{Validations: []validators.LLMValidationDecision{{CorrectionIndex: 0, Approved: true, Confidence: 0.99, Reason: "ok"}}},
{Validations: []validators.LLMValidationDecision{{CorrectionIndex: 0, Approved: true, Confidence: 0.99, Reason: secret}}},
},
}
t.Setenv("AUDITA_LLM_API_KEY", secret)
t.Setenv("AUDITA_VALIDATION_LLM_API_KEY", secret)
t.Cleanup(func() {
processProposalLLMClient = nil
processValidationLLMClient = nil
processProposalLLMScheduler = nil
processValidationLLMScheduler = nil
})
var stdout, stderr bytes.Buffer
workDir := t.TempDir()
reportPath := filepath.Join(t.TempDir(), "report.json")
outputPath := filepath.Join(t.TempDir(), "out.json")
transcriptPath := writeFile(t, "transcript.json", `[
{"id":1,"speaker":"Alice","start":0.0,"end":1.0,"text":"hello ,world"}
]`)
exitCode := Run([]string{
"process", transcriptPath,
"--glossary", fixturePath("tiny_glossary.yaml"),
"--modules", "grammar",
"--output", outputPath,
"--report-json", reportPath,
"--work-dir", workDir,
"--work-dir-retention", "always",
}, &stdout, &stderr)
if exitCode != 0 {
t.Fatalf("expected success, got %d stderr=%q", exitCode, stderr.String())
}
if stdout.Len() != 0 {
t.Fatalf("expected empty stdout with --output, got %q", stdout.String())
}
parsed, err := schema.ParseTranscriptJSON(readFile(t, outputPath))
if err != nil {
t.Fatalf("parse output: %v", err)
}
if parsed.Segments[0].Text != "Hello, world" {
t.Fatalf("expected grammar correction applied, got %q", parsed.Segments[0].Text)
}
report := readProcessReport(t, reportPath)
if len(report.ModuleResults) != 1 || report.ModuleResults[0].ModuleKey != "grammar" {
t.Fatalf("expected one grammar module result, got %+v", report.ModuleResults)
}
if len(report.ModuleResults[0].AppliedChanges) != 1 {
t.Fatalf("expected one applied grammar change, got %+v", report.ModuleResults[0].AppliedChanges)
}
if len(report.ModuleResults[0].ValidatorDecisions) == 0 {
t.Fatalf("expected validator decisions in report")
}
runDir := onlyRunDir(t, workDir)
runReport := readProcessReport(t, filepath.Join(runDir, "report.json"))
if len(runReport.ModuleResults) != 1 {
t.Fatalf("expected module results in run-dir report")
}
diagFiles, globErr := filepath.Glob(filepath.Join(runDir, "grammar", "*response-payload.json"))
if globErr != nil {
t.Fatalf("glob diagnostics: %v", globErr)
}
if len(diagFiles) == 0 {
t.Fatalf("expected grammar diagnostics payload files in %s", filepath.Join(runDir, "grammar"))
}
for _, f := range diagFiles {
raw := string(readFile(t, f))
if strings.Contains(raw, secret) {
t.Fatalf("secret leaked in diagnostics %q: %s", f, raw)
}
}
}
func TestRunProcessExplicitGrammarRejectedAndApplicationSkipAreDistinct(t *testing.T) {
processProposalLLMClient = &fakeStructuredLLMClient{
proposalResponses: []proposal_generation.StructuredCorrectionSet{
{
Corrections: []proposal_generation.StructuredCorrectionProposal{
{TargetSegmentID: 1, OriginalText: "Hello", CorrectedText: "Hi", Confidence: 0.99},
{TargetSegmentID: 1, OriginalText: "Hello world", CorrectedText: "Hi world", Confidence: 0.99},
{TargetSegmentID: 1, OriginalText: "world", CorrectedText: "earth", Confidence: 0.99},
},
},
},
}
processValidationLLMClient = &fakeStructuredLLMClient{
validationResponses: []validators.LLMValidationResponse{
{
Validations: []validators.LLMValidationDecision{
{CorrectionIndex: 0, Approved: true, Confidence: 0.9, Reason: "ok"},
{CorrectionIndex: 1, Approved: true, Confidence: 0.9, Reason: "ok"},
{CorrectionIndex: 2, Approved: false, Confidence: 0.9, Reason: "reject"},
},
},
{
Validations: []validators.LLMValidationDecision{
{CorrectionIndex: 0, Approved: true, Confidence: 0.9, Reason: "ok"},
{CorrectionIndex: 1, Approved: true, Confidence: 0.9, Reason: "ok"},
},
},
},
}
t.Cleanup(func() {
processProposalLLMClient = nil
processValidationLLMClient = nil
})
var stdout, stderr bytes.Buffer
reportPath := filepath.Join(t.TempDir(), "report.json")
outputPath := filepath.Join(t.TempDir(), "out.json")
transcriptPath := writeFile(t, "transcript.json", `[
{"id":1,"speaker":"Alice","start":0.0,"end":1.0,"text":"Hello world"}
]`)
exitCode := Run([]string{
"process", transcriptPath,
"--glossary", fixturePath("tiny_glossary.yaml"),
"--modules", "grammar",
"--output", outputPath,
"--report-json", reportPath,
}, &stdout, &stderr)
if exitCode != 0 {
t.Fatalf("expected success, got %d stderr=%q", exitCode, stderr.String())
}
report := readProcessReport(t, reportPath)
if len(report.ModuleResults) != 1 {
t.Fatalf("expected one module result")
}
module := report.ModuleResults[0]
if len(module.ValidatorRejected) != 1 {
t.Fatalf("expected one validator rejection, got %+v", module.ValidatorRejected)
}
if len(module.SkippedChanges) != 1 {
t.Fatalf("expected one application skip, got %+v", module.SkippedChanges)
}
if module.ValidatorRejected[0].ReasonCode == string(module.SkippedChanges[0].SkipReason) {
t.Fatalf("validator rejection and application skip should remain distinct")
}
}
func TestRunProcessExplicitGrammarMalformedLLMOutputFailsWithErrorLog(t *testing.T) {
processProposalLLMClient = &fakeStructuredLLMClient{err: errors.New("malformed structured output")}
t.Cleanup(func() { processProposalLLMClient = nil })
var stdout, stderr bytes.Buffer
workDir := t.TempDir()
reportPath := filepath.Join(t.TempDir(), "report.json")
transcriptPath := writeFile(t, "transcript.json", `[
{"id":1,"speaker":"Alice","start":0.0,"end":1.0,"text":"hello"}
]`)
exitCode := Run([]string{
"process", transcriptPath,
"--glossary", fixturePath("tiny_glossary.yaml"),
"--modules", "grammar",
"--work-dir", workDir,
"--work-dir-retention", "always",
"--report-json", reportPath,
}, &stdout, &stderr)
if exitCode == 0 {
t.Fatal("expected failure")
}
if stdout.Len() != 0 {
t.Fatalf("expected empty stdout on failure, got %q", stdout.String())
}
if !strings.Contains(stderr.String(), "runner_execution") {
t.Fatalf("expected runner_execution error, got %q", stderr.String())
}
runDir := onlyRunDir(t, workDir)
if _, err := os.Stat(filepath.Join(runDir, "error.log")); err != nil {
t.Fatalf("expected error.log on failed grammar run: %v", err)
}
report := readProcessReport(t, reportPath)
if report.Status != "failed" || report.ErrorPhase != "runner_execution" {
t.Fatalf("expected failed runner_execution report, got %+v", report)
}
}
func TestRunProcessDefaultBehaviorRemainsDeterministicWithoutExplicitModules(t *testing.T) {
var stdout, stderr bytes.Buffer
reportPath := filepath.Join(t.TempDir(), "report.json")
outputPath := filepath.Join(t.TempDir(), "out.json")
transcriptPath := writeFile(t, "transcript.json", `[
{"id":1,"speaker":"Alice","start":0.0,"end":1.0,"text":"hello ,world"}
]`)
exitCode := Run([]string{
"process", transcriptPath,
"--glossary", fixturePath("tiny_glossary.yaml"),
"--output", outputPath,
"--report-json", reportPath,
}, &stdout, &stderr)
if exitCode != 0 {
t.Fatalf("expected success, got %d stderr=%q", exitCode, stderr.String())
}
report := readProcessReport(t, reportPath)
if report.ModulesSummary != nil || len(report.ModuleResults) != 0 {
t.Fatalf("expected no module execution without explicit --modules, got summary=%+v results=%+v", report.ModulesSummary, report.ModuleResults)
}
}
func TestRunProcessChunkingSummaryArtifactWritten(t *testing.T) { func TestRunProcessChunkingSummaryArtifactWritten(t *testing.T) {
var stdout bytes.Buffer var stdout bytes.Buffer
var stderr bytes.Buffer var stderr bytes.Buffer

View File

@@ -11,7 +11,7 @@ import (
func TestProcessReportModuleResultsJSONSuccessAndSkipped(t *testing.T) { func TestProcessReportModuleResultsJSONSuccessAndSkipped(t *testing.T) {
now := time.Now().UTC() now := time.Now().UTC()
report := ProcessReport{ report := ProcessReport{
Phase: "phase11-proposal-generation-framework", Phase: "phase12-grammar-module",
Status: "success", Status: "success",
ModuleResults: []ModuleReport{ ModuleResults: []ModuleReport{
{ {
@@ -80,7 +80,7 @@ func TestProcessReportModuleResultsJSONSuccessAndSkipped(t *testing.T) {
func TestProcessReportModuleResultsJSONFailedModule(t *testing.T) { func TestProcessReportModuleResultsJSONFailedModule(t *testing.T) {
now := time.Now().UTC() now := time.Now().UTC()
report := ProcessReport{ report := ProcessReport{
Phase: "phase11-proposal-generation-framework", Phase: "phase12-grammar-module",
Status: "failed", Status: "failed",
ModuleResults: []ModuleReport{ ModuleResults: []ModuleReport{
{ {

View File

@@ -8,6 +8,7 @@ import (
"gitea.maximumdirect.net/eric/audita/internal/core/config" "gitea.maximumdirect.net/eric/audita/internal/core/config"
"gitea.maximumdirect.net/eric/audita/internal/core/schema" "gitea.maximumdirect.net/eric/audita/internal/core/schema"
"gitea.maximumdirect.net/eric/audita/internal/framework/contracts" "gitea.maximumdirect.net/eric/audita/internal/framework/contracts"
grammarmodule "gitea.maximumdirect.net/eric/audita/internal/modules/grammar"
) )
const ( const (
@@ -65,10 +66,12 @@ type Factory struct {
// NewFactory creates a production registry scaffold with known module keys but // NewFactory creates a production registry scaffold with known module keys but
// no real module constructors registered yet. // no real module constructors registered yet.
func NewFactory(deps Dependencies) *Factory { func NewFactory(deps Dependencies) *Factory {
return &Factory{ factory := &Factory{
deps: deps, deps: deps,
constructors: make(map[string]Constructor, len(knownModuleKeys)), constructors: make(map[string]Constructor, len(knownModuleKeys)),
} }
_ = factory.RegisterConstructor(ModuleKeyGrammar, constructGrammarModule)
return factory
} }
// RegisterConstructor registers a constructor for a known module key. // RegisterConstructor registers a constructor for a known module key.
@@ -87,6 +90,12 @@ func (f *Factory) RegisterConstructor(moduleKey string, constructor Constructor)
return nil return nil
} }
func constructGrammarModule(ctx context.Context, req ConstructRequest) (contracts.TranscriptModule, error) {
_ = ctx
_ = req
return grammarmodule.New()
}
// ModuleForSpec resolves one configured run spec into a module instance. // ModuleForSpec resolves one configured run spec into a module instance.
func (f *Factory) ModuleForSpec(spec contracts.ModuleRunSpec) (contracts.TranscriptModule, error) { func (f *Factory) ModuleForSpec(spec contracts.ModuleRunSpec) (contracts.TranscriptModule, error) {
if f == nil { if f == nil {

View File

@@ -66,7 +66,7 @@ func TestUnsupportedUnknownModuleKeyFailsCleanly(t *testing.T) {
func TestRecognizedButUnimplementedModuleKeyFailsCleanly(t *testing.T) { func TestRecognizedButUnimplementedModuleKeyFailsCleanly(t *testing.T) {
factory := NewFactory(Dependencies{}) factory := NewFactory(Dependencies{})
for _, key := range []string{ModuleKeyGlossary, ModuleKeyHomophones, ModuleKeySpokenWord, ModuleKeyGrammar} { for _, key := range []string{ModuleKeyGlossary, ModuleKeyHomophones, ModuleKeySpokenWord} {
t.Run(key, func(t *testing.T) { t.Run(key, func(t *testing.T) {
_, err := factory.ModuleForSpec(contracts.ModuleRunSpec{ModuleKey: key, InstanceName: key}) _, err := factory.ModuleForSpec(contracts.ModuleRunSpec{ModuleKey: key, InstanceName: key})
if err == nil { if err == nil {
@@ -84,6 +84,17 @@ func TestRecognizedButUnimplementedModuleKeyFailsCleanly(t *testing.T) {
} }
} }
func TestGrammarIsRegisteredAndConstructibleByDefault(t *testing.T) {
factory := NewFactory(Dependencies{})
module, err := factory.ModuleForSpec(contracts.ModuleRunSpec{ModuleKey: ModuleKeyGrammar, InstanceName: ModuleKeyGrammar})
if err != nil {
t.Fatalf("ModuleForSpec error: %v", err)
}
if module.Key() != ModuleKeyGrammar {
t.Fatalf("expected grammar module key, got %q", module.Key())
}
}
func TestRegisterConstructorAndConstruct(t *testing.T) { func TestRegisterConstructorAndConstruct(t *testing.T) {
factory := NewFactory(Dependencies{}) factory := NewFactory(Dependencies{})
if err := factory.RegisterConstructor(ModuleKeyGlossary, func(ctx context.Context, req ConstructRequest) (contracts.TranscriptModule, error) { if err := factory.RegisterConstructor(ModuleKeyGlossary, func(ctx context.Context, req ConstructRequest) (contracts.TranscriptModule, error) {

View File

@@ -54,6 +54,7 @@ type Request struct {
Config *config.Config `json:"-"` Config *config.Config `json:"-"`
Messages []contracts.LLMMessage `json:"messages"` Messages []contracts.LLMMessage `json:"messages"`
Model string `json:"model,omitempty"` Model string `json:"model,omitempty"`
StageName string `json:"stage_name,omitempty"`
StartIndex int `json:"start_index"` StartIndex int `json:"start_index"`
LLMClient contracts.StructuredLLMClient LLMClient contracts.StructuredLLMClient
Scheduler contracts.LLMScheduler Scheduler contracts.LLMScheduler
@@ -87,7 +88,10 @@ func GenerateCandidates(ctx context.Context, req Request) (Result, error) {
return Result{}, fmt.Errorf("messages must not be empty") return Result{}, fmt.Errorf("messages must not be empty")
} }
stage := buildStageName(req.ModuleInstance, req.Section) stage := strings.TrimSpace(req.StageName)
if stage == "" {
stage = buildStageName(req.ModuleInstance, req.Section)
}
model := resolveModel(req.Config, req.Model) model := resolveModel(req.Config, req.Model)
messages := append([]contracts.LLMMessage(nil), req.Messages...) messages := append([]contracts.LLMMessage(nil), req.Messages...)

View File

@@ -12,6 +12,7 @@ import (
"gitea.maximumdirect.net/eric/audita/internal/core/schema" "gitea.maximumdirect.net/eric/audita/internal/core/schema"
"gitea.maximumdirect.net/eric/audita/internal/framework/contracts" "gitea.maximumdirect.net/eric/audita/internal/framework/contracts"
"gitea.maximumdirect.net/eric/audita/internal/framework/llm" "gitea.maximumdirect.net/eric/audita/internal/framework/llm"
"gitea.maximumdirect.net/eric/audita/internal/framework/modules"
"gitea.maximumdirect.net/eric/audita/internal/framework/proposal_generation" "gitea.maximumdirect.net/eric/audita/internal/framework/proposal_generation"
"gitea.maximumdirect.net/eric/audita/internal/framework/proposals" "gitea.maximumdirect.net/eric/audita/internal/framework/proposals"
"gitea.maximumdirect.net/eric/audita/internal/framework/validators" "gitea.maximumdirect.net/eric/audita/internal/framework/validators"
@@ -651,3 +652,37 @@ func TestRunnerProposalGenerationHelperFlowsThroughPipeline(t *testing.T) {
t.Fatalf("expected proposal-generation diagnostics artifacts in %s", filepath.Join(diagDir, "m")) t.Fatalf("expected proposal-generation diagnostics artifacts in %s", filepath.Join(diagDir, "m"))
} }
} }
func TestRunnerGrammarModuleUsesConfidenceThreshold(t *testing.T) {
client := &fakeProposalStructuredClient{
responses: []proposal_generation.StructuredCorrectionSet{
{
Corrections: []proposal_generation.StructuredCorrectionProposal{
{TargetSegmentID: 1, OriginalText: "hello ,world", CorrectedText: "Hello, world", Confidence: 0.5},
},
},
},
}
cfg := config.Default()
cfg.Thresholds.Grammar = 0.9
factory := modules.NewFactory(modules.Dependencies{})
out, err := New(factory).Run(context.Background(), RunInput{
Config: &cfg,
Transcript: &schema.Transcript{Segments: []schema.Segment{{ID: 1, Text: "hello ,world"}}},
Glossary: &schema.Glossary{},
ModuleSpecs: []contracts.ModuleRunSpec{{ModuleKey: "grammar", InstanceName: "grammar"}},
ProposalLLMClient: client,
})
if err != nil {
t.Fatalf("Run error: %v", err)
}
if out.FinalTranscript.Segments[0].Text != "hello ,world" {
t.Fatalf("expected no changes due to confidence threshold, got %q", out.FinalTranscript.Segments[0].Text)
}
if len(out.ModuleResults) != 1 || len(out.ModuleResults[0].ValidatorRejected) == 0 {
t.Fatalf("expected validator rejection, got %+v", out.ModuleResults)
}
if out.ModuleResults[0].ValidatorRejected[0].ReasonCode != validators.ReasonLowConfidence {
t.Fatalf("expected low confidence reason, got %+v", out.ModuleResults[0].ValidatorRejected[0])
}
}

View File

@@ -0,0 +1,76 @@
package grammar
import (
"context"
"fmt"
"gitea.maximumdirect.net/eric/audita/internal/framework/contracts"
"gitea.maximumdirect.net/eric/audita/internal/framework/proposal_generation"
"gitea.maximumdirect.net/eric/audita/internal/framework/proposals"
"gitea.maximumdirect.net/eric/audita/internal/framework/validators"
)
type Module struct {
validators []contracts.Validator
}
func New() (*Module, error) {
grammarOnlyGuard, err := validators.NewLLMBackedValidator("grammar_only_guard", validators.LLMValidatorTypeGrammarReview, "")
if err != nil {
return nil, err
}
meaningReversal, err := validators.NewLLMBackedValidator("meaning_reversal_review", validators.LLMValidatorTypeMeaningReversal, "")
if err != nil {
return nil, err
}
return &Module{
validators: []contracts.Validator{
validators.NoEffectValidator{},
validators.OriginalTextPresenceValidator{},
validators.ConfidenceThresholdValidator{},
validators.ProtectedGlossaryTermValidator{},
validators.NonEmptyCorrectionValidator{},
grammarOnlyGuard,
meaningReversal,
},
}, nil
}
func (m *Module) Key() string { return "grammar" }
func (m *Module) ReplacementPolicy() proposals.ReplacementPolicy {
// Python grammar module uses require_unique for conservative single-span replacement.
return proposals.ReplacementPolicyRequireUnique
}
func (m *Module) Validators() []contracts.Validator {
return append([]contracts.Validator(nil), m.validators...)
}
func (m *Module) Propose(ctx context.Context, req contracts.ProposalRequest) ([]proposals.CorrectionProposal, error) {
messages, err := BuildProposalMessages(req.WorkingTranscript, req.Glossary)
if err != nil {
return nil, err
}
generated, err := proposal_generation.GenerateCandidates(ctx, proposal_generation.Request{
ModuleKey: req.RunSpec.ModuleKey,
ModuleInstance: req.RunSpec.InstanceName,
ReplacementPolicy: req.RunSpec.ReplacementPolicy,
WorkingTranscript: req.WorkingTranscript,
Section: req.Section,
Glossary: req.Glossary,
Config: req.Config,
Messages: messages,
StageName: fmt.Sprintf("%s:proposal", req.RunSpec.InstanceName),
StartIndex: 0,
LLMClient: req.LLMClient,
Scheduler: req.LLMScheduler,
DiagnosticsDir: req.DiagnosticsDir,
})
if err != nil {
return nil, err
}
return generated.Corrections, nil
}

View File

@@ -0,0 +1,194 @@
package grammar
import (
"context"
"errors"
"os"
"path/filepath"
"strings"
"testing"
"gitea.maximumdirect.net/eric/audita/internal/core/config"
"gitea.maximumdirect.net/eric/audita/internal/core/schema"
"gitea.maximumdirect.net/eric/audita/internal/framework/contracts"
"gitea.maximumdirect.net/eric/audita/internal/framework/proposal_generation"
"gitea.maximumdirect.net/eric/audita/internal/framework/proposals"
)
type fakeLLMClient struct {
responses []map[string]any
err error
calls []contracts.StructuredCompletionRequest
}
func (f *fakeLLMClient) CompleteStructured(ctx context.Context, req contracts.StructuredCompletionRequest, out any) (contracts.StructuredCompletionResponse, error) {
_ = ctx
f.calls = append(f.calls, req)
if f.err != nil {
return contracts.StructuredCompletionResponse{}, f.err
}
if len(f.responses) == 0 {
return contracts.StructuredCompletionResponse{}, errors.New("unexpected call")
}
payload := f.responses[0]
f.responses = f.responses[1:]
target, ok := out.(*proposal_generation.StructuredCorrectionSet)
if !ok {
return contracts.StructuredCompletionResponse{}, errors.New("unexpected output type")
}
items, _ := payload["corrections"].([]map[string]any)
for _, item := range items {
target.Corrections = append(target.Corrections, proposal_generation.StructuredCorrectionProposal{
TargetSegmentID: item["id"].(int),
OriginalText: item["original_text"].(string),
CorrectedText: item["corrected_text"].(string),
Confidence: item["confidence"].(float64),
})
}
return contracts.StructuredCompletionResponse{}, nil
}
type countingScheduler struct {
runs int
}
func (s *countingScheduler) Run(ctx context.Context, fn func(context.Context) error) error {
s.runs++
return fn(ctx)
}
func tinyTranscript() *schema.Transcript {
return &schema.Transcript{Segments: []schema.Segment{
{ID: 1, Speaker: "A", Start: 0, End: 1, Text: "hello ,world"},
}}
}
func tinyGlossary() *schema.Glossary {
return &schema.Glossary{Entries: []schema.GlossaryEntry{
{Name: "Jesters", Category: "faction", Summary: "Faction", Aliases: []string{"Jester"}},
}}
}
func TestBuildProposalMessagesConstraints(t *testing.T) {
msgs, err := BuildProposalMessages(tinyTranscript(), tinyGlossary())
if err != nil {
t.Fatalf("BuildProposalMessages error: %v", err)
}
if len(msgs) != 2 {
t.Fatalf("expected 2 messages, got %d", len(msgs))
}
combined := msgs[0].Content + "\n" + msgs[1].Content
for _, want := range []string{
"Transcript section:",
"Protected glossary/context:",
"punctuation, capitalization, spacing, and article cleanup only",
"Do not make word substitutions",
"Do not return speaker, start, or end fields",
`"id": 1`,
} {
if !strings.Contains(combined, want) {
t.Fatalf("expected prompt to contain %q", want)
}
}
if strings.Contains(strings.ToLower(combined), "summarize") {
t.Fatalf("prompt should not invite summarization")
}
}
func TestGrammarModuleReplacementPolicy(t *testing.T) {
m, err := New()
if err != nil {
t.Fatalf("New error: %v", err)
}
if m.ReplacementPolicy() != proposals.ReplacementPolicyRequireUnique {
t.Fatalf("unexpected replacement policy: %q", m.ReplacementPolicy())
}
}
func TestGrammarModuleValidatorChain(t *testing.T) {
m, err := New()
if err != nil {
t.Fatalf("New error: %v", err)
}
got := make([]string, 0, len(m.Validators()))
for _, v := range m.Validators() {
got = append(got, v.Name())
}
want := []string{
"no_effect",
"original_text_presence",
"confidence_threshold",
"protected_glossary_terms",
"non_empty_correction",
"grammar_only_guard",
"meaning_reversal_review",
}
if strings.Join(got, ",") != strings.Join(want, ",") {
t.Fatalf("unexpected validator chain\n got: %v\nwant: %v", got, want)
}
}
func TestGrammarModuleProposeMapsCorrectionsAndWritesDiagnostics(t *testing.T) {
secret := "super-secret-key"
client := &fakeLLMClient{
responses: []map[string]any{
{
"corrections": []map[string]any{
{"id": 1, "original_text": "hello ,world", "corrected_text": "Hello, world", "confidence": 0.93},
},
},
},
}
scheduler := &countingScheduler{}
cfg := config.Default()
cfg.PrimaryLLM.APIKey = secret
module, err := New()
if err != nil {
t.Fatalf("New error: %v", err)
}
diagDir := t.TempDir()
out, err := module.Propose(context.Background(), contracts.ProposalRequest{
ExecutionContext: contracts.ExecutionContext{
Config: &cfg,
WorkingTranscript: tinyTranscript(),
Glossary: tinyGlossary(),
DiagnosticsDir: diagDir,
},
RunSpec: contracts.ModuleRunSpec{
ModuleKey: "grammar",
InstanceName: "grammar",
ReplacementPolicy: proposals.ReplacementPolicyRequireUnique,
},
LLMClient: client,
LLMScheduler: scheduler,
})
if err != nil {
t.Fatalf("Propose error: %v", err)
}
if scheduler.runs != 1 {
t.Fatalf("expected scheduler run count 1, got %d", scheduler.runs)
}
if len(client.calls) != 1 || client.calls[0].StageName != "grammar:proposal" {
t.Fatalf("expected one grammar:proposal call, got %+v", client.calls)
}
if len(out) != 1 || out[0].CorrectedText != "Hello, world" {
t.Fatalf("unexpected proposals: %+v", out)
}
diagFiles, globErr := filepath.Glob(filepath.Join(diagDir, "grammar", "*proposal*response-payload.json"))
if globErr != nil {
t.Fatalf("glob error: %v", globErr)
}
if len(diagFiles) == 0 {
t.Fatalf("expected diagnostics response payload under %s", filepath.Join(diagDir, "grammar"))
}
for _, f := range diagFiles {
raw, readErr := os.ReadFile(f)
if readErr != nil {
t.Fatalf("read diag %q: %v", f, readErr)
}
if strings.Contains(string(raw), secret) {
t.Fatalf("secret leaked in diagnostics: %s", string(raw))
}
}
}

View File

@@ -0,0 +1,85 @@
package grammar
import (
"encoding/json"
"fmt"
"gitea.maximumdirect.net/eric/audita/internal/core/schema"
"gitea.maximumdirect.net/eric/audita/internal/framework/contracts"
)
type promptSegment struct {
ID int `json:"id"`
Speaker string `json:"speaker"`
Start float64 `json:"start"`
End float64 `json:"end"`
Text string `json:"text"`
Categories []string `json:"categories,omitempty"`
}
type promptTranscriptSection struct {
SectionIndex int `json:"section_index"`
Segments []promptSegment `json:"segments"`
}
// BuildProposalMessages mirrors the Python grammar-module prompt intent:
// punctuation/capitalization/spacing cleanup only, with strict meaning guards.
func BuildProposalMessages(transcript *schema.Transcript, glossary *schema.Glossary) ([]contracts.LLMMessage, error) {
glossaryJSON, err := json.MarshalIndent(glossary, "", " ")
if err != nil {
return nil, fmt.Errorf("marshal glossary prompt context: %w", err)
}
sectionPayload := promptTranscriptSection{
SectionIndex: 0,
Segments: make([]promptSegment, 0),
}
if transcript != nil {
for _, s := range transcript.Segments {
sectionPayload.Segments = append(sectionPayload.Segments, promptSegment{
ID: s.ID,
Speaker: s.Speaker,
Start: s.Start,
End: s.End,
Text: s.Text,
Categories: append([]string(nil), s.Categories...),
})
}
}
sectionJSON, err := json.MarshalIndent(sectionPayload, "", " ")
if err != nil {
return nil, fmt.Errorf("marshal transcript prompt context: %w", err)
}
system := "You are Audita, a conservative grammar cleanup assistant. Identify only punctuation, capitalization, and spacing cleanup that preserves the same underlying words. Do not change content, substitute words, or rewrite the speaker's phrasing."
user := "Review this transcript section and return only grammar cleanup corrections that should be applied.\n\n" +
"Rules:\n" +
"- Allowed changes are punctuation, capitalization, spacing, and article cleanup only.\n" +
"- You may add, remove, or adjust commas, periods, quotation marks, apostrophes, dashes, ellipses, spacing, and capitalization when the underlying words stay the same.\n" +
"- You may change the whole-word article \"a\" to \"an\" or \"an\" to \"a\" when the surrounding text otherwise stays the same.\n" +
"- Homophone, spoken-form, and mistranscription corrections are handled during a later review stage; do not propose them here.\n" +
"- Do not make word substitutions, spelling fixes, homophone fixes, filler cleanup, repetition cleanup, paraphrases, or other semantic rewrites.\n" +
"- Do not change one written word into a different written word, except for capitalization changes to the same letters.\n" +
"- If a possible correction depends on changing a content word into a different word, omit it here rather than bundling it together with formatting cleanup.\n" +
"- Treat glossary names and aliases as protected spellings and context.\n" +
"- Do not replace, Anglicize, normalize, lowercase, or otherwise alter protected glossary names or aliases away from their glossary spelling.\n" +
"- Preserve canonical glossary capitalization for protected names and aliases, even if they look unusual.\n" +
"- If a segment includes categories, treat them as additional transcript context.\n" +
"- Use the exact id from the input segment.\n" +
"- For returned corrections, original_text must be only the exact text span that needs replacement, not the full segment text unless the whole segment is the replacement span.\n" +
"- Choose an original_text span that appears exactly once in the current segment text.\n" +
"- corrected_text must be only the replacement text for that span, not the full corrected segment text unless the whole segment is the replacement span.\n" +
"- Each returned correction must contain only id, original_text, corrected_text, and confidence.\n" +
"- Do not return corrections where original_text and corrected_text are identical.\n" +
"- Do not return speaker, start, or end fields.\n" +
"- Return only changed segments; do not return entries for unchanged segments.\n" +
"- confidence must be between 0.0 and 1.0.\n" +
"- If no corrections are needed, return an empty corrections list.\n\n" +
fmt.Sprintf("Protected glossary/context:\n%s\n\nTranscript section:\n%s", string(glossaryJSON), string(sectionJSON))
return []contracts.LLMMessage{
{Role: "system", Content: system},
{Role: "user", Content: user},
}, nil
}