diff --git a/docs/architecture.md b/docs/architecture.md index c92c3ab..dfba01c 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -43,7 +43,6 @@ Implemented today: - Explicit runtime support for `--modules spoken_word` through the production runner path. Not implemented in CLI runtime path today: -- Python parity fixture suite and parity verification workflow. - Operational hardening tasks beyond current runtime/reporting/diagnostics behavior. - Rollout and Python retirement work. @@ -66,7 +65,8 @@ Phase sequencing note: - Phase 14 homophones module implementation and explicit runtime wiring are complete; - Phase 15 spoken-word module implementation and explicit runtime wiring are complete; - Phase 16 default full pipeline integration is complete; -- next recommended phase is Phase 17 (Python parity fixture suite). +- Phase 17 parity fixture suite is complete; +- next recommended phase is Phase 18 (operational hardening and subprocess integration). ## Actual Go package layout @@ -184,6 +184,12 @@ Current runtime flow (`internal/cli/run.go`): 15. Optionally write `--report-json`; always write run-dir `report.json`. 16. Apply work-dir retention. +Parity fixture status: +- representative Python-parity fixture coverage exists under `internal/cli/testdata/parity`; +- parity tests use fake structured LLM responses for deterministic behavior, including default full-pipeline shape assertions; +- parity comparisons intentionally ignore nondeterministic metadata (timestamps, run IDs, temp paths, token usage) and remain strict for deterministic contract fields (transcript content, module order/instance naming, applied/skipped/rejected counts, and status). +- intentional Python-vs-Go differences and open parity gaps are documented in `docs/python-parity.md`. + Important behavior details: - Glossary is validated and is used for explicit glossary/grammar/homophones/spoken_word module correction paths. - Default production CLI behavior now executes the full production module sequence unless `--modules` override is supplied. @@ -254,6 +260,7 @@ Current caveat: Current runtime boundary: - the default CLI runtime path (without explicit module selection) instantiates the full production module sequence. - LLM calls are exercised in production in both default full-pipeline runs and explicit `--modules` runs, and in tests when fake/injected clients are used. +- normal `go test ./...` does not require real LLM credentials or Python dependencies. `internal/framework/llm` also provides: - a bounded `Scheduler` for controlled concurrent LLM calls with reliable permit release; diff --git a/docs/python-parity.md b/docs/python-parity.md new file mode 100644 index 0000000..74abdfa --- /dev/null +++ b/docs/python-parity.md @@ -0,0 +1,78 @@ +# Python vs Go Parity Notes + +This document tracks Phase 17 parity-fixture coverage and differences between the original Python implementation and the Go rewrite. + +## Scope + +- Uses deterministic fixture-driven tests in `internal/cli/testdata/parity`. +- Uses fake structured LLM responses for proposal generation and LLM validators. +- Verifies functional contract fields (module order, instance naming, applied/skipped counts, report status, diagnostics presence). +- Does not require real LLM credentials or Python dependencies during `go test ./...`. + +## Intentional Differences + +The following differences are expected and treated as intentional unless they break contract behavior: + +1. JSON formatting and field ordering +- Serialized JSON whitespace and object key order may differ. +- Parity tests compare JSON semantically, not byte-for-byte. + +2. Diagnostics path values +- Absolute run-directory paths, run IDs, and temp directory roots differ by runtime and platform. +- Parity checks assert artifact presence/shape, not exact absolute paths. + +3. Time-variant metadata +- Timestamps (`started_at`, `completed_at`) and generated run IDs are runtime-specific. +- Parity checks ignore exact timestamp/run-id values. + +4. Provider metadata/token accounting +- Provider/token usage metadata may vary by adapter behavior and is not asserted as strict parity fields. + +5. Internal adapter implementation details +- Go uses its own structured LLM adapter implementation details while preserving the same high-level contract semantics. + +## Current Fixture Coverage + +Current parity fixtures cover: + +- Transcript schema handling failure path. +- Glossary schema handling failure path. +- Default full module sequence shape: + - `glossary_1`, `homophones`, `glossary_2`, `spoken_word`, `grammar`. +- Mutable transcript handoff across default stages. +- Module-specific behavior inside the default sequence: + - glossary correction + - homophone-style correction + - spoken-word cleanup + - grammar cleanup +- Protected glossary-term guardrail behavior. +- Deterministic validator rejection behavior. +- LLM validator decision/rejection behavior. +- Application-level skip behavior (`ambiguous_original_text`). +- Mid-pipeline failure with partial progress preserved in reports. +- Diagnostics artifact presence and secret-redaction checks. + +## Open Parity Gaps (Not Intentional) + +These are known Phase 17 expansion opportunities and should not be labeled as intentional compatibility differences: + +1. Broader Python fixture import +- The current Go parity fixtures are native fixture cases; they do not yet ingest all existing Python test fixtures directly. + +2. Side-by-side runner command +- No repository-standard Python+Go side-by-side parity command is required or enforced yet. + +3. Wider transcript corpus +- Current fixtures are representative but not exhaustive across all transcript/glossary edge combinations. + +## How To Extend + +1. Add a new `*.case.json` file under `internal/cli/testdata/parity`. +2. Add referenced transcript/glossary/fake-LLM response files. +3. Encode deterministic expectations in the case: +- module order and instance names +- transcript output +- applied/skipped/rejected counts +- report status and failure metadata +- diagnostics artifact presence/redaction markers +4. Run `go test ./...`. diff --git a/docs/rewrite-notes.md b/docs/rewrite-notes.md index c88bce2..a4fe6fc 100644 --- a/docs/rewrite-notes.md +++ b/docs/rewrite-notes.md @@ -98,7 +98,6 @@ Implemented: - Generic JSON diagnostics primitives for LLM interactions (request metadata, request payload, response payload, optional error payload) with secret redaction. Not yet implemented in runtime pipeline: -- Python parity fixture suite and parity verification workflow. - Operational hardening beyond current Phase 16 runtime/reporting/diagnostics scope. - Rollout/Python retirement work. @@ -238,7 +237,7 @@ Not implemented in Phase 8 (by design): ## Remaining work plan -Next recommended phase: **Phase 17 (Python parity fixture suite)**. +Next recommended phase: **Phase 18 (operational hardening and subprocess integration)**. ## Phase 9: Structured LLM client and scheduler infrastructure @@ -527,6 +526,8 @@ Intentionally deferred: ## Phase 17: Python parity fixture suite +Completed. + ### Purpose Establish confidence that the Go implementation matches the behavior and safety posture of the initial Python implementation. @@ -566,6 +567,35 @@ The repository has a durable test suite demonstrating that the Go implementation - Unintentional compatibility breaks are fixed. - `go test ./...` passes. +### Phase 17 completion status + +Implemented: +- Durable parity fixture harness in Go test path (`internal/cli/parity_test.go`). +- Representative parity fixture corpus under `internal/cli/testdata/parity`. +- Fake LLM proposal/validator fixtures driving deterministic parity tests. +- Default full-pipeline parity coverage including sequence/order and repeated glossary naming: + - `glossary_1` + - `homophones` + - `glossary_2` + - `spoken_word` + - `grammar` +- Parity checks that are strict for deterministic contract fields: + - transcript content + - module order and instance names + - applied/skipped/rejected counts + - report status and failed-module metadata +- Parity checks that ignore nondeterministic metadata fields: + - timestamps + - run IDs + - temp/absolute paths + - provider token usage details +- Documentation of intentional Python-vs-Go differences and honest open parity gaps in `docs/python-parity.md`. +- Normal `go test ./...` path remains independent of real LLM credentials and Python dependencies. + +Intentionally deferred: +- Phase 18 operational hardening and subprocess integration expansion. +- Phase 19 rollout and Python retirement work. + ## Phase 18: Operational hardening and subprocess integration ### Purpose diff --git a/internal/cli/parity_test.go b/internal/cli/parity_test.go new file mode 100644 index 0000000..44a1d2d --- /dev/null +++ b/internal/cli/parity_test.go @@ -0,0 +1,338 @@ +package cli + +import ( + "bytes" + "encoding/json" + "os" + "path/filepath" + "reflect" + "strings" + "testing" + + "gitea.maximumdirect.net/eric/audita/internal/core/reporting" +) + +type parityFixtureCase struct { + Name string `json:"name"` + TranscriptFile string `json:"transcript_file"` + GlossaryFile string `json:"glossary_file"` + ModulesCSV string `json:"modules_csv,omitempty"` + ProposalResponsesFile string `json:"proposal_responses_file,omitempty"` + ValidationResponsesFile string `json:"validation_responses_file,omitempty"` + Env map[string]string `json:"env,omitempty"` + Expect parityExpectation `json:"expect"` +} + +type parityExpectation struct { + ExitCode int `json:"exit_code"` + Status string `json:"status,omitempty"` + ErrorPhase string `json:"error_phase,omitempty"` + StdoutMode string `json:"stdout_mode,omitempty"` // empty|json + StderrContains string `json:"stderr_contains,omitempty"` + OutputTranscriptFile string `json:"output_transcript_file,omitempty"` + ModuleInstances []string `json:"module_instances,omitempty"` + ModuleCount int `json:"module_count,omitempty"` + TotalAppliedChanges int `json:"total_applied_changes,omitempty"` + TotalSkippedChanges int `json:"total_skipped_changes,omitempty"` + FailedModuleInstance string `json:"failed_module_instance,omitempty"` + ValidatorRejectedReasonCodes []string `json:"validator_rejected_reason_codes,omitempty"` + ApplicationSkipReasonCodes []string `json:"application_skip_reason_codes,omitempty"` + RequireErrorLog bool `json:"require_error_log,omitempty"` + SecretMarkers []string `json:"secret_markers,omitempty"` + ExpectedProposalCalls []string `json:"expected_proposal_calls,omitempty"` + ExpectedValidationCalls []string `json:"expected_validation_calls,omitempty"` + ModuleAppliedCounts []int `json:"module_applied_counts,omitempty"` + ModuleRejectedCounts []int `json:"module_rejected_counts,omitempty"` + ModuleSkipCounts []int `json:"module_skip_counts,omitempty"` + MinResponsePayloadArtifacts int `json:"min_response_payload_artifacts,omitempty"` +} + +func TestParityFixtures(t *testing.T) { + casePaths, err := filepath.Glob(parityFixturePath("*.case.json")) + if err != nil { + t.Fatalf("glob parity fixtures: %v", err) + } + if len(casePaths) == 0 { + t.Fatal("expected at least one parity fixture case") + } + + for _, casePath := range casePaths { + fx := loadParityFixtureCase(t, casePath) + t.Run(fx.Name, func(t *testing.T) { + runParityFixtureCase(t, filepath.Dir(casePath), fx) + }) + } +} + +func loadParityFixtureCase(t *testing.T, casePath string) parityFixtureCase { + t.Helper() + var fx parityFixtureCase + raw := readFile(t, casePath) + if err := json.Unmarshal(raw, &fx); err != nil { + t.Fatalf("parse parity case %q: %v", casePath, err) + } + if strings.TrimSpace(fx.Name) == "" { + t.Fatalf("parity case %q missing name", casePath) + } + return fx +} + +func runParityFixtureCase(t *testing.T, caseDir string, fx parityFixtureCase) { + t.Helper() + for k, v := range fx.Env { + t.Setenv(k, v) + } + + proposalClient := &fakeStructuredLLMClient{} + validationClient := &fakeStructuredLLMClient{} + + if strings.TrimSpace(fx.ProposalResponsesFile) != "" { + raw := readFile(t, filepath.Join(caseDir, fx.ProposalResponsesFile)) + if err := json.Unmarshal(raw, &proposalClient.proposalResponses); err != nil { + t.Fatalf("parse proposal responses: %v", err) + } + processProposalLLMClient = proposalClient + } + if strings.TrimSpace(fx.ValidationResponsesFile) != "" { + raw := readFile(t, filepath.Join(caseDir, fx.ValidationResponsesFile)) + if err := json.Unmarshal(raw, &validationClient.validationResponses); err != nil { + t.Fatalf("parse validation responses: %v", err) + } + processValidationLLMClient = validationClient + } + t.Cleanup(func() { + processProposalLLMClient = nil + processValidationLLMClient = nil + }) + + var stdout, stderr bytes.Buffer + workDir := t.TempDir() + outputPath := filepath.Join(t.TempDir(), "out.json") + reportPath := filepath.Join(t.TempDir(), "report.json") + args := []string{ + "process", + filepath.Join(caseDir, fx.TranscriptFile), + "--glossary", + filepath.Join(caseDir, fx.GlossaryFile), + "--report-json", + reportPath, + "--work-dir", + workDir, + "--work-dir-retention", + "always", + } + // Keep stdout shape deterministic for parity tests. + if fx.Expect.StdoutMode != "json" { + args = append(args, "--output", outputPath) + } + if strings.TrimSpace(fx.ModulesCSV) != "" { + args = append(args, "--modules", fx.ModulesCSV) + } + + exitCode := Run(args, &stdout, &stderr) + if exitCode != fx.Expect.ExitCode { + t.Fatalf("expected exit code %d, got %d stderr=%q", fx.Expect.ExitCode, exitCode, stderr.String()) + } + + switch fx.Expect.StdoutMode { + case "json": + if _, err := json.Marshal(stdout.String()); err != nil { + t.Fatalf("unexpected stdout marshal error: %v", err) + } + if !json.Valid(stdout.Bytes()) { + t.Fatalf("expected JSON stdout, got %q", stdout.String()) + } + default: + if stdout.Len() != 0 { + t.Fatalf("expected empty stdout, got %q", stdout.String()) + } + } + + if fx.Expect.StderrContains != "" && !strings.Contains(stderr.String(), fx.Expect.StderrContains) { + t.Fatalf("expected stderr to contain %q, got %q", fx.Expect.StderrContains, stderr.String()) + } + + report := readProcessReport(t, reportPath) + assertParityReport(t, report, fx.Expect) + + runDir := onlyRunDir(t, workDir) + runDirReport := readProcessReport(t, filepath.Join(runDir, "report.json")) + assertParityReport(t, runDirReport, fx.Expect) + + if fx.Expect.RequireErrorLog { + if _, err := os.Stat(filepath.Join(runDir, "error.log")); err != nil { + t.Fatalf("expected error.log: %v", err) + } + } + + if strings.TrimSpace(fx.Expect.OutputTranscriptFile) != "" && fx.Expect.ExitCode == 0 { + got := readFile(t, outputPath) + want := readFile(t, filepath.Join(caseDir, fx.Expect.OutputTranscriptFile)) + assertJSONSemanticEqual(t, want, got) + } + + if len(fx.Expect.ExpectedProposalCalls) > 0 && !reflect.DeepEqual(proposalClient.calls, fx.Expect.ExpectedProposalCalls) { + t.Fatalf("unexpected proposal calls: got %v want %v", proposalClient.calls, fx.Expect.ExpectedProposalCalls) + } + if len(fx.Expect.ExpectedValidationCalls) > 0 && !reflect.DeepEqual(validationClient.calls, fx.Expect.ExpectedValidationCalls) { + t.Fatalf("unexpected validation calls: got %v want %v", validationClient.calls, fx.Expect.ExpectedValidationCalls) + } + + if len(fx.Expect.SecretMarkers) > 0 { + assertNoSecretMarkers(t, reportPath, fx.Expect.SecretMarkers) + assertNoSecretMarkersInTree(t, runDir, fx.Expect.SecretMarkers) + } + if fx.Expect.MinResponsePayloadArtifacts > 0 { + matches, err := filepath.Glob(filepath.Join(runDir, "*", "*response-payload.json")) + if err != nil { + t.Fatalf("glob response payload artifacts: %v", err) + } + if len(matches) < fx.Expect.MinResponsePayloadArtifacts { + t.Fatalf("expected at least %d response payload artifacts, got %d", fx.Expect.MinResponsePayloadArtifacts, len(matches)) + } + } +} + +func assertParityReport(t *testing.T, report reporting.ProcessReport, exp parityExpectation) { + t.Helper() + if exp.Status != "" && report.Status != exp.Status { + t.Fatalf("expected report status %q, got %q", exp.Status, report.Status) + } + if exp.ErrorPhase != "" && report.ErrorPhase != exp.ErrorPhase { + t.Fatalf("expected report error_phase %q, got %q", exp.ErrorPhase, report.ErrorPhase) + } + if len(exp.ModuleInstances) > 0 { + got := make([]string, 0, len(report.ModuleResults)) + for _, mr := range report.ModuleResults { + got = append(got, mr.ModuleInstance) + } + if !reflect.DeepEqual(got, exp.ModuleInstances) { + t.Fatalf("unexpected module instances: got %v want %v", got, exp.ModuleInstances) + } + } + if exp.ModuleCount > 0 { + if report.ModulesSummary == nil || report.ModulesSummary.ModuleCount != exp.ModuleCount { + t.Fatalf("expected module_count=%d, got %+v", exp.ModuleCount, report.ModulesSummary) + } + } + if exp.TotalAppliedChanges > 0 { + if report.ModulesSummary == nil || report.ModulesSummary.TotalAppliedChanges != exp.TotalAppliedChanges { + t.Fatalf("expected total_applied_changes=%d, got %+v", exp.TotalAppliedChanges, report.ModulesSummary) + } + } + if exp.TotalSkippedChanges > 0 { + if report.ModulesSummary == nil || report.ModulesSummary.TotalSkippedChanges != exp.TotalSkippedChanges { + t.Fatalf("expected total_skipped_changes=%d, got %+v", exp.TotalSkippedChanges, report.ModulesSummary) + } + } + if exp.FailedModuleInstance != "" { + if report.ModulesSummary == nil || report.ModulesSummary.FailedModuleInstance != exp.FailedModuleInstance { + t.Fatalf("expected failed_module_instance=%q, got %+v", exp.FailedModuleInstance, report.ModulesSummary) + } + } + + if len(exp.ValidatorRejectedReasonCodes) > 0 { + got := collectValidatorRejectedReasonCodes(report.ModuleResults) + if !reflect.DeepEqual(got, exp.ValidatorRejectedReasonCodes) { + t.Fatalf("unexpected validator rejected reason codes: got %v want %v", got, exp.ValidatorRejectedReasonCodes) + } + } + if len(exp.ApplicationSkipReasonCodes) > 0 { + got := collectApplicationSkipReasonCodes(report.ModuleResults) + if !reflect.DeepEqual(got, exp.ApplicationSkipReasonCodes) { + t.Fatalf("unexpected application skip reason codes: got %v want %v", got, exp.ApplicationSkipReasonCodes) + } + } + if len(exp.ModuleAppliedCounts) > 0 { + got := make([]int, 0, len(report.ModuleResults)) + for _, mr := range report.ModuleResults { + got = append(got, len(mr.AppliedChanges)) + } + if !reflect.DeepEqual(got, exp.ModuleAppliedCounts) { + t.Fatalf("unexpected per-module applied counts: got %v want %v", got, exp.ModuleAppliedCounts) + } + } + if len(exp.ModuleRejectedCounts) > 0 { + got := make([]int, 0, len(report.ModuleResults)) + for _, mr := range report.ModuleResults { + got = append(got, len(mr.ValidatorRejected)) + } + if !reflect.DeepEqual(got, exp.ModuleRejectedCounts) { + t.Fatalf("unexpected per-module rejected counts: got %v want %v", got, exp.ModuleRejectedCounts) + } + } + if len(exp.ModuleSkipCounts) > 0 { + got := make([]int, 0, len(report.ModuleResults)) + for _, mr := range report.ModuleResults { + got = append(got, len(mr.SkippedChanges)) + } + if !reflect.DeepEqual(got, exp.ModuleSkipCounts) { + t.Fatalf("unexpected per-module skip counts: got %v want %v", got, exp.ModuleSkipCounts) + } + } +} + +func collectValidatorRejectedReasonCodes(results []reporting.ModuleReport) []string { + out := make([]string, 0) + for _, mr := range results { + for _, vr := range mr.ValidatorRejected { + out = append(out, vr.ReasonCode) + } + } + return out +} + +func collectApplicationSkipReasonCodes(results []reporting.ModuleReport) []string { + out := make([]string, 0) + for _, mr := range results { + for _, sk := range mr.SkippedChanges { + out = append(out, string(sk.SkipReason)) + } + } + return out +} + +func assertJSONSemanticEqual(t *testing.T, expected []byte, actual []byte) { + t.Helper() + var exp any + var act any + if err := json.Unmarshal(expected, &exp); err != nil { + t.Fatalf("unmarshal expected json: %v", err) + } + if err := json.Unmarshal(actual, &act); err != nil { + t.Fatalf("unmarshal actual json: %v", err) + } + if !reflect.DeepEqual(exp, act) { + t.Fatalf("JSON mismatch\nexpected=%s\nactual=%s", string(expected), string(actual)) + } +} + +func assertNoSecretMarkers(t *testing.T, filePath string, markers []string) { + t.Helper() + raw := string(readFile(t, filePath)) + for _, marker := range markers { + if marker != "" && strings.Contains(raw, marker) { + t.Fatalf("secret marker %q leaked in %s", marker, filePath) + } + } +} + +func assertNoSecretMarkersInTree(t *testing.T, root string, markers []string) { + t.Helper() + _ = filepath.WalkDir(root, func(path string, d os.DirEntry, err error) error { + if err != nil || d == nil || d.IsDir() { + return nil + } + raw := string(readFile(t, path)) + for _, marker := range markers { + if marker != "" && strings.Contains(raw, marker) { + t.Fatalf("secret marker %q leaked in %s", marker, path) + } + } + return nil + }) +} + +func parityFixturePath(name string) string { + return filepath.Join("testdata", "parity", name) +} diff --git a/internal/cli/testdata/parity/application-skip-ambiguous.case.json b/internal/cli/testdata/parity/application-skip-ambiguous.case.json new file mode 100644 index 0000000..c8b0720 --- /dev/null +++ b/internal/cli/testdata/parity/application-skip-ambiguous.case.json @@ -0,0 +1,22 @@ +{ + "name": "ambiguous_match_rejected_before_application", + "transcript_file": "application-skip-ambiguous.transcript.json", + "glossary_file": "default-handoff.glossary.yaml", + "modules_csv": "homophones", + "proposal_responses_file": "application-skip-ambiguous.proposals.json", + "validation_responses_file": "application-skip-ambiguous.validations.json", + "expect": { + "exit_code": 0, + "status": "success", + "output_transcript_file": "application-skip-ambiguous.expected-transcript.json", + "module_instances": ["homophones"], + "module_count": 1, + "total_applied_changes": 0, + "total_skipped_changes": 1, + "module_applied_counts": [0], + "module_rejected_counts": [1], + "module_skip_counts": [0], + "validator_rejected_reason_codes": ["ambiguous_original_text"], + "expected_proposal_calls": ["homophones:proposal"] + } +} diff --git a/internal/cli/testdata/parity/application-skip-ambiguous.expected-transcript.json b/internal/cli/testdata/parity/application-skip-ambiguous.expected-transcript.json new file mode 100644 index 0000000..2eb4292 --- /dev/null +++ b/internal/cli/testdata/parity/application-skip-ambiguous.expected-transcript.json @@ -0,0 +1,3 @@ +[ + {"id":1,"speaker":"Alice","start":0,"end":1,"text":"the site near another site"} +] diff --git a/internal/cli/testdata/parity/application-skip-ambiguous.proposals.json b/internal/cli/testdata/parity/application-skip-ambiguous.proposals.json new file mode 100644 index 0000000..faace9c --- /dev/null +++ b/internal/cli/testdata/parity/application-skip-ambiguous.proposals.json @@ -0,0 +1,3 @@ +[ + {"corrections": [{"id": 1, "original_text": "site", "corrected_text": "sight", "confidence": 0.99}]} +] diff --git a/internal/cli/testdata/parity/application-skip-ambiguous.transcript.json b/internal/cli/testdata/parity/application-skip-ambiguous.transcript.json new file mode 100644 index 0000000..263aa37 --- /dev/null +++ b/internal/cli/testdata/parity/application-skip-ambiguous.transcript.json @@ -0,0 +1,3 @@ +[ + {"id":1,"speaker":"Alice","start":0.0,"end":1.0,"text":"the site near another site"} +] diff --git a/internal/cli/testdata/parity/application-skip-ambiguous.validations.json b/internal/cli/testdata/parity/application-skip-ambiguous.validations.json new file mode 100644 index 0000000..17d0059 --- /dev/null +++ b/internal/cli/testdata/parity/application-skip-ambiguous.validations.json @@ -0,0 +1,4 @@ +[ + {"validations": [{"correction_index": 0, "approved": true, "confidence": 0.99, "reason": "ok"}]}, + {"validations": [{"correction_index": 0, "approved": true, "confidence": 0.99, "reason": "ok"}]} +] diff --git a/internal/cli/testdata/parity/default-full-pipeline.case.json b/internal/cli/testdata/parity/default-full-pipeline.case.json new file mode 100644 index 0000000..a1519e6 --- /dev/null +++ b/internal/cli/testdata/parity/default-full-pipeline.case.json @@ -0,0 +1,28 @@ +{ + "name": "default_full_pipeline_shape_and_reports", + "transcript_file": "default-full-pipeline.transcript.json", + "glossary_file": "default-full-pipeline.glossary.yaml", + "proposal_responses_file": "default-full-pipeline.proposals.json", + "validation_responses_file": "default-full-pipeline.validations.json", + "env": { + "AUDITA_LLM_API_KEY": "phase17-secret", + "AUDITA_VALIDATION_LLM_API_KEY": "phase17-secret" + }, + "expect": { + "exit_code": 0, + "status": "success", + "output_transcript_file": "default-full-pipeline.expected-transcript.json", + "module_instances": ["glossary_1", "homophones", "glossary_2", "spoken_word", "grammar"], + "module_count": 5, + "total_applied_changes": 3, + "total_skipped_changes": 3, + "secret_markers": ["phase17-secret"], + "expected_proposal_calls": [ + "glossary_1:proposal", + "homophones:proposal", + "glossary_2:proposal", + "spoken_word:proposal", + "grammar:proposal" + ] + } +} diff --git a/internal/cli/testdata/parity/default-full-pipeline.expected-transcript.json b/internal/cli/testdata/parity/default-full-pipeline.expected-transcript.json new file mode 100644 index 0000000..92f501d --- /dev/null +++ b/internal/cli/testdata/parity/default-full-pipeline.expected-transcript.json @@ -0,0 +1,9 @@ +[ + { + "id": 1, + "speaker": "Alice", + "start": 0, + "end": 1, + "text": "Hello, there were Jesters hmm" + } +] diff --git a/internal/cli/testdata/parity/default-full-pipeline.glossary.yaml b/internal/cli/testdata/parity/default-full-pipeline.glossary.yaml new file mode 100644 index 0000000..e3882aa --- /dev/null +++ b/internal/cli/testdata/parity/default-full-pipeline.glossary.yaml @@ -0,0 +1,7 @@ +glossary: + - name: Jesters + aliases: + - jester + plural: jesters + category: faction + summary: A protected in-world faction term. diff --git a/internal/cli/testdata/parity/default-full-pipeline.proposals.json b/internal/cli/testdata/parity/default-full-pipeline.proposals.json new file mode 100644 index 0000000..9abf1dc --- /dev/null +++ b/internal/cli/testdata/parity/default-full-pipeline.proposals.json @@ -0,0 +1,28 @@ +[ + { + "corrections": [ + {"id": 1, "original_text": "gestures", "corrected_text": "Jesters", "confidence": 0.99} + ] + }, + { + "corrections": [ + {"id": 1, "original_text": "Jesters", "corrected_text": "jesters", "confidence": 0.99}, + {"id": 1, "original_text": "Jesters", "corrected_text": "JESTERX", "confidence": 0.99} + ] + }, + { + "corrections": [ + {"id": 1, "original_text": "jesters", "corrected_text": "JESTERS", "confidence": 0.99} + ] + }, + { + "corrections": [ + {"id": 1, "original_text": "uh", "corrected_text": "hmm", "confidence": 0.99} + ] + }, + { + "corrections": [ + {"id": 1, "original_text": "hello ,", "corrected_text": "Hello,", "confidence": 0.99} + ] + } +] diff --git a/internal/cli/testdata/parity/default-full-pipeline.transcript.json b/internal/cli/testdata/parity/default-full-pipeline.transcript.json new file mode 100644 index 0000000..90d2247 --- /dev/null +++ b/internal/cli/testdata/parity/default-full-pipeline.transcript.json @@ -0,0 +1,3 @@ +[ + {"id":1,"speaker":"Alice","start":0.0,"end":1.0,"text":"hello , there were gestures uh"} +] diff --git a/internal/cli/testdata/parity/default-full-pipeline.validations.json b/internal/cli/testdata/parity/default-full-pipeline.validations.json new file mode 100644 index 0000000..332905b --- /dev/null +++ b/internal/cli/testdata/parity/default-full-pipeline.validations.json @@ -0,0 +1,12 @@ +[ + {"validations": [{"correction_index": 0, "approved": true, "confidence": 0.99, "reason": "ok"}]}, + {"validations": [{"correction_index": 0, "approved": true, "confidence": 0.99, "reason": "ok"}]}, + {"validations": [{"correction_index": 0, "approved": true, "confidence": 0.99, "reason": "ok"}]}, + {"validations": [{"correction_index": 0, "approved": true, "confidence": 0.99, "reason": "ok"}]}, + {"validations": [{"correction_index": 0, "approved": true, "confidence": 0.99, "reason": "ok"}]}, + {"validations": [{"correction_index": 0, "approved": true, "confidence": 0.99, "reason": "ok"}]}, + {"validations": [{"correction_index": 0, "approved": true, "confidence": 0.99, "reason": "ok"}]}, + {"validations": [{"correction_index": 0, "approved": false, "confidence": 0.99, "reason": "reject cleanup"}]}, + {"validations": [{"correction_index": 0, "approved": true, "confidence": 0.99, "reason": "phase17-secret"}]}, + {"validations": [{"correction_index": 0, "approved": true, "confidence": 0.99, "reason": "ok"}]} +] diff --git a/internal/cli/testdata/parity/default-handoff.case.json b/internal/cli/testdata/parity/default-handoff.case.json new file mode 100644 index 0000000..7bb8bb4 --- /dev/null +++ b/internal/cli/testdata/parity/default-handoff.case.json @@ -0,0 +1,27 @@ +{ + "name": "default_pipeline_handoff_and_module_order", + "transcript_file": "default-handoff.transcript.json", + "glossary_file": "default-handoff.glossary.yaml", + "proposal_responses_file": "default-handoff.proposals.json", + "validation_responses_file": "default-handoff.validations.json", + "expect": { + "exit_code": 0, + "status": "success", + "output_transcript_file": "default-handoff.expected-transcript.json", + "module_instances": ["glossary_1", "homophones", "glossary_2", "spoken_word", "grammar"], + "module_count": 5, + "total_applied_changes": 5, + "total_skipped_changes": 0, + "module_applied_counts": [1, 1, 1, 1, 1], + "module_rejected_counts": [0, 0, 0, 0, 0], + "module_skip_counts": [0, 0, 0, 0, 0], + "expected_proposal_calls": [ + "glossary_1:proposal", + "homophones:proposal", + "glossary_2:proposal", + "spoken_word:proposal", + "grammar:proposal" + ], + "min_response_payload_artifacts": 15 + } +} diff --git a/internal/cli/testdata/parity/default-handoff.expected-transcript.json b/internal/cli/testdata/parity/default-handoff.expected-transcript.json new file mode 100644 index 0000000..758ce58 --- /dev/null +++ b/internal/cli/testdata/parity/default-handoff.expected-transcript.json @@ -0,0 +1,3 @@ +[ + {"id":1,"speaker":"Alice","start":0,"end":1,"text":"Hello, there were Jesters at the Sight um"} +] diff --git a/internal/cli/testdata/parity/default-handoff.glossary.yaml b/internal/cli/testdata/parity/default-handoff.glossary.yaml new file mode 100644 index 0000000..4c5255e --- /dev/null +++ b/internal/cli/testdata/parity/default-handoff.glossary.yaml @@ -0,0 +1,6 @@ +glossary: + - name: Jesters + aliases: [jester] + plural: jesters + category: faction + summary: A protected in-world faction term. diff --git a/internal/cli/testdata/parity/default-handoff.proposals.json b/internal/cli/testdata/parity/default-handoff.proposals.json new file mode 100644 index 0000000..4ef6f41 --- /dev/null +++ b/internal/cli/testdata/parity/default-handoff.proposals.json @@ -0,0 +1,7 @@ +[ + {"corrections": [{"id": 1, "original_text": "gestures", "corrected_text": "Jesters", "confidence": 0.99}]}, + {"corrections": [{"id": 1, "original_text": "site", "corrected_text": "sight", "confidence": 0.99}]}, + {"corrections": [{"id": 1, "original_text": "sight", "corrected_text": "Sight", "confidence": 0.99}]}, + {"corrections": [{"id": 1, "original_text": "um um", "corrected_text": "um", "confidence": 0.99}]}, + {"corrections": [{"id": 1, "original_text": "hello ,", "corrected_text": "Hello,", "confidence": 0.99}]} +] diff --git a/internal/cli/testdata/parity/default-handoff.transcript.json b/internal/cli/testdata/parity/default-handoff.transcript.json new file mode 100644 index 0000000..8f4c15c --- /dev/null +++ b/internal/cli/testdata/parity/default-handoff.transcript.json @@ -0,0 +1,3 @@ +[ + {"id":1,"speaker":"Alice","start":0.0,"end":1.0,"text":"hello , there were gestures at the site um um"} +] diff --git a/internal/cli/testdata/parity/default-handoff.validations.json b/internal/cli/testdata/parity/default-handoff.validations.json new file mode 100644 index 0000000..1b38c93 --- /dev/null +++ b/internal/cli/testdata/parity/default-handoff.validations.json @@ -0,0 +1,12 @@ +[ + {"validations": [{"correction_index": 0, "approved": true, "confidence": 0.99, "reason": "ok"}]}, + {"validations": [{"correction_index": 0, "approved": true, "confidence": 0.99, "reason": "ok"}]}, + {"validations": [{"correction_index": 0, "approved": true, "confidence": 0.99, "reason": "ok"}]}, + {"validations": [{"correction_index": 0, "approved": true, "confidence": 0.99, "reason": "ok"}]}, + {"validations": [{"correction_index": 0, "approved": true, "confidence": 0.99, "reason": "ok"}]}, + {"validations": [{"correction_index": 0, "approved": true, "confidence": 0.99, "reason": "ok"}]}, + {"validations": [{"correction_index": 0, "approved": true, "confidence": 0.99, "reason": "ok"}]}, + {"validations": [{"correction_index": 0, "approved": true, "confidence": 0.99, "reason": "ok"}]}, + {"validations": [{"correction_index": 0, "approved": true, "confidence": 0.99, "reason": "ok"}]}, + {"validations": [{"correction_index": 0, "approved": true, "confidence": 0.99, "reason": "ok"}]} +] diff --git a/internal/cli/testdata/parity/deterministic-validator-low-confidence.case.json b/internal/cli/testdata/parity/deterministic-validator-low-confidence.case.json new file mode 100644 index 0000000..dc7263b --- /dev/null +++ b/internal/cli/testdata/parity/deterministic-validator-low-confidence.case.json @@ -0,0 +1,17 @@ +{ + "name": "deterministic_validator_low_confidence", + "transcript_file": "deterministic-validator-low-confidence.transcript.json", + "glossary_file": "default-full-pipeline.glossary.yaml", + "modules_csv": "grammar", + "proposal_responses_file": "deterministic-validator-low-confidence.proposals.json", + "expect": { + "exit_code": 0, + "status": "success", + "output_transcript_file": "deterministic-validator-low-confidence.expected-transcript.json", + "module_instances": ["grammar"], + "module_count": 1, + "total_skipped_changes": 1, + "validator_rejected_reason_codes": ["low_confidence"], + "expected_proposal_calls": ["grammar:proposal"] + } +} diff --git a/internal/cli/testdata/parity/deterministic-validator-low-confidence.expected-transcript.json b/internal/cli/testdata/parity/deterministic-validator-low-confidence.expected-transcript.json new file mode 100644 index 0000000..4805248 --- /dev/null +++ b/internal/cli/testdata/parity/deterministic-validator-low-confidence.expected-transcript.json @@ -0,0 +1,3 @@ +[ + {"id":1,"speaker":"Alice","start":0,"end":1,"text":"hello , world"} +] diff --git a/internal/cli/testdata/parity/deterministic-validator-low-confidence.proposals.json b/internal/cli/testdata/parity/deterministic-validator-low-confidence.proposals.json new file mode 100644 index 0000000..0181162 --- /dev/null +++ b/internal/cli/testdata/parity/deterministic-validator-low-confidence.proposals.json @@ -0,0 +1,7 @@ +[ + { + "corrections": [ + {"id": 1, "original_text": "hello ,", "corrected_text": "Hello,", "confidence": 0.1} + ] + } +] diff --git a/internal/cli/testdata/parity/deterministic-validator-low-confidence.transcript.json b/internal/cli/testdata/parity/deterministic-validator-low-confidence.transcript.json new file mode 100644 index 0000000..3fabfea --- /dev/null +++ b/internal/cli/testdata/parity/deterministic-validator-low-confidence.transcript.json @@ -0,0 +1,3 @@ +[ + {"id":1,"speaker":"Alice","start":0.0,"end":1.0,"text":"hello , world"} +] diff --git a/internal/cli/testdata/parity/glossary-schema-error.case.json b/internal/cli/testdata/parity/glossary-schema-error.case.json new file mode 100644 index 0000000..f5adcdd --- /dev/null +++ b/internal/cli/testdata/parity/glossary-schema-error.case.json @@ -0,0 +1,12 @@ +{ + "name": "glossary_schema_handling", + "transcript_file": "default-full-pipeline.transcript.json", + "glossary_file": "glossary-schema-error.glossary.yaml", + "expect": { + "exit_code": 1, + "status": "failed", + "error_phase": "glossary_schema", + "stderr_contains": "glossary_schema", + "require_error_log": true + } +} diff --git a/internal/cli/testdata/parity/glossary-schema-error.glossary.yaml b/internal/cli/testdata/parity/glossary-schema-error.glossary.yaml new file mode 100644 index 0000000..0e0bc18 --- /dev/null +++ b/internal/cli/testdata/parity/glossary-schema-error.glossary.yaml @@ -0,0 +1,2 @@ +glossary: + - name: MissingCategoryAndSummary diff --git a/internal/cli/testdata/parity/llm-validator-rejection.case.json b/internal/cli/testdata/parity/llm-validator-rejection.case.json new file mode 100644 index 0000000..afae85a --- /dev/null +++ b/internal/cli/testdata/parity/llm-validator-rejection.case.json @@ -0,0 +1,19 @@ +{ + "name": "llm_validator_decision_handling", + "transcript_file": "llm-validator-rejection.transcript.json", + "glossary_file": "default-full-pipeline.glossary.yaml", + "modules_csv": "grammar", + "proposal_responses_file": "llm-validator-rejection.proposals.json", + "validation_responses_file": "llm-validator-rejection.validations.json", + "expect": { + "exit_code": 0, + "status": "success", + "output_transcript_file": "llm-validator-rejection.expected-transcript.json", + "module_instances": ["grammar"], + "module_count": 1, + "total_skipped_changes": 1, + "validator_rejected_reason_codes": ["llm_rejected"], + "expected_proposal_calls": ["grammar:proposal"], + "expected_validation_calls": ["grammar:grammar_only_guard:batch-0000"] + } +} diff --git a/internal/cli/testdata/parity/llm-validator-rejection.expected-transcript.json b/internal/cli/testdata/parity/llm-validator-rejection.expected-transcript.json new file mode 100644 index 0000000..4805248 --- /dev/null +++ b/internal/cli/testdata/parity/llm-validator-rejection.expected-transcript.json @@ -0,0 +1,3 @@ +[ + {"id":1,"speaker":"Alice","start":0,"end":1,"text":"hello , world"} +] diff --git a/internal/cli/testdata/parity/llm-validator-rejection.proposals.json b/internal/cli/testdata/parity/llm-validator-rejection.proposals.json new file mode 100644 index 0000000..95bf617 --- /dev/null +++ b/internal/cli/testdata/parity/llm-validator-rejection.proposals.json @@ -0,0 +1,7 @@ +[ + { + "corrections": [ + {"id": 1, "original_text": "hello ,", "corrected_text": "Hello,", "confidence": 0.99} + ] + } +] diff --git a/internal/cli/testdata/parity/llm-validator-rejection.transcript.json b/internal/cli/testdata/parity/llm-validator-rejection.transcript.json new file mode 100644 index 0000000..3fabfea --- /dev/null +++ b/internal/cli/testdata/parity/llm-validator-rejection.transcript.json @@ -0,0 +1,3 @@ +[ + {"id":1,"speaker":"Alice","start":0.0,"end":1.0,"text":"hello , world"} +] diff --git a/internal/cli/testdata/parity/llm-validator-rejection.validations.json b/internal/cli/testdata/parity/llm-validator-rejection.validations.json new file mode 100644 index 0000000..0ad3b3c --- /dev/null +++ b/internal/cli/testdata/parity/llm-validator-rejection.validations.json @@ -0,0 +1,7 @@ +[ + { + "validations": [ + {"correction_index": 0, "approved": false, "confidence": 0.99, "reason": "reject stylistic overreach"} + ] + } +] diff --git a/internal/cli/testdata/parity/mid-pipeline-failure.case.json b/internal/cli/testdata/parity/mid-pipeline-failure.case.json new file mode 100644 index 0000000..9a3458e --- /dev/null +++ b/internal/cli/testdata/parity/mid-pipeline-failure.case.json @@ -0,0 +1,23 @@ +{ + "name": "mid_pipeline_failure_partial_progress", + "transcript_file": "default-handoff.transcript.json", + "glossary_file": "default-handoff.glossary.yaml", + "proposal_responses_file": "mid-pipeline-failure.proposals.json", + "validation_responses_file": "mid-pipeline-failure.validations.json", + "expect": { + "exit_code": 1, + "status": "failed", + "error_phase": "runner_execution", + "stderr_contains": "runner_execution", + "module_instances": ["glossary_1", "homophones", "glossary_2"], + "module_count": 3, + "total_applied_changes": 2, + "total_skipped_changes": 0, + "failed_module_instance": "glossary_2", + "module_applied_counts": [1, 1, 0], + "module_rejected_counts": [0, 0, 0], + "module_skip_counts": [0, 0, 0], + "require_error_log": true, + "expected_proposal_calls": ["glossary_1:proposal", "homophones:proposal", "glossary_2:proposal"] + } +} diff --git a/internal/cli/testdata/parity/mid-pipeline-failure.proposals.json b/internal/cli/testdata/parity/mid-pipeline-failure.proposals.json new file mode 100644 index 0000000..d0a8e10 --- /dev/null +++ b/internal/cli/testdata/parity/mid-pipeline-failure.proposals.json @@ -0,0 +1,4 @@ +[ + {"corrections": [{"id": 1, "original_text": "gestures", "corrected_text": "Jesters", "confidence": 0.99}]}, + {"corrections": [{"id": 1, "original_text": "site", "corrected_text": "sight", "confidence": 0.99}]} +] diff --git a/internal/cli/testdata/parity/mid-pipeline-failure.validations.json b/internal/cli/testdata/parity/mid-pipeline-failure.validations.json new file mode 100644 index 0000000..327196a --- /dev/null +++ b/internal/cli/testdata/parity/mid-pipeline-failure.validations.json @@ -0,0 +1,6 @@ +[ + {"validations": [{"correction_index": 0, "approved": true, "confidence": 0.99, "reason": "ok"}]}, + {"validations": [{"correction_index": 0, "approved": true, "confidence": 0.99, "reason": "ok"}]}, + {"validations": [{"correction_index": 0, "approved": true, "confidence": 0.99, "reason": "ok"}]}, + {"validations": [{"correction_index": 0, "approved": true, "confidence": 0.99, "reason": "ok"}]} +] diff --git a/internal/cli/testdata/parity/protected-term-rejection.case.json b/internal/cli/testdata/parity/protected-term-rejection.case.json new file mode 100644 index 0000000..2022045 --- /dev/null +++ b/internal/cli/testdata/parity/protected-term-rejection.case.json @@ -0,0 +1,21 @@ +{ + "name": "protected_glossary_term_behavior", + "transcript_file": "protected-term-rejection.transcript.json", + "glossary_file": "default-handoff.glossary.yaml", + "modules_csv": "homophones", + "proposal_responses_file": "protected-term-rejection.proposals.json", + "expect": { + "exit_code": 0, + "status": "success", + "output_transcript_file": "protected-term-rejection.expected-transcript.json", + "module_instances": ["homophones"], + "module_count": 1, + "total_applied_changes": 0, + "total_skipped_changes": 1, + "module_applied_counts": [0], + "module_rejected_counts": [1], + "module_skip_counts": [0], + "validator_rejected_reason_codes": ["protected_glossary_term"], + "expected_proposal_calls": ["homophones:proposal"] + } +} diff --git a/internal/cli/testdata/parity/protected-term-rejection.expected-transcript.json b/internal/cli/testdata/parity/protected-term-rejection.expected-transcript.json new file mode 100644 index 0000000..a8a2883 --- /dev/null +++ b/internal/cli/testdata/parity/protected-term-rejection.expected-transcript.json @@ -0,0 +1,3 @@ +[ + {"id":1,"speaker":"Alice","start":0,"end":1,"text":"The Jesters entered the hall."} +] diff --git a/internal/cli/testdata/parity/protected-term-rejection.proposals.json b/internal/cli/testdata/parity/protected-term-rejection.proposals.json new file mode 100644 index 0000000..30a9f94 --- /dev/null +++ b/internal/cli/testdata/parity/protected-term-rejection.proposals.json @@ -0,0 +1,3 @@ +[ + {"corrections": [{"id": 1, "original_text": "Jesters", "corrected_text": "Gestures", "confidence": 0.99}]} +] diff --git a/internal/cli/testdata/parity/protected-term-rejection.transcript.json b/internal/cli/testdata/parity/protected-term-rejection.transcript.json new file mode 100644 index 0000000..ce3fce1 --- /dev/null +++ b/internal/cli/testdata/parity/protected-term-rejection.transcript.json @@ -0,0 +1,3 @@ +[ + {"id":1,"speaker":"Alice","start":0.0,"end":1.0,"text":"The Jesters entered the hall."} +] diff --git a/internal/cli/testdata/parity/transcript-schema-error.case.json b/internal/cli/testdata/parity/transcript-schema-error.case.json new file mode 100644 index 0000000..74c5e04 --- /dev/null +++ b/internal/cli/testdata/parity/transcript-schema-error.case.json @@ -0,0 +1,12 @@ +{ + "name": "transcript_schema_handling", + "transcript_file": "transcript-schema-error.transcript.json", + "glossary_file": "default-full-pipeline.glossary.yaml", + "expect": { + "exit_code": 1, + "status": "failed", + "error_phase": "transcript_schema", + "stderr_contains": "transcript_schema", + "require_error_log": true + } +} diff --git a/internal/cli/testdata/parity/transcript-schema-error.transcript.json b/internal/cli/testdata/parity/transcript-schema-error.transcript.json new file mode 100644 index 0000000..36671cf --- /dev/null +++ b/internal/cli/testdata/parity/transcript-schema-error.transcript.json @@ -0,0 +1,3 @@ +[ + {"id":1,"speaker":"","start":0.0,"end":1.0,"text":"bad"} +]