Complete Phase 17 parity fixture suite

docs/architecture.md

@@ -43,7 +43,6 @@ Implemented today:
 - Explicit runtime support for `--modules spoken_word` through the production runner path.
 
 Not implemented in CLI runtime path today:
-- Python parity fixture suite and parity verification workflow.
 - Operational hardening tasks beyond current runtime/reporting/diagnostics behavior.
 - Rollout and Python retirement work.
 
@@ -66,7 +65,8 @@ Phase sequencing note:
 - Phase 14 homophones module implementation and explicit runtime wiring are complete;
 - Phase 15 spoken-word module implementation and explicit runtime wiring are complete;
 - Phase 16 default full pipeline integration is complete;
-- next recommended phase is Phase 17 (Python parity fixture suite).
+- Phase 17 parity fixture suite is complete;
+- next recommended phase is Phase 18 (operational hardening and subprocess integration).
 
 ## Actual Go package layout
 
@@ -184,6 +184,12 @@ Current runtime flow (`internal/cli/run.go`):
 15. Optionally write `--report-json`; always write run-dir `report.json`.
 16. Apply work-dir retention.
 
+Parity fixture status:
+- representative Python-parity fixture coverage exists under `internal/cli/testdata/parity`;
+- parity tests use fake structured LLM responses for deterministic behavior, including default full-pipeline shape assertions;
+- parity comparisons intentionally ignore nondeterministic metadata (timestamps, run IDs, temp paths, token usage) and remain strict for deterministic contract fields (transcript content, module order/instance naming, applied/skipped/rejected counts, and status).
+- intentional Python-vs-Go differences and open parity gaps are documented in `docs/python-parity.md`.
+
 Important behavior details:
 - Glossary is validated and is used for explicit glossary/grammar/homophones/spoken_word module correction paths.
 - Default production CLI behavior now executes the full production module sequence unless `--modules` override is supplied.
@@ -254,6 +260,7 @@ Current caveat:
 Current runtime boundary:
 - the default CLI runtime path (without explicit module selection) instantiates the full production module sequence.
 - LLM calls are exercised in production in both default full-pipeline runs and explicit `--modules` runs, and in tests when fake/injected clients are used.
+- normal `go test ./...` does not require real LLM credentials or Python dependencies.
 
 `internal/framework/llm` also provides:
 - a bounded `Scheduler` for controlled concurrent LLM calls with reliable permit release;

docs/python-parity.md

@@ -0,0 +1,78 @@
+# Python vs Go Parity Notes
+
+This document tracks Phase 17 parity-fixture coverage and differences between the original Python implementation and the Go rewrite.
+
+## Scope
+
+- Uses deterministic fixture-driven tests in `internal/cli/testdata/parity`.
+- Uses fake structured LLM responses for proposal generation and LLM validators.
+- Verifies functional contract fields (module order, instance naming, applied/skipped counts, report status, diagnostics presence).
+- Does not require real LLM credentials or Python dependencies during `go test ./...`.
+
+## Intentional Differences
+
+The following differences are expected and treated as intentional unless they break contract behavior:
+
+1. JSON formatting and field ordering
+- Serialized JSON whitespace and object key order may differ.
+- Parity tests compare JSON semantically, not byte-for-byte.
+
+2. Diagnostics path values
+- Absolute run-directory paths, run IDs, and temp directory roots differ by runtime and platform.
+- Parity checks assert artifact presence/shape, not exact absolute paths.
+
+3. Time-variant metadata
+- Timestamps (`started_at`, `completed_at`) and generated run IDs are runtime-specific.
+- Parity checks ignore exact timestamp/run-id values.
+
+4. Provider metadata/token accounting
+- Provider/token usage metadata may vary by adapter behavior and is not asserted as strict parity fields.
+
+5. Internal adapter implementation details
+- Go uses its own structured LLM adapter implementation details while preserving the same high-level contract semantics.
+
+## Current Fixture Coverage
+
+Current parity fixtures cover:
+
+- Transcript schema handling failure path.
+- Glossary schema handling failure path.
+- Default full module sequence shape:
+  - `glossary_1`, `homophones`, `glossary_2`, `spoken_word`, `grammar`.
+- Mutable transcript handoff across default stages.
+- Module-specific behavior inside the default sequence:
+  - glossary correction
+  - homophone-style correction
+  - spoken-word cleanup
+  - grammar cleanup
+- Protected glossary-term guardrail behavior.
+- Deterministic validator rejection behavior.
+- LLM validator decision/rejection behavior.
+- Application-level skip behavior (`ambiguous_original_text`).
+- Mid-pipeline failure with partial progress preserved in reports.
+- Diagnostics artifact presence and secret-redaction checks.
+
+## Open Parity Gaps (Not Intentional)
+
+These are known Phase 17 expansion opportunities and should not be labeled as intentional compatibility differences:
+
+1. Broader Python fixture import
+- The current Go parity fixtures are native fixture cases; they do not yet ingest all existing Python test fixtures directly.
+
+2. Side-by-side runner command
+- No repository-standard Python+Go side-by-side parity command is required or enforced yet.
+
+3. Wider transcript corpus
+- Current fixtures are representative but not exhaustive across all transcript/glossary edge combinations.
+
+## How To Extend
+
+1. Add a new `*.case.json` file under `internal/cli/testdata/parity`.
+2. Add referenced transcript/glossary/fake-LLM response files.
+3. Encode deterministic expectations in the case:
+- module order and instance names
+- transcript output
+- applied/skipped/rejected counts
+- report status and failure metadata
+- diagnostics artifact presence/redaction markers
+4. Run `go test ./...`.

docs/rewrite-notes.md

@@ -98,7 +98,6 @@ Implemented:
 - Generic JSON diagnostics primitives for LLM interactions (request metadata, request payload, response payload, optional error payload) with secret redaction.
 
 Not yet implemented in runtime pipeline:
-- Python parity fixture suite and parity verification workflow.
 - Operational hardening beyond current Phase 16 runtime/reporting/diagnostics scope.
 - Rollout/Python retirement work.
 
@@ -238,7 +237,7 @@ Not implemented in Phase 8 (by design):
 
 ## Remaining work plan
 
-Next recommended phase: **Phase 17 (Python parity fixture suite)**.
+Next recommended phase: **Phase 18 (operational hardening and subprocess integration)**.
 
 ## Phase 9: Structured LLM client and scheduler infrastructure
 
@@ -527,6 +526,8 @@ Intentionally deferred:
 
 ## Phase 17: Python parity fixture suite
 
+Completed.
+
 ### Purpose
 
 Establish confidence that the Go implementation matches the behavior and safety posture of the initial Python implementation.
@@ -566,6 +567,35 @@ The repository has a durable test suite demonstrating that the Go implementation
 - Unintentional compatibility breaks are fixed.
 - `go test ./...` passes.
 
+### Phase 17 completion status
+
+Implemented:
+- Durable parity fixture harness in Go test path (`internal/cli/parity_test.go`).
+- Representative parity fixture corpus under `internal/cli/testdata/parity`.
+- Fake LLM proposal/validator fixtures driving deterministic parity tests.
+- Default full-pipeline parity coverage including sequence/order and repeated glossary naming:
+  - `glossary_1`
+  - `homophones`
+  - `glossary_2`
+  - `spoken_word`
+  - `grammar`
+- Parity checks that are strict for deterministic contract fields:
+  - transcript content
+  - module order and instance names
+  - applied/skipped/rejected counts
+  - report status and failed-module metadata
+- Parity checks that ignore nondeterministic metadata fields:
+  - timestamps
+  - run IDs
+  - temp/absolute paths
+  - provider token usage details
+- Documentation of intentional Python-vs-Go differences and honest open parity gaps in `docs/python-parity.md`.
+- Normal `go test ./...` path remains independent of real LLM credentials and Python dependencies.
+
+Intentionally deferred:
+- Phase 18 operational hardening and subprocess integration expansion.
+- Phase 19 rollout and Python retirement work.
+
 ## Phase 18: Operational hardening and subprocess integration
 
 ### Purpose

internal/cli/parity_test.go

@@ -0,0 +1,338 @@
+package cli
+
+import (
+	"bytes"
+	"encoding/json"
+	"os"
+	"path/filepath"
+	"reflect"
+	"strings"
+	"testing"
+
+	"gitea.maximumdirect.net/eric/audita/internal/core/reporting"
+)
+
+type parityFixtureCase struct {
+	Name                    string            `json:"name"`
+	TranscriptFile          string            `json:"transcript_file"`
+	GlossaryFile            string            `json:"glossary_file"`
+	ModulesCSV              string            `json:"modules_csv,omitempty"`
+	ProposalResponsesFile   string            `json:"proposal_responses_file,omitempty"`
+	ValidationResponsesFile string            `json:"validation_responses_file,omitempty"`
+	Env                     map[string]string `json:"env,omitempty"`
+	Expect                  parityExpectation `json:"expect"`
+}
+
+type parityExpectation struct {
+	ExitCode                     int      `json:"exit_code"`
+	Status                       string   `json:"status,omitempty"`
+	ErrorPhase                   string   `json:"error_phase,omitempty"`
+	StdoutMode                   string   `json:"stdout_mode,omitempty"` // empty|json
+	StderrContains               string   `json:"stderr_contains,omitempty"`
+	OutputTranscriptFile         string   `json:"output_transcript_file,omitempty"`
+	ModuleInstances              []string `json:"module_instances,omitempty"`
+	ModuleCount                  int      `json:"module_count,omitempty"`
+	TotalAppliedChanges          int      `json:"total_applied_changes,omitempty"`
+	TotalSkippedChanges          int      `json:"total_skipped_changes,omitempty"`
+	FailedModuleInstance         string   `json:"failed_module_instance,omitempty"`
+	ValidatorRejectedReasonCodes []string `json:"validator_rejected_reason_codes,omitempty"`
+	ApplicationSkipReasonCodes   []string `json:"application_skip_reason_codes,omitempty"`
+	RequireErrorLog              bool     `json:"require_error_log,omitempty"`
+	SecretMarkers                []string `json:"secret_markers,omitempty"`
+	ExpectedProposalCalls        []string `json:"expected_proposal_calls,omitempty"`
+	ExpectedValidationCalls      []string `json:"expected_validation_calls,omitempty"`
+	ModuleAppliedCounts          []int    `json:"module_applied_counts,omitempty"`
+	ModuleRejectedCounts         []int    `json:"module_rejected_counts,omitempty"`
+	ModuleSkipCounts             []int    `json:"module_skip_counts,omitempty"`
+	MinResponsePayloadArtifacts  int      `json:"min_response_payload_artifacts,omitempty"`
+}
+
+func TestParityFixtures(t *testing.T) {
+	casePaths, err := filepath.Glob(parityFixturePath("*.case.json"))
+	if err != nil {
+		t.Fatalf("glob parity fixtures: %v", err)
+	}
+	if len(casePaths) == 0 {
+		t.Fatal("expected at least one parity fixture case")
+	}
+
+	for _, casePath := range casePaths {
+		fx := loadParityFixtureCase(t, casePath)
+		t.Run(fx.Name, func(t *testing.T) {
+			runParityFixtureCase(t, filepath.Dir(casePath), fx)
+		})
+	}
+}
+
+func loadParityFixtureCase(t *testing.T, casePath string) parityFixtureCase {
+	t.Helper()
+	var fx parityFixtureCase
+	raw := readFile(t, casePath)
+	if err := json.Unmarshal(raw, &fx); err != nil {
+		t.Fatalf("parse parity case %q: %v", casePath, err)
+	}
+	if strings.TrimSpace(fx.Name) == "" {
+		t.Fatalf("parity case %q missing name", casePath)
+	}
+	return fx
+}
+
+func runParityFixtureCase(t *testing.T, caseDir string, fx parityFixtureCase) {
+	t.Helper()
+	for k, v := range fx.Env {
+		t.Setenv(k, v)
+	}
+
+	proposalClient := &fakeStructuredLLMClient{}
+	validationClient := &fakeStructuredLLMClient{}
+
+	if strings.TrimSpace(fx.ProposalResponsesFile) != "" {
+		raw := readFile(t, filepath.Join(caseDir, fx.ProposalResponsesFile))
+		if err := json.Unmarshal(raw, &proposalClient.proposalResponses); err != nil {
+			t.Fatalf("parse proposal responses: %v", err)
+		}
+		processProposalLLMClient = proposalClient
+	}
+	if strings.TrimSpace(fx.ValidationResponsesFile) != "" {
+		raw := readFile(t, filepath.Join(caseDir, fx.ValidationResponsesFile))
+		if err := json.Unmarshal(raw, &validationClient.validationResponses); err != nil {
+			t.Fatalf("parse validation responses: %v", err)
+		}
+		processValidationLLMClient = validationClient
+	}
+	t.Cleanup(func() {
+		processProposalLLMClient = nil
+		processValidationLLMClient = nil
+	})
+
+	var stdout, stderr bytes.Buffer
+	workDir := t.TempDir()
+	outputPath := filepath.Join(t.TempDir(), "out.json")
+	reportPath := filepath.Join(t.TempDir(), "report.json")
+	args := []string{
+		"process",
+		filepath.Join(caseDir, fx.TranscriptFile),
+		"--glossary",
+		filepath.Join(caseDir, fx.GlossaryFile),
+		"--report-json",
+		reportPath,
+		"--work-dir",
+		workDir,
+		"--work-dir-retention",
+		"always",
+	}
+	// Keep stdout shape deterministic for parity tests.
+	if fx.Expect.StdoutMode != "json" {
+		args = append(args, "--output", outputPath)
+	}
+	if strings.TrimSpace(fx.ModulesCSV) != "" {
+		args = append(args, "--modules", fx.ModulesCSV)
+	}
+
+	exitCode := Run(args, &stdout, &stderr)
+	if exitCode != fx.Expect.ExitCode {
+		t.Fatalf("expected exit code %d, got %d stderr=%q", fx.Expect.ExitCode, exitCode, stderr.String())
+	}
+
+	switch fx.Expect.StdoutMode {
+	case "json":
+		if _, err := json.Marshal(stdout.String()); err != nil {
+			t.Fatalf("unexpected stdout marshal error: %v", err)
+		}
+		if !json.Valid(stdout.Bytes()) {
+			t.Fatalf("expected JSON stdout, got %q", stdout.String())
+		}
+	default:
+		if stdout.Len() != 0 {
+			t.Fatalf("expected empty stdout, got %q", stdout.String())
+		}
+	}
+
+	if fx.Expect.StderrContains != "" && !strings.Contains(stderr.String(), fx.Expect.StderrContains) {
+		t.Fatalf("expected stderr to contain %q, got %q", fx.Expect.StderrContains, stderr.String())
+	}
+
+	report := readProcessReport(t, reportPath)
+	assertParityReport(t, report, fx.Expect)
+
+	runDir := onlyRunDir(t, workDir)
+	runDirReport := readProcessReport(t, filepath.Join(runDir, "report.json"))
+	assertParityReport(t, runDirReport, fx.Expect)
+
+	if fx.Expect.RequireErrorLog {
+		if _, err := os.Stat(filepath.Join(runDir, "error.log")); err != nil {
+			t.Fatalf("expected error.log: %v", err)
+		}
+	}
+
+	if strings.TrimSpace(fx.Expect.OutputTranscriptFile) != "" && fx.Expect.ExitCode == 0 {
+		got := readFile(t, outputPath)
+		want := readFile(t, filepath.Join(caseDir, fx.Expect.OutputTranscriptFile))
+		assertJSONSemanticEqual(t, want, got)
+	}
+
+	if len(fx.Expect.ExpectedProposalCalls) > 0 && !reflect.DeepEqual(proposalClient.calls, fx.Expect.ExpectedProposalCalls) {
+		t.Fatalf("unexpected proposal calls: got %v want %v", proposalClient.calls, fx.Expect.ExpectedProposalCalls)
+	}
+	if len(fx.Expect.ExpectedValidationCalls) > 0 && !reflect.DeepEqual(validationClient.calls, fx.Expect.ExpectedValidationCalls) {
+		t.Fatalf("unexpected validation calls: got %v want %v", validationClient.calls, fx.Expect.ExpectedValidationCalls)
+	}
+
+	if len(fx.Expect.SecretMarkers) > 0 {
+		assertNoSecretMarkers(t, reportPath, fx.Expect.SecretMarkers)
+		assertNoSecretMarkersInTree(t, runDir, fx.Expect.SecretMarkers)
+	}
+	if fx.Expect.MinResponsePayloadArtifacts > 0 {
+		matches, err := filepath.Glob(filepath.Join(runDir, "*", "*response-payload.json"))
+		if err != nil {
+			t.Fatalf("glob response payload artifacts: %v", err)
+		}
+		if len(matches) < fx.Expect.MinResponsePayloadArtifacts {
+			t.Fatalf("expected at least %d response payload artifacts, got %d", fx.Expect.MinResponsePayloadArtifacts, len(matches))
+		}
+	}
+}
+
+func assertParityReport(t *testing.T, report reporting.ProcessReport, exp parityExpectation) {
+	t.Helper()
+	if exp.Status != "" && report.Status != exp.Status {
+		t.Fatalf("expected report status %q, got %q", exp.Status, report.Status)
+	}
+	if exp.ErrorPhase != "" && report.ErrorPhase != exp.ErrorPhase {
+		t.Fatalf("expected report error_phase %q, got %q", exp.ErrorPhase, report.ErrorPhase)
+	}
+	if len(exp.ModuleInstances) > 0 {
+		got := make([]string, 0, len(report.ModuleResults))
+		for _, mr := range report.ModuleResults {
+			got = append(got, mr.ModuleInstance)
+		}
+		if !reflect.DeepEqual(got, exp.ModuleInstances) {
+			t.Fatalf("unexpected module instances: got %v want %v", got, exp.ModuleInstances)
+		}
+	}
+	if exp.ModuleCount > 0 {
+		if report.ModulesSummary == nil || report.ModulesSummary.ModuleCount != exp.ModuleCount {
+			t.Fatalf("expected module_count=%d, got %+v", exp.ModuleCount, report.ModulesSummary)
+		}
+	}
+	if exp.TotalAppliedChanges > 0 {
+		if report.ModulesSummary == nil || report.ModulesSummary.TotalAppliedChanges != exp.TotalAppliedChanges {
+			t.Fatalf("expected total_applied_changes=%d, got %+v", exp.TotalAppliedChanges, report.ModulesSummary)
+		}
+	}
+	if exp.TotalSkippedChanges > 0 {
+		if report.ModulesSummary == nil || report.ModulesSummary.TotalSkippedChanges != exp.TotalSkippedChanges {
+			t.Fatalf("expected total_skipped_changes=%d, got %+v", exp.TotalSkippedChanges, report.ModulesSummary)
+		}
+	}
+	if exp.FailedModuleInstance != "" {
+		if report.ModulesSummary == nil || report.ModulesSummary.FailedModuleInstance != exp.FailedModuleInstance {
+			t.Fatalf("expected failed_module_instance=%q, got %+v", exp.FailedModuleInstance, report.ModulesSummary)
+		}
+	}
+
+	if len(exp.ValidatorRejectedReasonCodes) > 0 {
+		got := collectValidatorRejectedReasonCodes(report.ModuleResults)
+		if !reflect.DeepEqual(got, exp.ValidatorRejectedReasonCodes) {
+			t.Fatalf("unexpected validator rejected reason codes: got %v want %v", got, exp.ValidatorRejectedReasonCodes)
+		}
+	}
+	if len(exp.ApplicationSkipReasonCodes) > 0 {
+		got := collectApplicationSkipReasonCodes(report.ModuleResults)
+		if !reflect.DeepEqual(got, exp.ApplicationSkipReasonCodes) {
+			t.Fatalf("unexpected application skip reason codes: got %v want %v", got, exp.ApplicationSkipReasonCodes)
+		}
+	}
+	if len(exp.ModuleAppliedCounts) > 0 {
+		got := make([]int, 0, len(report.ModuleResults))
+		for _, mr := range report.ModuleResults {
+			got = append(got, len(mr.AppliedChanges))
+		}
+		if !reflect.DeepEqual(got, exp.ModuleAppliedCounts) {
+			t.Fatalf("unexpected per-module applied counts: got %v want %v", got, exp.ModuleAppliedCounts)
+		}
+	}
+	if len(exp.ModuleRejectedCounts) > 0 {
+		got := make([]int, 0, len(report.ModuleResults))
+		for _, mr := range report.ModuleResults {
+			got = append(got, len(mr.ValidatorRejected))
+		}
+		if !reflect.DeepEqual(got, exp.ModuleRejectedCounts) {
+			t.Fatalf("unexpected per-module rejected counts: got %v want %v", got, exp.ModuleRejectedCounts)
+		}
+	}
+	if len(exp.ModuleSkipCounts) > 0 {
+		got := make([]int, 0, len(report.ModuleResults))
+		for _, mr := range report.ModuleResults {
+			got = append(got, len(mr.SkippedChanges))
+		}
+		if !reflect.DeepEqual(got, exp.ModuleSkipCounts) {
+			t.Fatalf("unexpected per-module skip counts: got %v want %v", got, exp.ModuleSkipCounts)
+		}
+	}
+}
+
+func collectValidatorRejectedReasonCodes(results []reporting.ModuleReport) []string {
+	out := make([]string, 0)
+	for _, mr := range results {
+		for _, vr := range mr.ValidatorRejected {
+			out = append(out, vr.ReasonCode)
+		}
+	}
+	return out
+}
+
+func collectApplicationSkipReasonCodes(results []reporting.ModuleReport) []string {
+	out := make([]string, 0)
+	for _, mr := range results {
+		for _, sk := range mr.SkippedChanges {
+			out = append(out, string(sk.SkipReason))
+		}
+	}
+	return out
+}
+
+func assertJSONSemanticEqual(t *testing.T, expected []byte, actual []byte) {
+	t.Helper()
+	var exp any
+	var act any
+	if err := json.Unmarshal(expected, &exp); err != nil {
+		t.Fatalf("unmarshal expected json: %v", err)
+	}
+	if err := json.Unmarshal(actual, &act); err != nil {
+		t.Fatalf("unmarshal actual json: %v", err)
+	}
+	if !reflect.DeepEqual(exp, act) {
+		t.Fatalf("JSON mismatch\nexpected=%s\nactual=%s", string(expected), string(actual))
+	}
+}
+
+func assertNoSecretMarkers(t *testing.T, filePath string, markers []string) {
+	t.Helper()
+	raw := string(readFile(t, filePath))
+	for _, marker := range markers {
+		if marker != "" && strings.Contains(raw, marker) {
+			t.Fatalf("secret marker %q leaked in %s", marker, filePath)
+		}
+	}
+}
+
+func assertNoSecretMarkersInTree(t *testing.T, root string, markers []string) {
+	t.Helper()
+	_ = filepath.WalkDir(root, func(path string, d os.DirEntry, err error) error {
+		if err != nil || d == nil || d.IsDir() {
+			return nil
+		}
+		raw := string(readFile(t, path))
+		for _, marker := range markers {
+			if marker != "" && strings.Contains(raw, marker) {
+				t.Fatalf("secret marker %q leaked in %s", marker, path)
+			}
+		}
+		return nil
+	})
+}
+
+func parityFixturePath(name string) string {
+	return filepath.Join("testdata", "parity", name)
+}

internal/cli/testdata/parity/application-skip-ambiguous.case.json

@@ -0,0 +1,22 @@
+{
+  "name": "ambiguous_match_rejected_before_application",
+  "transcript_file": "application-skip-ambiguous.transcript.json",
+  "glossary_file": "default-handoff.glossary.yaml",
+  "modules_csv": "homophones",
+  "proposal_responses_file": "application-skip-ambiguous.proposals.json",
+  "validation_responses_file": "application-skip-ambiguous.validations.json",
+  "expect": {
+    "exit_code": 0,
+    "status": "success",
+    "output_transcript_file": "application-skip-ambiguous.expected-transcript.json",
+    "module_instances": ["homophones"],
+    "module_count": 1,
+    "total_applied_changes": 0,
+    "total_skipped_changes": 1,
+    "module_applied_counts": [0],
+    "module_rejected_counts": [1],
+    "module_skip_counts": [0],
+    "validator_rejected_reason_codes": ["ambiguous_original_text"],
+    "expected_proposal_calls": ["homophones:proposal"]
+  }
+}

internal/cli/testdata/parity/application-skip-ambiguous.expected-transcript.json

@@ -0,0 +1,3 @@
+[
+  {"id":1,"speaker":"Alice","start":0,"end":1,"text":"the site near another site"}
+]

internal/cli/testdata/parity/application-skip-ambiguous.proposals.json

@@ -0,0 +1,3 @@
+[
+  {"corrections": [{"id": 1, "original_text": "site", "corrected_text": "sight", "confidence": 0.99}]}
+]

internal/cli/testdata/parity/application-skip-ambiguous.transcript.json

@@ -0,0 +1,3 @@
+[
+  {"id":1,"speaker":"Alice","start":0.0,"end":1.0,"text":"the site near another site"}
+]

internal/cli/testdata/parity/application-skip-ambiguous.validations.json

@@ -0,0 +1,4 @@
+[
+  {"validations": [{"correction_index": 0, "approved": true, "confidence": 0.99, "reason": "ok"}]},
+  {"validations": [{"correction_index": 0, "approved": true, "confidence": 0.99, "reason": "ok"}]}
+]

internal/cli/testdata/parity/default-full-pipeline.case.json

@@ -0,0 +1,28 @@
+{
+  "name": "default_full_pipeline_shape_and_reports",
+  "transcript_file": "default-full-pipeline.transcript.json",
+  "glossary_file": "default-full-pipeline.glossary.yaml",
+  "proposal_responses_file": "default-full-pipeline.proposals.json",
+  "validation_responses_file": "default-full-pipeline.validations.json",
+  "env": {
+    "AUDITA_LLM_API_KEY": "phase17-secret",
+    "AUDITA_VALIDATION_LLM_API_KEY": "phase17-secret"
+  },
+  "expect": {
+    "exit_code": 0,
+    "status": "success",
+    "output_transcript_file": "default-full-pipeline.expected-transcript.json",
+    "module_instances": ["glossary_1", "homophones", "glossary_2", "spoken_word", "grammar"],
+    "module_count": 5,
+    "total_applied_changes": 3,
+    "total_skipped_changes": 3,
+    "secret_markers": ["phase17-secret"],
+    "expected_proposal_calls": [
+      "glossary_1:proposal",
+      "homophones:proposal",
+      "glossary_2:proposal",
+      "spoken_word:proposal",
+      "grammar:proposal"
+    ]
+  }
+}

internal/cli/testdata/parity/default-full-pipeline.expected-transcript.json

@@ -0,0 +1,9 @@
+[
+  {
+    "id": 1,
+    "speaker": "Alice",
+    "start": 0,
+    "end": 1,
+    "text": "Hello, there were Jesters hmm"
+  }
+]

internal/cli/testdata/parity/default-full-pipeline.glossary.yaml

@@ -0,0 +1,7 @@
+glossary:
+  - name: Jesters
+    aliases:
+      - jester
+    plural: jesters
+    category: faction
+    summary: A protected in-world faction term.

internal/cli/testdata/parity/default-full-pipeline.proposals.json

@@ -0,0 +1,28 @@
+[
+  {
+    "corrections": [
+      {"id": 1, "original_text": "gestures", "corrected_text": "Jesters", "confidence": 0.99}
+    ]
+  },
+  {
+    "corrections": [
+      {"id": 1, "original_text": "Jesters", "corrected_text": "jesters", "confidence": 0.99},
+      {"id": 1, "original_text": "Jesters", "corrected_text": "JESTERX", "confidence": 0.99}
+    ]
+  },
+  {
+    "corrections": [
+      {"id": 1, "original_text": "jesters", "corrected_text": "JESTERS", "confidence": 0.99}
+    ]
+  },
+  {
+    "corrections": [
+      {"id": 1, "original_text": "uh", "corrected_text": "hmm", "confidence": 0.99}
+    ]
+  },
+  {
+    "corrections": [
+      {"id": 1, "original_text": "hello ,", "corrected_text": "Hello,", "confidence": 0.99}
+    ]
+  }
+]

internal/cli/testdata/parity/default-full-pipeline.transcript.json

@@ -0,0 +1,3 @@
+[
+  {"id":1,"speaker":"Alice","start":0.0,"end":1.0,"text":"hello , there were gestures uh"}
+]

internal/cli/testdata/parity/default-full-pipeline.validations.json

@@ -0,0 +1,12 @@
+[
+  {"validations": [{"correction_index": 0, "approved": true, "confidence": 0.99, "reason": "ok"}]},
+  {"validations": [{"correction_index": 0, "approved": true, "confidence": 0.99, "reason": "ok"}]},
+  {"validations": [{"correction_index": 0, "approved": true, "confidence": 0.99, "reason": "ok"}]},
+  {"validations": [{"correction_index": 0, "approved": true, "confidence": 0.99, "reason": "ok"}]},
+  {"validations": [{"correction_index": 0, "approved": true, "confidence": 0.99, "reason": "ok"}]},
+  {"validations": [{"correction_index": 0, "approved": true, "confidence": 0.99, "reason": "ok"}]},
+  {"validations": [{"correction_index": 0, "approved": true, "confidence": 0.99, "reason": "ok"}]},
+  {"validations": [{"correction_index": 0, "approved": false, "confidence": 0.99, "reason": "reject cleanup"}]},
+  {"validations": [{"correction_index": 0, "approved": true, "confidence": 0.99, "reason": "phase17-secret"}]},
+  {"validations": [{"correction_index": 0, "approved": true, "confidence": 0.99, "reason": "ok"}]}
+]

internal/cli/testdata/parity/default-handoff.case.json

@@ -0,0 +1,27 @@
+{
+  "name": "default_pipeline_handoff_and_module_order",
+  "transcript_file": "default-handoff.transcript.json",
+  "glossary_file": "default-handoff.glossary.yaml",
+  "proposal_responses_file": "default-handoff.proposals.json",
+  "validation_responses_file": "default-handoff.validations.json",
+  "expect": {
+    "exit_code": 0,
+    "status": "success",
+    "output_transcript_file": "default-handoff.expected-transcript.json",
+    "module_instances": ["glossary_1", "homophones", "glossary_2", "spoken_word", "grammar"],
+    "module_count": 5,
+    "total_applied_changes": 5,
+    "total_skipped_changes": 0,
+    "module_applied_counts": [1, 1, 1, 1, 1],
+    "module_rejected_counts": [0, 0, 0, 0, 0],
+    "module_skip_counts": [0, 0, 0, 0, 0],
+    "expected_proposal_calls": [
+      "glossary_1:proposal",
+      "homophones:proposal",
+      "glossary_2:proposal",
+      "spoken_word:proposal",
+      "grammar:proposal"
+    ],
+    "min_response_payload_artifacts": 15
+  }
+}

internal/cli/testdata/parity/default-handoff.expected-transcript.json

@@ -0,0 +1,3 @@
+[
+  {"id":1,"speaker":"Alice","start":0,"end":1,"text":"Hello, there were Jesters at the Sight um"}
+]

internal/cli/testdata/parity/default-handoff.glossary.yaml

@@ -0,0 +1,6 @@
+glossary:
+  - name: Jesters
+    aliases: [jester]
+    plural: jesters
+    category: faction
+    summary: A protected in-world faction term.

internal/cli/testdata/parity/default-handoff.proposals.json

@@ -0,0 +1,7 @@
+[
+  {"corrections": [{"id": 1, "original_text": "gestures", "corrected_text": "Jesters", "confidence": 0.99}]},
+  {"corrections": [{"id": 1, "original_text": "site", "corrected_text": "sight", "confidence": 0.99}]},
+  {"corrections": [{"id": 1, "original_text": "sight", "corrected_text": "Sight", "confidence": 0.99}]},
+  {"corrections": [{"id": 1, "original_text": "um um", "corrected_text": "um", "confidence": 0.99}]},
+  {"corrections": [{"id": 1, "original_text": "hello ,", "corrected_text": "Hello,", "confidence": 0.99}]}
+]

internal/cli/testdata/parity/default-handoff.transcript.json

@@ -0,0 +1,3 @@
+[
+  {"id":1,"speaker":"Alice","start":0.0,"end":1.0,"text":"hello , there were gestures at the site um um"}
+]

internal/cli/testdata/parity/default-handoff.validations.json

@@ -0,0 +1,12 @@
+[
+  {"validations": [{"correction_index": 0, "approved": true, "confidence": 0.99, "reason": "ok"}]},
+  {"validations": [{"correction_index": 0, "approved": true, "confidence": 0.99, "reason": "ok"}]},
+  {"validations": [{"correction_index": 0, "approved": true, "confidence": 0.99, "reason": "ok"}]},
+  {"validations": [{"correction_index": 0, "approved": true, "confidence": 0.99, "reason": "ok"}]},
+  {"validations": [{"correction_index": 0, "approved": true, "confidence": 0.99, "reason": "ok"}]},
+  {"validations": [{"correction_index": 0, "approved": true, "confidence": 0.99, "reason": "ok"}]},
+  {"validations": [{"correction_index": 0, "approved": true, "confidence": 0.99, "reason": "ok"}]},
+  {"validations": [{"correction_index": 0, "approved": true, "confidence": 0.99, "reason": "ok"}]},
+  {"validations": [{"correction_index": 0, "approved": true, "confidence": 0.99, "reason": "ok"}]},
+  {"validations": [{"correction_index": 0, "approved": true, "confidence": 0.99, "reason": "ok"}]}
+]

internal/cli/testdata/parity/deterministic-validator-low-confidence.case.json

@@ -0,0 +1,17 @@
+{
+  "name": "deterministic_validator_low_confidence",
+  "transcript_file": "deterministic-validator-low-confidence.transcript.json",
+  "glossary_file": "default-full-pipeline.glossary.yaml",
+  "modules_csv": "grammar",
+  "proposal_responses_file": "deterministic-validator-low-confidence.proposals.json",
+  "expect": {
+    "exit_code": 0,
+    "status": "success",
+    "output_transcript_file": "deterministic-validator-low-confidence.expected-transcript.json",
+    "module_instances": ["grammar"],
+    "module_count": 1,
+    "total_skipped_changes": 1,
+    "validator_rejected_reason_codes": ["low_confidence"],
+    "expected_proposal_calls": ["grammar:proposal"]
+  }
+}

internal/cli/testdata/parity/deterministic-validator-low-confidence.expected-transcript.json

@@ -0,0 +1,3 @@
+[
+  {"id":1,"speaker":"Alice","start":0,"end":1,"text":"hello , world"}
+]

internal/cli/testdata/parity/deterministic-validator-low-confidence.proposals.json

@@ -0,0 +1,7 @@
+[
+  {
+    "corrections": [
+      {"id": 1, "original_text": "hello ,", "corrected_text": "Hello,", "confidence": 0.1}
+    ]
+  }
+]

internal/cli/testdata/parity/deterministic-validator-low-confidence.transcript.json

@@ -0,0 +1,3 @@
+[
+  {"id":1,"speaker":"Alice","start":0.0,"end":1.0,"text":"hello , world"}
+]

internal/cli/testdata/parity/glossary-schema-error.case.json

@@ -0,0 +1,12 @@
+{
+  "name": "glossary_schema_handling",
+  "transcript_file": "default-full-pipeline.transcript.json",
+  "glossary_file": "glossary-schema-error.glossary.yaml",
+  "expect": {
+    "exit_code": 1,
+    "status": "failed",
+    "error_phase": "glossary_schema",
+    "stderr_contains": "glossary_schema",
+    "require_error_log": true
+  }
+}

internal/cli/testdata/parity/glossary-schema-error.glossary.yaml

@@ -0,0 +1,2 @@
+glossary:
+  - name: MissingCategoryAndSummary

internal/cli/testdata/parity/llm-validator-rejection.case.json

@@ -0,0 +1,19 @@
+{
+  "name": "llm_validator_decision_handling",
+  "transcript_file": "llm-validator-rejection.transcript.json",
+  "glossary_file": "default-full-pipeline.glossary.yaml",
+  "modules_csv": "grammar",
+  "proposal_responses_file": "llm-validator-rejection.proposals.json",
+  "validation_responses_file": "llm-validator-rejection.validations.json",
+  "expect": {
+    "exit_code": 0,
+    "status": "success",
+    "output_transcript_file": "llm-validator-rejection.expected-transcript.json",
+    "module_instances": ["grammar"],
+    "module_count": 1,
+    "total_skipped_changes": 1,
+    "validator_rejected_reason_codes": ["llm_rejected"],
+    "expected_proposal_calls": ["grammar:proposal"],
+    "expected_validation_calls": ["grammar:grammar_only_guard:batch-0000"]
+  }
+}

internal/cli/testdata/parity/llm-validator-rejection.expected-transcript.json

@@ -0,0 +1,3 @@
+[
+  {"id":1,"speaker":"Alice","start":0,"end":1,"text":"hello , world"}
+]

internal/cli/testdata/parity/llm-validator-rejection.proposals.json

@@ -0,0 +1,7 @@
+[
+  {
+    "corrections": [
+      {"id": 1, "original_text": "hello ,", "corrected_text": "Hello,", "confidence": 0.99}
+    ]
+  }
+]

internal/cli/testdata/parity/llm-validator-rejection.transcript.json

@@ -0,0 +1,3 @@
+[
+  {"id":1,"speaker":"Alice","start":0.0,"end":1.0,"text":"hello , world"}
+]

internal/cli/testdata/parity/llm-validator-rejection.validations.json

@@ -0,0 +1,7 @@
+[
+  {
+    "validations": [
+      {"correction_index": 0, "approved": false, "confidence": 0.99, "reason": "reject stylistic overreach"}
+    ]
+  }
+]

internal/cli/testdata/parity/mid-pipeline-failure.case.json

@@ -0,0 +1,23 @@
+{
+  "name": "mid_pipeline_failure_partial_progress",
+  "transcript_file": "default-handoff.transcript.json",
+  "glossary_file": "default-handoff.glossary.yaml",
+  "proposal_responses_file": "mid-pipeline-failure.proposals.json",
+  "validation_responses_file": "mid-pipeline-failure.validations.json",
+  "expect": {
+    "exit_code": 1,
+    "status": "failed",
+    "error_phase": "runner_execution",
+    "stderr_contains": "runner_execution",
+    "module_instances": ["glossary_1", "homophones", "glossary_2"],
+    "module_count": 3,
+    "total_applied_changes": 2,
+    "total_skipped_changes": 0,
+    "failed_module_instance": "glossary_2",
+    "module_applied_counts": [1, 1, 0],
+    "module_rejected_counts": [0, 0, 0],
+    "module_skip_counts": [0, 0, 0],
+    "require_error_log": true,
+    "expected_proposal_calls": ["glossary_1:proposal", "homophones:proposal", "glossary_2:proposal"]
+  }
+}

internal/cli/testdata/parity/mid-pipeline-failure.proposals.json

@@ -0,0 +1,4 @@
+[
+  {"corrections": [{"id": 1, "original_text": "gestures", "corrected_text": "Jesters", "confidence": 0.99}]},
+  {"corrections": [{"id": 1, "original_text": "site", "corrected_text": "sight", "confidence": 0.99}]}
+]

internal/cli/testdata/parity/mid-pipeline-failure.validations.json

@@ -0,0 +1,6 @@
+[
+  {"validations": [{"correction_index": 0, "approved": true, "confidence": 0.99, "reason": "ok"}]},
+  {"validations": [{"correction_index": 0, "approved": true, "confidence": 0.99, "reason": "ok"}]},
+  {"validations": [{"correction_index": 0, "approved": true, "confidence": 0.99, "reason": "ok"}]},
+  {"validations": [{"correction_index": 0, "approved": true, "confidence": 0.99, "reason": "ok"}]}
+]

internal/cli/testdata/parity/protected-term-rejection.case.json

@@ -0,0 +1,21 @@
+{
+  "name": "protected_glossary_term_behavior",
+  "transcript_file": "protected-term-rejection.transcript.json",
+  "glossary_file": "default-handoff.glossary.yaml",
+  "modules_csv": "homophones",
+  "proposal_responses_file": "protected-term-rejection.proposals.json",
+  "expect": {
+    "exit_code": 0,
+    "status": "success",
+    "output_transcript_file": "protected-term-rejection.expected-transcript.json",
+    "module_instances": ["homophones"],
+    "module_count": 1,
+    "total_applied_changes": 0,
+    "total_skipped_changes": 1,
+    "module_applied_counts": [0],
+    "module_rejected_counts": [1],
+    "module_skip_counts": [0],
+    "validator_rejected_reason_codes": ["protected_glossary_term"],
+    "expected_proposal_calls": ["homophones:proposal"]
+  }
+}

internal/cli/testdata/parity/protected-term-rejection.expected-transcript.json

@@ -0,0 +1,3 @@
+[
+  {"id":1,"speaker":"Alice","start":0,"end":1,"text":"The Jesters entered the hall."}
+]

internal/cli/testdata/parity/protected-term-rejection.proposals.json

@@ -0,0 +1,3 @@
+[
+  {"corrections": [{"id": 1, "original_text": "Jesters", "corrected_text": "Gestures", "confidence": 0.99}]}
+]

internal/cli/testdata/parity/protected-term-rejection.transcript.json

@@ -0,0 +1,3 @@
+[
+  {"id":1,"speaker":"Alice","start":0.0,"end":1.0,"text":"The Jesters entered the hall."}
+]

internal/cli/testdata/parity/transcript-schema-error.case.json

@@ -0,0 +1,12 @@
+{
+  "name": "transcript_schema_handling",
+  "transcript_file": "transcript-schema-error.transcript.json",
+  "glossary_file": "default-full-pipeline.glossary.yaml",
+  "expect": {
+    "exit_code": 1,
+    "status": "failed",
+    "error_phase": "transcript_schema",
+    "stderr_contains": "transcript_schema",
+    "require_error_log": true
+  }
+}

internal/cli/testdata/parity/transcript-schema-error.transcript.json

@@ -0,0 +1,3 @@
+[
+  {"id":1,"speaker":"","start":0.0,"end":1.0,"text":"bad"}
+]