diff --git a/docs/integrations/output-files.md b/docs/integrations/output-files.md index bb92db8..c99a54a 100644 --- a/docs/integrations/output-files.md +++ b/docs/integrations/output-files.md @@ -11,7 +11,7 @@ With reporting enabled, Audita produces three ordinary files: - `errors.yml`: affected work units and run-level failures; - `recoveries.yml`: successful logical model operations with repair, service - retry, or defective-proposal filtering; + retry, or response filtering; - `report.md`: a concise overview and complete per-chunk matrices. The YAML documents have Audita-owned schemas `audita-errors` and @@ -55,6 +55,9 @@ work unit form one error record. `scope: run` instead records phase, safe reason and message, optional origin, and output path where relevant. It can describe a fatal error or secondary output-write failure without inventing a failed model work unit. Raw provider errors and transcript text are absent. +For `validation_decisions_invalid`, proposal indices identify only candidates +whose required decisions were unavailable. Other candidates in the same batch +may still be approved or rejected normally. Each recovery identifies its work unit and successful operation, with sorted `reasons`, a safe message, available execution counts, and sorted @@ -65,6 +68,11 @@ ordinary editorial rejection, and an apply-time conflict caused by an earlier accepted correction are not filtering recoveries. Failed operations are errors, not recoveries. The initial chunk identity remains stable even if earlier modules change its text. +`validation_decisions_filtered` records unexpected validator decision indices +ignored from a successful validation batch. It may accompany repair or service +recovery reasons, but it does not add `filtered_proposals` or increment +`proposal_responses_with_filtering`. A batch with unavailable requested decisions +is an error even when unexpected indices were also ignored. ## Markdown Overview @@ -91,8 +99,14 @@ These files are diagnostic evidence, not an alternate ordinary report schema. Each candidate-backed ledger entry includes `proposed_corrected_text`, the literal replacement span, even when it is `""` for a deletion. Applied entries also include `applied_corrected_text`, likewise `""` for an applied deletion. -Rejected, skipped, and interrupted entries omit the applied field; a module +Rejected, unavailable, skipped, and interrupted entries omit the applied field; a module failure without a candidate omits both. These fields are replacement spans, not full resulting-segment text. Deterministic decision records now identify the blank-result guard as `non_empty_resulting_segment` (formerly `non_empty_corrected_text`); its `empty_resulting_segment` reason is unchanged. +Validator decision records carry `disposition`: `approved`, `rejected`, or +`unavailable`. A missing or duplicate required decision is `unavailable`, not +an editorial rejection. The candidate ledger uses the same distinction in its +top-level disposition. Interaction response artifacts retain supplied model +decisions and the reconciliation result, including unavailable and ignored +indices; unexpected indices never create candidate ledger entries. diff --git a/docs/internal/diagnostics-reporting.md b/docs/internal/diagnostics-reporting.md index 8f11049..8019e3c 100644 --- a/docs/internal/diagnostics-reporting.md +++ b/docs/internal/diagnostics-reporting.md @@ -15,6 +15,11 @@ Audita's versioned YAML DTOs, fixed Markdown rendering, and brief CLI completion formatter. YAML redaction walks serialized string nodes so numeric and boolean values retain their types. Ordinary reports omit raw causes, transcript text, complete decisions, and token/timing payloads. +Validation operations with unavailable decisions report only affected requested +indices, even when other candidates in the batch continue. Ignored extra +validator entries yield `validation_decisions_filtered` only for a successful +operation; they do not count as filtered correction proposals. Failed operations +retain their execution counts and never produce a recovery record. `internal/core/diagnostics.OutputSession` owns active destination checks, transcript preflight, report initialization, regular-file replacement, and direct @@ -40,6 +45,12 @@ Debug write failures reach the CLI through an explicit concurrency-safe diagnost callback and produce redacted stderr messages without aborting processing. Completed validator decisions returned with a fatal batch error remain available to the debug ledger; they never authorize applying an incompletely validated edit. +Runner decision records carry the validator's explicit approved, rejected, or +unavailable disposition into the ledger. Unavailable candidates have a distinct +top-level ledger disposition, while fatal interruptions remain interrupted when +no rejection or unavailable decision was returned. Successful validation +interaction payloads contain the full decoded response, final content, and +reconciliation outcomes; ignored indices do not become ledger candidates. Every debug writer uses all configured backend secrets for redaction, including unselected backend credentials and source snapshots. Debug can still contain sensitive transcript and prompt content. diff --git a/docs/roadmap/implementation.md b/docs/roadmap/implementation.md index 958bdfe..d4fe91a 100644 --- a/docs/roadmap/implementation.md +++ b/docs/roadmap/implementation.md @@ -192,6 +192,12 @@ successful extra filtering. No extra model generation is introduced. ## Stage 2 — Carry Dispositions Through Debug and Report Partial Outcomes +**Completed:** 2026-09-25. Runner and ledger records carry explicit +dispositions; ordinary reports distinguish unavailable decision errors from +extra-index filtering recoveries. Focused report, ledger, runner, and YAML +checks passed with `go test ./internal/framework/runner +./internal/framework/processreport ./internal/core/reporting`. + **Work** 1. Add and populate disposition fields through the runner and ledger as selected diff --git a/internal/framework/processreport/correction_ledger.go b/internal/framework/processreport/correction_ledger.go index fff9be1..158e30c 100644 --- a/internal/framework/processreport/correction_ledger.go +++ b/internal/framework/processreport/correction_ledger.go @@ -6,12 +6,14 @@ import ( "sort" "gitea.maximumdirect.net/eric/audita/internal/framework/runner" + "gitea.maximumdirect.net/eric/audita/internal/framework/validators" validatormetadata "gitea.maximumdirect.net/eric/audita/internal/validators/metadata" ) const ( correctionDispositionApplied = "applied" correctionDispositionRejected = "rejected" + correctionDispositionUnavailable = "unavailable" correctionDispositionSkipped = "skipped" correctionDispositionFailed = "failed" correctionDispositionInterrupted = "interrupted" @@ -40,10 +42,11 @@ type CorrectionLedgerEntry struct { } type LedgerValidatorDecisionRecord struct { - ValidatorKey string `json:"validator_key"` - Approved bool `json:"approved"` - ReasonCode string `json:"reason_code"` - Message string `json:"message,omitempty"` + ValidatorKey string `json:"validator_key"` + Approved bool `json:"approved"` + Disposition validators.DecisionDisposition `json:"disposition"` + ReasonCode string `json:"reason_code"` + Message string `json:"message,omitempty"` } func BuildCorrectionLedger(input CorrectionLedgerInput) []CorrectionLedgerEntry { @@ -101,6 +104,10 @@ func BuildCorrectionLedger(input CorrectionLedgerInput) []CorrectionLedgerEntry }) } for _, rejection := range module.ValidatorRejected { + disposition := correctionDispositionRejected + if rejection.Disposition == validators.DispositionUnavailable { + disposition = correctionDispositionUnavailable + } seen[identity(module.ModuleInstance, rejection.ProposalIndex)] = true entries = append(entries, CorrectionLedgerEntry{ RunID: runID, @@ -111,7 +118,7 @@ func BuildCorrectionLedger(input CorrectionLedgerInput) []CorrectionLedgerEntry OriginalText: rejection.OriginalText, ProposedCorrectedText: &rejection.CorrectedText, ReplacementPolicy: string(module.ReplacementPolicy), - Disposition: correctionDispositionRejected, + Disposition: disposition, DispositionReasonCode: rejection.ReasonCode, DispositionMessage: rejection.Message, DeterministicValidatorResults: filterLedgerDecisions(decisionsByProposal[rejection.ProposalIndex], false), @@ -184,6 +191,7 @@ func filterLedgerDecisions(in []runner.ValidatorDecisionRecord, wantLLM bool) [] out = append(out, LedgerValidatorDecisionRecord{ ValidatorKey: decision.ValidatorName, Approved: decision.Approved, + Disposition: decision.Disposition, ReasonCode: decision.ReasonCode, Message: decision.Message, }) diff --git a/internal/framework/processreport/correction_ledger_test.go b/internal/framework/processreport/correction_ledger_test.go index ca3ae9d..f29c35c 100644 --- a/internal/framework/processreport/correction_ledger_test.go +++ b/internal/framework/processreport/correction_ledger_test.go @@ -6,6 +6,7 @@ import ( "gitea.maximumdirect.net/eric/audita/internal/framework/proposals" "gitea.maximumdirect.net/eric/audita/internal/framework/runner" + "gitea.maximumdirect.net/eric/audita/internal/framework/validators" ) func TestBuildCorrectionLedgerClassifiesValidatorDecisionsFromCanonicalMetadata(t *testing.T) { @@ -16,8 +17,8 @@ func TestBuildCorrectionLedgerClassifiesValidatorDecisionsFromCanonicalMetadata( ModuleInstance: "glossary", ReplacementPolicy: proposals.ReplacementPolicyReplaceAll, ValidatorDecisions: []runner.ValidatorDecisionRecord{ - {ValidatorName: "proposal_shape", ProposalIndex: 3, Approved: true, ReasonCode: "approved"}, - {ValidatorName: "spoken_form_plausibility", ProposalIndex: 3, Approved: true, ReasonCode: "approved"}, + {ValidatorName: "proposal_shape", ProposalIndex: 3, Approved: true, Disposition: validators.DispositionApproved, ReasonCode: "approved"}, + {ValidatorName: "spoken_form_plausibility", ProposalIndex: 3, Approved: true, Disposition: validators.DispositionApproved, ReasonCode: "approved"}, }, AppliedChanges: []proposals.AppliedChange{ { @@ -44,10 +45,10 @@ func TestBuildCorrectionLedgerClassifiesValidatorDecisionsFromCanonicalMetadata( if entry.RunID != "audita-run-id" || entry.Disposition != "applied" || entry.AppliedCorrectedText == nil || *entry.AppliedCorrectedText != "Jesters" { t.Fatalf("unexpected applied ledger entry: %+v", entry) } - if len(entry.DeterministicValidatorResults) != 1 || entry.DeterministicValidatorResults[0].ValidatorKey != "proposal_shape" { + if len(entry.DeterministicValidatorResults) != 1 || entry.DeterministicValidatorResults[0].ValidatorKey != "proposal_shape" || entry.DeterministicValidatorResults[0].Disposition != validators.DispositionApproved { t.Fatalf("unexpected deterministic decision split: %+v", entry.DeterministicValidatorResults) } - if len(entry.LLMValidatorResults) != 1 || entry.LLMValidatorResults[0].ValidatorKey != "spoken_form_plausibility" { + if len(entry.LLMValidatorResults) != 1 || entry.LLMValidatorResults[0].ValidatorKey != "spoken_form_plausibility" || entry.LLMValidatorResults[0].Disposition != validators.DispositionApproved { t.Fatalf("unexpected llm-backed decision split: %+v", entry.LLMValidatorResults) } } @@ -76,6 +77,7 @@ func TestBuildCorrectionLedgerPreservesDispositionPolicy(t *testing.T) { TargetSegmentID: 8, OriginalText: "before", CorrectedText: "after", + Disposition: validators.DispositionRejected, ReasonCode: "protected_term", Message: "blocked", }, @@ -117,7 +119,7 @@ func TestCorrectionLedgerIncludesUnappliedCandidateAfterFatalValidation(t *testi output := &runner.RunOutput{ Candidates: []proposals.EnrichedCorrectionProposal{{CorrectionProposal: proposals.CorrectionProposal{TargetSegmentID: 7, OriginalText: "old", CorrectedText: "new"}, ProposalMetadata: proposals.ProposalMetadata{ModuleKey: "grammar", ModuleInstance: "grammar", ProposalIndex: 2}}}, ModuleResults: []runner.ModuleResult{{ModuleKey: "grammar", ModuleInstance: "grammar", Status: runner.ModuleStatusFailed, - ValidatorDecisions: []runner.ValidatorDecisionRecord{{ValidatorName: "spoken_form_plausibility", ProposalIndex: 2, Approved: false, ReasonCode: "transport_failed"}}}}, + ValidatorDecisions: []runner.ValidatorDecisionRecord{{ValidatorName: "spoken_form_plausibility", ProposalIndex: 2, Approved: false, Disposition: validators.DispositionUnavailable, ReasonCode: "transport_failed"}}}}, } ledger := BuildCorrectionLedger(CorrectionLedgerInput{RunOutput: output}) if len(ledger) != 2 { @@ -127,7 +129,7 @@ func TestCorrectionLedgerIncludesUnappliedCandidateAfterFatalValidation(t *testi if entry.ProposalIndex != 2 { continue } - if entry.Disposition != correctionDispositionInterrupted || entry.OriginalText != "old" || len(entry.LLMValidatorResults) != 1 { + if entry.Disposition != correctionDispositionInterrupted || entry.OriginalText != "old" || len(entry.LLMValidatorResults) != 1 || entry.LLMValidatorResults[0].Disposition != validators.DispositionUnavailable { t.Fatalf("candidate evidence lost: %+v", entry) } return @@ -140,9 +142,16 @@ func TestCorrectionLedgerSerializesEmptyReplacementByDisposition(t *testing.T) { ModuleResults: []runner.ModuleResult{ { ModuleKey: "spoken_word", ModuleInstance: "spoken_word", ReplacementPolicy: proposals.ReplacementPolicyRequireUnique, - AppliedChanges: []proposals.AppliedChange{{ProposalIndex: 0, TargetSegmentID: 1, OriginalText: "uh ", CorrectedText: ""}}, - ValidatorRejected: []runner.ValidatorRejectedChange{{ProposalIndex: 1, TargetSegmentID: 2, OriginalText: "uh ", CorrectedText: "", ReasonCode: "llm_rejected"}}, - SkippedChanges: []proposals.SkippedChange{{ProposalIndex: 2, TargetSegmentID: 3, OriginalText: "uh ", CorrectedText: "", SkipReason: proposals.SkipReasonEmptyResultingText}}, + AppliedChanges: []proposals.AppliedChange{{ProposalIndex: 0, TargetSegmentID: 1, OriginalText: "uh ", CorrectedText: ""}}, + ValidatorRejected: []runner.ValidatorRejectedChange{ + {ProposalIndex: 1, TargetSegmentID: 2, OriginalText: "uh ", CorrectedText: "", Disposition: validators.DispositionRejected, ReasonCode: "llm_rejected"}, + {ProposalIndex: 4, TargetSegmentID: 5, OriginalText: "uh ", CorrectedText: "", Disposition: validators.DispositionUnavailable, ReasonCode: validators.ReasonMissingValidationDecision}, + }, + ValidatorDecisions: []runner.ValidatorDecisionRecord{ + {ValidatorName: "editorial_review", ProposalIndex: 1, Disposition: validators.DispositionRejected, ReasonCode: "llm_rejected"}, + {ValidatorName: "editorial_review", ProposalIndex: 4, Disposition: validators.DispositionUnavailable, ReasonCode: validators.ReasonMissingValidationDecision}, + }, + SkippedChanges: []proposals.SkippedChange{{ProposalIndex: 2, TargetSegmentID: 3, OriginalText: "uh ", CorrectedText: "", SkipReason: proposals.SkipReasonEmptyResultingText}}, }, {ModuleKey: "grammar", ModuleInstance: "grammar", Status: runner.ModuleStatusFailed, ErrorMessage: "failed"}, }, @@ -167,6 +176,7 @@ func TestCorrectionLedgerSerializesEmptyReplacementByDisposition(t *testing.T) { }{ {correctionDispositionApplied, true, true}, {correctionDispositionRejected, true, false}, + {correctionDispositionUnavailable, true, false}, {correctionDispositionSkipped, true, false}, {correctionDispositionInterrupted, true, false}, {correctionDispositionFailed, false, false}, @@ -180,5 +190,11 @@ func TestCorrectionLedgerSerializesEmptyReplacementByDisposition(t *testing.T) { if proposedPresent != tc.proposed || appliedPresent != tc.applied || (proposedPresent && string(proposed) != `""`) || (appliedPresent && string(applied) != `""`) { t.Fatalf("incorrect serialized replacement fields for %s: %+v", tc.disposition, fields) } + if tc.disposition == correctionDispositionUnavailable || tc.disposition == correctionDispositionRejected { + var decisions []LedgerValidatorDecisionRecord + if err := json.Unmarshal(fields["llm_validator_decisions"], &decisions); err != nil || len(decisions) != 1 || string(decisions[0].Disposition) != tc.disposition { + t.Fatalf("incorrect validator disposition for %s: %+v, %v", tc.disposition, decisions, err) + } + } } } diff --git a/internal/framework/processreport/partial_validation_test.go b/internal/framework/processreport/partial_validation_test.go index 800e310..774630b 100644 --- a/internal/framework/processreport/partial_validation_test.go +++ b/internal/framework/processreport/partial_validation_test.go @@ -51,6 +51,10 @@ func (c *interruptedValidationClient) CompleteStructured(_ context.Context, req return structuredoutput.Response{}, &structuredoutput.Failure{Class: structuredoutput.FailureFatal, Reason: outcomes.ReasonSetupFailure, Message: "validation interrupted", Cause: c.cause} } index := *req.Context.BatchIndex + if index == 0 { + *target.(*validators.LLMValidationResponse) = validators.LLMValidationResponse{} + return structuredoutput.Response{}, nil + } *target.(*validators.LLMValidationResponse) = validators.LLMValidationResponse{Validations: []validators.LLMValidationDecision{{CorrectionIndex: index, Approved: index == 1, Confidence: 1, Reason: "completed assessment"}}} return structuredoutput.Response{}, nil } @@ -89,11 +93,15 @@ func TestFatalBatchRetainsEarlierDecisionsInLedger(t *testing.T) { } for index := 0; index < 2; index++ { entry := candidates[index] - if len(entry.LLMValidatorResults) != 1 || entry.LLMValidatorResults[0].Message != "completed assessment" || entry.LLMValidatorResults[0].Approved != (index == 1) { + wantDisposition := validators.DispositionUnavailable + if index == 1 { + wantDisposition = validators.DispositionApproved + } + if len(entry.LLMValidatorResults) != 1 || entry.LLMValidatorResults[0].Approved != (index == 1) || entry.LLMValidatorResults[0].Disposition != wantDisposition { t.Errorf("lost batch %d decision: %+v", index, entry) } } - if candidates[0].Disposition != correctionDispositionRejected || candidates[1].Disposition != correctionDispositionInterrupted || candidates[2].Disposition != correctionDispositionInterrupted { + if candidates[0].Disposition != correctionDispositionUnavailable || candidates[1].Disposition != correctionDispositionInterrupted || candidates[2].Disposition != correctionDispositionInterrupted { t.Fatalf("wrong partial dispositions: %+v", ledger) } } diff --git a/internal/framework/processreport/report.go b/internal/framework/processreport/report.go index 50c4b85..3927f59 100644 --- a/internal/framework/processreport/report.go +++ b/internal/framework/processreport/report.go @@ -151,7 +151,7 @@ func mapRecovery(unit runner.WorkUnitOutcome, operation outcomes.Operation) repo return reporting.RecoveryRecord{StageIndex: unit.ModuleIndex, ModuleKey: unit.ModuleKey, ModuleInstance: unit.ModuleInstance, ChunkIndex: unit.Section.Index, StartSegmentID: unit.Section.StartSegmentID, EndSegmentID: unit.Section.EndSegmentID, Operation: string(operation.Reference.Kind), Validator: operation.Reference.ValidatorName, BatchIndex: operation.Reference.BatchIndex, - Reasons: reasons, Message: "Model operation succeeded after recovery or proposal filtering.", + Reasons: reasons, Message: "Model operation succeeded after recovery or response filtering.", ExecutionCounts: mapCounts(operation.Counts), FilteredProposals: filtered} } @@ -195,7 +195,7 @@ func safeMessage(reason outcomes.ReasonCode) string { case outcomes.ReasonValidationInputTooLarge: return "Validation input exceeded its size limit." case outcomes.ReasonValidationDecisionsInvalid: - return "Validation decisions were unusable." + return "Some required validation decisions were unavailable." case outcomes.ReasonCanceled: return "Processing was canceled." case outcomes.ReasonSetupFailure: diff --git a/internal/framework/processreport/report_test.go b/internal/framework/processreport/report_test.go index 0d5714d..cde82fd 100644 --- a/internal/framework/processreport/report_test.go +++ b/internal/framework/processreport/report_test.go @@ -57,6 +57,61 @@ func TestBuildPartialUnitsAndCombinedRecovery(t *testing.T) { } } +func TestBuildReportsUnavailableDecisionsAndFilteredExtras(t *testing.T) { + started := time.Date(2026, 9, 25, 12, 0, 0, 0, time.UTC) + completed := started.Add(time.Second) + failedBatch, cleanBatch := 0, 1 + output := &runner.RunOutput{InventoryAvailable: true, WorkUnits: []runner.WorkUnitOutcome{{ + ModuleKey: "grammar", ModuleInstance: "grammar", Section: contracts.SectionMetadata{Index: 0, StartSegmentID: 1, EndSegmentID: 1}, + State: outcomes.WorkCompleted, CorrectionsApplied: 1, Operations: []outcomes.Operation{ + {Reference: outcomes.Reference{Kind: outcomes.OperationProposal}, Succeeded: true}, + {Reference: outcomes.Reference{Kind: outcomes.OperationValidation, ValidatorName: "editorial_review", BatchIndex: &failedBatch}, Counts: outcomes.ExecutionCounts{Available: true, ProviderAttempts: 2, StructuralRepairs: 1}, FailureReason: outcomes.ReasonValidationDecisionsInvalid, ProposalIndices: []int{3, 1}}, + {Reference: outcomes.Reference{Kind: outcomes.OperationValidation, ValidatorName: "editorial_review", BatchIndex: &cleanBatch}, Counts: outcomes.ExecutionCounts{Available: true, ProviderAttempts: 1}, Succeeded: true, RecoveryReasons: []outcomes.RecoveryReason{outcomes.RecoveryValidationDecisionsFiltered}}, + }, + }}} + snapshot := Build(BuildInput{RunID: "partial", Status: reporting.StatusCompleted, StartedAt: started, CompletedAt: &completed, + ModuleSpecs: []contracts.ModuleRunSpec{{ModuleKey: "grammar", InstanceName: "grammar"}}, RunOutput: output}) + if snapshot.Summary.WorkUnitsWithErrors == nil || *snapshot.Summary.WorkUnitsWithErrors != 1 || *snapshot.Summary.CorrectionsApplied != 1 || *snapshot.Summary.Recoveries != 1 || *snapshot.Summary.ProposalResponsesWithFiltering != 0 || *snapshot.Summary.LLMOperations != 3 { + t.Fatalf("summary: %+v", snapshot.Summary) + } + if len(snapshot.Errors) != 1 || len(snapshot.Errors[0].Failures) != 1 || !reflect.DeepEqual(snapshot.Errors[0].Failures[0].ProposalIndices, []int{1, 3}) || snapshot.Errors[0].Failures[0].Reason != string(outcomes.ReasonValidationDecisionsInvalid) || snapshot.Errors[0].Failures[0].ExecutionCounts.StructuralRepairs == nil || *snapshot.Errors[0].Failures[0].ExecutionCounts.StructuralRepairs != 1 { + t.Fatalf("failure attribution: %+v", snapshot.Errors) + } + if len(snapshot.Recoveries) != 1 || !reflect.DeepEqual(snapshot.Recoveries[0].Reasons, []string{string(outcomes.RecoveryValidationDecisionsFiltered)}) || len(snapshot.Recoveries[0].FilteredProposals) != 0 { + t.Fatalf("recovery attribution: %+v", snapshot.Recoveries) + } + for _, document := range []any{snapshot.ErrorsDocument(), snapshot.RecoveriesDocument()} { + encoded, err := reporting.YAML(document) + if err != nil { + t.Fatal(err) + } + var decoded map[string]any + if err := yaml.Unmarshal(encoded, &decoded); err != nil { + t.Fatal(err) + } + if decoded["version"] != 1 || decoded["status"] != "completed" { + t.Fatalf("YAML envelope: %#v", decoded) + } + summary := decoded["summary"].(map[string]any) + if summary["work_units_with_errors"] != 1 || summary["recoveries"] != 1 || summary["proposal_responses_with_filtering"] != 0 { + t.Fatalf("YAML summary: %#v", summary) + } + if errors, ok := decoded["errors"].([]any); ok { + failure := errors[0].(map[string]any)["failures"].([]any)[0].(map[string]any) + if !reflect.DeepEqual(failure["proposal_indices"], []any{1, 3}) { + t.Fatalf("YAML failure indices: %#v", failure) + } + } + if recoveries, ok := decoded["recoveries"].([]any); ok { + recovery := recoveries[0].(map[string]any) + filtered, hasFiltered := recovery["filtered_proposals"] + if !reflect.DeepEqual(recovery["reasons"], []any{string(outcomes.RecoveryValidationDecisionsFiltered)}) || hasFiltered && len(filtered.([]any)) != 0 { + t.Fatalf("YAML recovery: %#v", recovery) + } + } + } +} + func TestBuildInitializationNoWorkAndRunFailure(t *testing.T) { start := time.Now().UTC() initial := Build(BuildInput{RunID: "r", Status: reporting.StatusInProgress, StartedAt: start}) diff --git a/internal/framework/runner/runner.go b/internal/framework/runner/runner.go index 51f3ada..58d0b2f 100644 --- a/internal/framework/runner/runner.go +++ b/internal/framework/runner/runner.go @@ -65,24 +65,26 @@ type WorkUnitOutcome struct { } type ValidatorDecisionRecord struct { - ValidatorName string `json:"validator_name"` - ProposalIndex int `json:"proposal_index"` - Approved bool `json:"approved"` - ReasonCode string `json:"reason_code"` - Message string `json:"message"` - DiagnosticArtifactPath string `json:"diagnostic_artifact_path,omitempty"` + ValidatorName string `json:"validator_name"` + ProposalIndex int `json:"proposal_index"` + Approved bool `json:"approved"` + Disposition validators.DecisionDisposition `json:"disposition"` + ReasonCode string `json:"reason_code"` + Message string `json:"message"` + DiagnosticArtifactPath string `json:"diagnostic_artifact_path,omitempty"` } type ValidatorRejectedChange struct { - ValidatorName string `json:"validator_name"` - ProposalIndex int `json:"proposal_index"` - ModuleKey string `json:"module_key"` - ModuleInstance string `json:"module_instance"` - TargetSegmentID int `json:"target_segment_id"` - OriginalText string `json:"original_text"` - CorrectedText string `json:"corrected_text"` - ReasonCode string `json:"reason_code"` - Message string `json:"message"` + ValidatorName string `json:"validator_name"` + ProposalIndex int `json:"proposal_index"` + ModuleKey string `json:"module_key"` + ModuleInstance string `json:"module_instance"` + TargetSegmentID int `json:"target_segment_id"` + OriginalText string `json:"original_text"` + CorrectedText string `json:"corrected_text"` + Disposition validators.DecisionDisposition `json:"disposition"` + ReasonCode string `json:"reason_code"` + Message string `json:"message"` } // RunInput is the deterministic runner input. @@ -673,6 +675,7 @@ func validateSectionCandidates(ctx context.Context, input validateSectionCandida ValidatorName: validator.Name(), ProposalIndex: d.ProposalIndex, Approved: d.Approved, + Disposition: d.Disposition, ReasonCode: d.ReasonCode, Message: d.Message, DiagnosticArtifactPath: d.DiagnosticArtifactPath, @@ -695,6 +698,7 @@ func validateSectionCandidates(ctx context.Context, input validateSectionCandida TargetSegmentID: p.TargetSegmentID, OriginalText: p.OriginalText, CorrectedText: p.CorrectedText, + Disposition: d.Disposition, ReasonCode: d.ReasonCode, Message: d.Message, }) diff --git a/internal/framework/runner/runner_test.go b/internal/framework/runner/runner_test.go index b1d4c50..74c90da 100644 --- a/internal/framework/runner/runner_test.go +++ b/internal/framework/runner/runner_test.go @@ -1067,6 +1067,10 @@ func TestRunnerReconciledSurvivorsCompleteLaterValidation(t *testing.T) { if unit.CorrectionsApplied != 1 || len(unit.Operations) != 2 || !unit.Operations[0].Succeeded || unit.Operations[1].FailureReason != outcomes.ReasonValidationDecisionsInvalid || len(unit.Operations[1].ProposalIndices) != 1 || unit.Operations[1].ProposalIndices[0] != 1 || len(unit.Operations[1].RecoveryReasons) != 0 { t.Fatalf("operation accounting: %+v", unit) } + moduleResult := out.ModuleResults[0] + if len(moduleResult.ValidatorDecisions) != 5 || len(moduleResult.ValidatorRejected) != 2 || moduleResult.ValidatorDecisions[2].Disposition != validators.DispositionUnavailable || moduleResult.ValidatorRejected[0].Disposition != validators.DispositionUnavailable || moduleResult.ValidatorRejected[1].Disposition != validators.DispositionRejected { + t.Fatalf("validator dispositions: %+v", moduleResult) + } } func TestRunnerLLMValidatorMissingDecisionRejectsBatch(t *testing.T) { diff --git a/internal/framework/validators/llm_validators_test.go b/internal/framework/validators/llm_validators_test.go index 1ef087a..b45bbef 100644 --- a/internal/framework/validators/llm_validators_test.go +++ b/internal/framework/validators/llm_validators_test.go @@ -25,6 +25,7 @@ type fakeStructuredLLMClient struct { type captureValidationDiagnosticsWriter struct { err error + path string lastRequestMetadata any lastResponsePayload any } @@ -35,7 +36,7 @@ func (w *captureValidationDiagnosticsWriter) WriteInteraction(stage string, requ _ = errorPayload w.lastRequestMetadata = requestMetadata w.lastResponsePayload = responsePayload - return InteractionArtifacts{}, w.err + return InteractionArtifacts{ResponsePayloadPath: w.path}, w.err } func (f *fakeStructuredLLMClient) CompleteStructured(ctx context.Context, req StructuredCompletionRequest, out any) (StructuredCompletionResponse, error) { @@ -233,7 +234,7 @@ func TestLLMBackedValidatorReconcilesDecisionIdentities(t *testing.T) { } { t.Run(tc.name, func(t *testing.T) { client := &fakeStructuredLLMClient{responses: []LLMValidationResponse{{Validations: tc.entries}}} - writer := &captureValidationDiagnosticsWriter{} + writer := &captureValidationDiagnosticsWriter{path: "validation-response.json"} v, _ := NewLLMBackedValidator("editorial_review", LLMValidatorTypeEditorialReview) req := makeReq([]proposals.EnrichedCorrectionProposal{mk(7, "gestures", "Jesters"), mk(11, "gestures", "actors"), mk(18, "gestures", "figures")}) req.LLMClient = client @@ -244,7 +245,7 @@ func TestLLMBackedValidatorReconcilesDecisionIdentities(t *testing.T) { } for i, want := range tc.want { got := result.Decisions[i] - if got.ProposalIndex != want.index || got.Approved != want.approved || got.ReasonCode != want.reason || (got.Disposition == DispositionUnavailable) != (want.reason == ReasonMissingValidationDecision || want.reason == ReasonDuplicateValidationDecision || want.reason == ReasonInvalidValidationDecision) { + if got.ProposalIndex != want.index || got.Approved != want.approved || got.ReasonCode != want.reason || got.DiagnosticArtifactPath != writer.path || (got.Disposition == DispositionUnavailable) != (want.reason == ReasonMissingValidationDecision || want.reason == ReasonDuplicateValidationDecision || want.reason == ReasonInvalidValidationDecision) { t.Fatalf("decision[%d]=%+v want=%+v", i, got, want) } if got.ReasonCode == "llm_rejected" && got.Message != "editorial concern" { @@ -262,6 +263,10 @@ func TestLLMBackedValidatorReconcilesDecisionIdentities(t *testing.T) { t.Fatalf("recovery reason=%+v", op) } payload := writer.lastResponsePayload.(map[string]any) + decoded := payload["decoded"].(LLMValidationResponse) + if len(decoded.Validations) != len(tc.entries) { + t.Fatalf("supplied model decisions lost from diagnostic: %+v", decoded) + } reconciliation := payload["reconciliation"].(validationReconciliation) if !equalIndices(reconciliation.UnavailableIndices, tc.unavailable) || !equalIndices(reconciliation.IgnoredIndices, tc.ignored) || len(reconciliation.Decisions) != len(tc.want) { t.Fatalf("diagnostic reconciliation=%+v", reconciliation)