eric/audita
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 6 Wiki Activity

Track unavailable validator decisions in reports and debug records

Browse Source
This commit is contained in:
Eric Rakestraw
2026-09-25 11:28:44 +00:00
parent d726ea8f6d
commit 22f0932389
11 changed files with 169 additions and 38 deletions
Download Patch File Download Diff File Expand all files Collapse all files

18
docs/integrations/output-files.md
View File

@@ -11,7 +11,7 @@ With reporting enabled, Audita produces three ordinary files:
- `errors.yml`: affected work units and run-level failures;
- `recoveries.yml`: successful logical model operations with repair, service
retry, or defective-proposal filtering;
retry, or response filtering;
- `report.md`: a concise overview and complete per-chunk matrices.
The YAML documents have Audita-owned schemas `audita-errors` and
@@ -55,6 +55,9 @@ work unit form one error record. `scope: run` instead records phase, safe reason
and message, optional origin, and output path where relevant. It can describe
a fatal error or secondary output-write failure without inventing a failed
model work unit. Raw provider errors and transcript text are absent.
For `validation_decisions_invalid`, proposal indices identify only candidates
whose required decisions were unavailable. Other candidates in the same batch
may still be approved or rejected normally.
Each recovery identifies its work unit and successful operation, with sorted
`reasons`, a safe message, available execution counts, and sorted
@@ -65,6 +68,11 @@ ordinary editorial rejection, and an apply-time conflict caused by an earlier
accepted correction are not filtering recoveries. Failed operations are errors,
not recoveries. The initial chunk identity remains stable even if earlier
modules change its text.
`validation_decisions_filtered` records unexpected validator decision indices
ignored from a successful validation batch. It may accompany repair or service
recovery reasons, but it does not add `filtered_proposals` or increment
`proposal_responses_with_filtering`. A batch with unavailable requested decisions
is an error even when unexpected indices were also ignored.
## Markdown Overview
@@ -91,8 +99,14 @@ These files are diagnostic evidence, not an alternate ordinary report schema.
Each candidate-backed ledger entry includes `proposed_corrected_text`, the
literal replacement span, even when it is `""` for a deletion. Applied entries
also include `applied_corrected_text`, likewise `""` for an applied deletion.
Rejected, skipped, and interrupted entries omit the applied field; a module
Rejected, unavailable, skipped, and interrupted entries omit the applied field; a module
failure without a candidate omits both. These fields are replacement spans,
not full resulting-segment text. Deterministic decision records now identify
the blank-result guard as `non_empty_resulting_segment` (formerly
`non_empty_corrected_text`); its `empty_resulting_segment` reason is unchanged.
Validator decision records carry `disposition`: `approved`, `rejected`, or
`unavailable`. A missing or duplicate required decision is `unavailable`, not
an editorial rejection. The candidate ledger uses the same distinction in its
top-level disposition. Interaction response artifacts retain supplied model
decisions and the reconciliation result, including unavailable and ignored
indices; unexpected indices never create candidate ledger entries.

11
docs/internal/diagnostics-reporting.md
View File

@@ -15,6 +15,11 @@ Audita's versioned YAML DTOs, fixed Markdown rendering, and brief CLI
completion formatter. YAML redaction walks serialized string nodes so numeric
and boolean values retain their types. Ordinary reports omit raw causes,
transcript text, complete decisions, and token/timing payloads.
Validation operations with unavailable decisions report only affected requested
indices, even when other candidates in the batch continue. Ignored extra
validator entries yield `validation_decisions_filtered` only for a successful
operation; they do not count as filtered correction proposals. Failed operations
retain their execution counts and never produce a recovery record.
`internal/core/diagnostics.OutputSession` owns active destination checks,
transcript preflight, report initialization, regular-file replacement, and direct
@@ -40,6 +45,12 @@ Debug write failures reach the CLI through an explicit concurrency-safe diagnost
callback and produce redacted stderr messages without aborting processing.
Completed validator decisions returned with a fatal batch error remain available
to the debug ledger; they never authorize applying an incompletely validated edit.
Runner decision records carry the validator's explicit approved, rejected, or
unavailable disposition into the ledger. Unavailable candidates have a distinct
top-level ledger disposition, while fatal interruptions remain interrupted when
no rejection or unavailable decision was returned. Successful validation
interaction payloads contain the full decoded response, final content, and
reconciliation outcomes; ignored indices do not become ledger candidates.
Every debug writer uses all configured backend secrets for redaction, including
unselected backend credentials and source snapshots. Debug can still contain
sensitive transcript and prompt content.

6
docs/roadmap/implementation.md
View File

@@ -192,6 +192,12 @@ successful extra filtering. No extra model generation is introduced.
## Stage 2 — Carry Dispositions Through Debug and Report Partial Outcomes
**Completed:** 2026-09-25. Runner and ledger records carry explicit
dispositions; ordinary reports distinguish unavailable decision errors from
extra-index filtering recoveries. Focused report, ledger, runner, and YAML
checks passed with `go test ./internal/framework/runner
./internal/framework/processreport ./internal/core/reporting`.
**Work**
1. Add and populate disposition fields through the runner and ledger as selected

18
internal/framework/processreport/correction_ledger.go
View File

@@ -6,12 +6,14 @@ import (
"sort"
"gitea.maximumdirect.net/eric/audita/internal/framework/runner"
"gitea.maximumdirect.net/eric/audita/internal/framework/validators"
validatormetadata "gitea.maximumdirect.net/eric/audita/internal/validators/metadata"
)
const (
correctionDispositionApplied = "applied"
correctionDispositionRejected = "rejected"
correctionDispositionUnavailable = "unavailable"
correctionDispositionSkipped = "skipped"
correctionDispositionFailed = "failed"
correctionDispositionInterrupted = "interrupted"
@@ -40,10 +42,11 @@ type CorrectionLedgerEntry struct {
}
type LedgerValidatorDecisionRecord struct {
ValidatorKey string `json:"validator_key"`
Approved bool `json:"approved"`
ReasonCode string `json:"reason_code"`
Message string `json:"message,omitempty"`
ValidatorKey string `json:"validator_key"`
Approved bool `json:"approved"`
Disposition validators.DecisionDisposition `json:"disposition"`
ReasonCode string `json:"reason_code"`
Message string `json:"message,omitempty"`
}
func BuildCorrectionLedger(input CorrectionLedgerInput) []CorrectionLedgerEntry {
@@ -101,6 +104,10 @@ func BuildCorrectionLedger(input CorrectionLedgerInput) []CorrectionLedgerEntry
})
}
for _, rejection := range module.ValidatorRejected {
disposition := correctionDispositionRejected
if rejection.Disposition == validators.DispositionUnavailable {
disposition = correctionDispositionUnavailable
}
seen[identity(module.ModuleInstance, rejection.ProposalIndex)] = true
entries = append(entries, CorrectionLedgerEntry{
RunID: runID,
@@ -111,7 +118,7 @@ func BuildCorrectionLedger(input CorrectionLedgerInput) []CorrectionLedgerEntry
OriginalText: rejection.OriginalText,
ProposedCorrectedText: &rejection.CorrectedText,
ReplacementPolicy: string(module.ReplacementPolicy),
Disposition: correctionDispositionRejected,
Disposition: disposition,
DispositionReasonCode: rejection.ReasonCode,
DispositionMessage: rejection.Message,
DeterministicValidatorResults: filterLedgerDecisions(decisionsByProposal[rejection.ProposalIndex], false),
@@ -184,6 +191,7 @@ func filterLedgerDecisions(in []runner.ValidatorDecisionRecord, wantLLM bool) []
out = append(out, LedgerValidatorDecisionRecord{
ValidatorKey: decision.ValidatorName,
Approved: decision.Approved,
Disposition: decision.Disposition,
ReasonCode: decision.ReasonCode,
Message: decision.Message,
})

34
internal/framework/processreport/correction_ledger_test.go
View File

@@ -6,6 +6,7 @@ import (
"gitea.maximumdirect.net/eric/audita/internal/framework/proposals"
"gitea.maximumdirect.net/eric/audita/internal/framework/runner"
"gitea.maximumdirect.net/eric/audita/internal/framework/validators"
)
func TestBuildCorrectionLedgerClassifiesValidatorDecisionsFromCanonicalMetadata(t *testing.T) {
@@ -16,8 +17,8 @@ func TestBuildCorrectionLedgerClassifiesValidatorDecisionsFromCanonicalMetadata(
ModuleInstance: "glossary",
ReplacementPolicy: proposals.ReplacementPolicyReplaceAll,
ValidatorDecisions: []runner.ValidatorDecisionRecord{
{ValidatorName: "proposal_shape", ProposalIndex: 3, Approved: true, ReasonCode: "approved"},
{ValidatorName: "spoken_form_plausibility", ProposalIndex: 3, Approved: true, ReasonCode: "approved"},
{ValidatorName: "proposal_shape", ProposalIndex: 3, Approved: true, Disposition: validators.DispositionApproved, ReasonCode: "approved"},
{ValidatorName: "spoken_form_plausibility", ProposalIndex: 3, Approved: true, Disposition: validators.DispositionApproved, ReasonCode: "approved"},
},
AppliedChanges: []proposals.AppliedChange{
{
@@ -44,10 +45,10 @@ func TestBuildCorrectionLedgerClassifiesValidatorDecisionsFromCanonicalMetadata(
if entry.RunID != "audita-run-id" || entry.Disposition != "applied" || entry.AppliedCorrectedText == nil || *entry.AppliedCorrectedText != "Jesters" {
t.Fatalf("unexpected applied ledger entry: %+v", entry)
}
if len(entry.DeterministicValidatorResults) != 1 || entry.DeterministicValidatorResults[0].ValidatorKey != "proposal_shape" {
if len(entry.DeterministicValidatorResults) != 1 || entry.DeterministicValidatorResults[0].ValidatorKey != "proposal_shape" || entry.DeterministicValidatorResults[0].Disposition != validators.DispositionApproved {
t.Fatalf("unexpected deterministic decision split: %+v", entry.DeterministicValidatorResults)
}
if len(entry.LLMValidatorResults) != 1 || entry.LLMValidatorResults[0].ValidatorKey != "spoken_form_plausibility" {
if len(entry.LLMValidatorResults) != 1 || entry.LLMValidatorResults[0].ValidatorKey != "spoken_form_plausibility" || entry.LLMValidatorResults[0].Disposition != validators.DispositionApproved {
t.Fatalf("unexpected llm-backed decision split: %+v", entry.LLMValidatorResults)
}
}
@@ -76,6 +77,7 @@ func TestBuildCorrectionLedgerPreservesDispositionPolicy(t *testing.T) {
TargetSegmentID: 8,
OriginalText: "before",
CorrectedText: "after",
Disposition: validators.DispositionRejected,
ReasonCode: "protected_term",
Message: "blocked",
},
@@ -117,7 +119,7 @@ func TestCorrectionLedgerIncludesUnappliedCandidateAfterFatalValidation(t *testi
output := &runner.RunOutput{
Candidates: []proposals.EnrichedCorrectionProposal{{CorrectionProposal: proposals.CorrectionProposal{TargetSegmentID: 7, OriginalText: "old", CorrectedText: "new"}, ProposalMetadata: proposals.ProposalMetadata{ModuleKey: "grammar", ModuleInstance: "grammar", ProposalIndex: 2}}},
ModuleResults: []runner.ModuleResult{{ModuleKey: "grammar", ModuleInstance: "grammar", Status: runner.ModuleStatusFailed,
ValidatorDecisions: []runner.ValidatorDecisionRecord{{ValidatorName: "spoken_form_plausibility", ProposalIndex: 2, Approved: false, ReasonCode: "transport_failed"}}}},
ValidatorDecisions: []runner.ValidatorDecisionRecord{{ValidatorName: "spoken_form_plausibility", ProposalIndex: 2, Approved: false, Disposition: validators.DispositionUnavailable, ReasonCode: "transport_failed"}}}},
}
ledger := BuildCorrectionLedger(CorrectionLedgerInput{RunOutput: output})
if len(ledger) != 2 {
@@ -127,7 +129,7 @@ func TestCorrectionLedgerIncludesUnappliedCandidateAfterFatalValidation(t *testi
if entry.ProposalIndex != 2 {
continue
}
if entry.Disposition != correctionDispositionInterrupted || entry.OriginalText != "old" || len(entry.LLMValidatorResults) != 1 {
if entry.Disposition != correctionDispositionInterrupted || entry.OriginalText != "old" || len(entry.LLMValidatorResults) != 1 || entry.LLMValidatorResults[0].Disposition != validators.DispositionUnavailable {
t.Fatalf("candidate evidence lost: %+v", entry)
}
return
@@ -140,9 +142,16 @@ func TestCorrectionLedgerSerializesEmptyReplacementByDisposition(t *testing.T) {
ModuleResults: []runner.ModuleResult{
{
ModuleKey: "spoken_word", ModuleInstance: "spoken_word", ReplacementPolicy: proposals.ReplacementPolicyRequireUnique,
AppliedChanges: []proposals.AppliedChange{{ProposalIndex: 0, TargetSegmentID: 1, OriginalText: "uh ", CorrectedText: ""}},
ValidatorRejected: []runner.ValidatorRejectedChange{{ProposalIndex: 1, TargetSegmentID: 2, OriginalText: "uh ", CorrectedText: "", ReasonCode: "llm_rejected"}},
SkippedChanges: []proposals.SkippedChange{{ProposalIndex: 2, TargetSegmentID: 3, OriginalText: "uh ", CorrectedText: "", SkipReason: proposals.SkipReasonEmptyResultingText}},
AppliedChanges: []proposals.AppliedChange{{ProposalIndex: 0, TargetSegmentID: 1, OriginalText: "uh ", CorrectedText: ""}},
ValidatorRejected: []runner.ValidatorRejectedChange{
{ProposalIndex: 1, TargetSegmentID: 2, OriginalText: "uh ", CorrectedText: "", Disposition: validators.DispositionRejected, ReasonCode: "llm_rejected"},
{ProposalIndex: 4, TargetSegmentID: 5, OriginalText: "uh ", CorrectedText: "", Disposition: validators.DispositionUnavailable, ReasonCode: validators.ReasonMissingValidationDecision},
},
ValidatorDecisions: []runner.ValidatorDecisionRecord{
{ValidatorName: "editorial_review", ProposalIndex: 1, Disposition: validators.DispositionRejected, ReasonCode: "llm_rejected"},
{ValidatorName: "editorial_review", ProposalIndex: 4, Disposition: validators.DispositionUnavailable, ReasonCode: validators.ReasonMissingValidationDecision},
},
SkippedChanges: []proposals.SkippedChange{{ProposalIndex: 2, TargetSegmentID: 3, OriginalText: "uh ", CorrectedText: "", SkipReason: proposals.SkipReasonEmptyResultingText}},
},
{ModuleKey: "grammar", ModuleInstance: "grammar", Status: runner.ModuleStatusFailed, ErrorMessage: "failed"},
},
@@ -167,6 +176,7 @@ func TestCorrectionLedgerSerializesEmptyReplacementByDisposition(t *testing.T) {
}{
{correctionDispositionApplied, true, true},
{correctionDispositionRejected, true, false},
{correctionDispositionUnavailable, true, false},
{correctionDispositionSkipped, true, false},
{correctionDispositionInterrupted, true, false},
{correctionDispositionFailed, false, false},
@@ -180,5 +190,11 @@ func TestCorrectionLedgerSerializesEmptyReplacementByDisposition(t *testing.T) {
if proposedPresent != tc.proposed || appliedPresent != tc.applied || (proposedPresent && string(proposed) != `""`) || (appliedPresent && string(applied) != `""`) {
t.Fatalf("incorrect serialized replacement fields for %s: %+v", tc.disposition, fields)
}
if tc.disposition == correctionDispositionUnavailable || tc.disposition == correctionDispositionRejected {
var decisions []LedgerValidatorDecisionRecord
if err := json.Unmarshal(fields["llm_validator_decisions"], &decisions); err != nil || len(decisions) != 1 || string(decisions[0].Disposition) != tc.disposition {
t.Fatalf("incorrect validator disposition for %s: %+v, %v", tc.disposition, decisions, err)
}
}
}
}

12
internal/framework/processreport/partial_validation_test.go
View File

@@ -51,6 +51,10 @@ func (c *interruptedValidationClient) CompleteStructured(_ context.Context, req
return structuredoutput.Response{}, &structuredoutput.Failure{Class: structuredoutput.FailureFatal, Reason: outcomes.ReasonSetupFailure, Message: "validation interrupted", Cause: c.cause}
}
index := *req.Context.BatchIndex
if index == 0 {
*target.(*validators.LLMValidationResponse) = validators.LLMValidationResponse{}
return structuredoutput.Response{}, nil
}
*target.(*validators.LLMValidationResponse) = validators.LLMValidationResponse{Validations: []validators.LLMValidationDecision{{CorrectionIndex: index, Approved: index == 1, Confidence: 1, Reason: "completed assessment"}}}
return structuredoutput.Response{}, nil
}
@@ -89,11 +93,15 @@ func TestFatalBatchRetainsEarlierDecisionsInLedger(t *testing.T) {
}
for index := 0; index < 2; index++ {
entry := candidates[index]
if len(entry.LLMValidatorResults) != 1 || entry.LLMValidatorResults[0].Message != "completed assessment" || entry.LLMValidatorResults[0].Approved != (index == 1) {
wantDisposition := validators.DispositionUnavailable
if index == 1 {
wantDisposition = validators.DispositionApproved
}
if len(entry.LLMValidatorResults) != 1 || entry.LLMValidatorResults[0].Approved != (index == 1) || entry.LLMValidatorResults[0].Disposition != wantDisposition {
t.Errorf("lost batch %d decision: %+v", index, entry)
}
}
if candidates[0].Disposition != correctionDispositionRejected || candidates[1].Disposition != correctionDispositionInterrupted || candidates[2].Disposition != correctionDispositionInterrupted {
if candidates[0].Disposition != correctionDispositionUnavailable || candidates[1].Disposition != correctionDispositionInterrupted || candidates[2].Disposition != correctionDispositionInterrupted {
t.Fatalf("wrong partial dispositions: %+v", ledger)
}
}

4
internal/framework/processreport/report.go
View File

@@ -151,7 +151,7 @@ func mapRecovery(unit runner.WorkUnitOutcome, operation outcomes.Operation) repo
return reporting.RecoveryRecord{StageIndex: unit.ModuleIndex, ModuleKey: unit.ModuleKey, ModuleInstance: unit.ModuleInstance,
ChunkIndex: unit.Section.Index, StartSegmentID: unit.Section.StartSegmentID, EndSegmentID: unit.Section.EndSegmentID,
Operation: string(operation.Reference.Kind), Validator: operation.Reference.ValidatorName, BatchIndex: operation.Reference.BatchIndex,
Reasons: reasons, Message: "Model operation succeeded after recovery or proposal filtering.",
Reasons: reasons, Message: "Model operation succeeded after recovery or response filtering.",
ExecutionCounts: mapCounts(operation.Counts), FilteredProposals: filtered}
}
@@ -195,7 +195,7 @@ func safeMessage(reason outcomes.ReasonCode) string {
case outcomes.ReasonValidationInputTooLarge:
return "Validation input exceeded its size limit."
case outcomes.ReasonValidationDecisionsInvalid:
return "Validation decisions were unusable."
return "Some required validation decisions were unavailable."
case outcomes.ReasonCanceled:
return "Processing was canceled."
case outcomes.ReasonSetupFailure:

55
internal/framework/processreport/report_test.go
View File

@@ -57,6 +57,61 @@ func TestBuildPartialUnitsAndCombinedRecovery(t *testing.T) {
}
}
func TestBuildReportsUnavailableDecisionsAndFilteredExtras(t *testing.T) {
started := time.Date(2026, 9, 25, 12, 0, 0, 0, time.UTC)
completed := started.Add(time.Second)
failedBatch, cleanBatch := 0, 1
output := &runner.RunOutput{InventoryAvailable: true, WorkUnits: []runner.WorkUnitOutcome{{
ModuleKey: "grammar", ModuleInstance: "grammar", Section: contracts.SectionMetadata{Index: 0, StartSegmentID: 1, EndSegmentID: 1},
State: outcomes.WorkCompleted, CorrectionsApplied: 1, Operations: []outcomes.Operation{
{Reference: outcomes.Reference{Kind: outcomes.OperationProposal}, Succeeded: true},
{Reference: outcomes.Reference{Kind: outcomes.OperationValidation, ValidatorName: "editorial_review", BatchIndex: &failedBatch}, Counts: outcomes.ExecutionCounts{Available: true, ProviderAttempts: 2, StructuralRepairs: 1}, FailureReason: outcomes.ReasonValidationDecisionsInvalid, ProposalIndices: []int{3, 1}},
{Reference: outcomes.Reference{Kind: outcomes.OperationValidation, ValidatorName: "editorial_review", BatchIndex: &cleanBatch}, Counts: outcomes.ExecutionCounts{Available: true, ProviderAttempts: 1}, Succeeded: true, RecoveryReasons: []outcomes.RecoveryReason{outcomes.RecoveryValidationDecisionsFiltered}},
},
}}}
snapshot := Build(BuildInput{RunID: "partial", Status: reporting.StatusCompleted, StartedAt: started, CompletedAt: &completed,
ModuleSpecs: []contracts.ModuleRunSpec{{ModuleKey: "grammar", InstanceName: "grammar"}}, RunOutput: output})
if snapshot.Summary.WorkUnitsWithErrors == nil || *snapshot.Summary.WorkUnitsWithErrors != 1 || *snapshot.Summary.CorrectionsApplied != 1 || *snapshot.Summary.Recoveries != 1 || *snapshot.Summary.ProposalResponsesWithFiltering != 0 || *snapshot.Summary.LLMOperations != 3 {
t.Fatalf("summary: %+v", snapshot.Summary)
}
if len(snapshot.Errors) != 1 || len(snapshot.Errors[0].Failures) != 1 || !reflect.DeepEqual(snapshot.Errors[0].Failures[0].ProposalIndices, []int{1, 3}) || snapshot.Errors[0].Failures[0].Reason != string(outcomes.ReasonValidationDecisionsInvalid) || snapshot.Errors[0].Failures[0].ExecutionCounts.StructuralRepairs == nil || *snapshot.Errors[0].Failures[0].ExecutionCounts.StructuralRepairs != 1 {
t.Fatalf("failure attribution: %+v", snapshot.Errors)
}
if len(snapshot.Recoveries) != 1 || !reflect.DeepEqual(snapshot.Recoveries[0].Reasons, []string{string(outcomes.RecoveryValidationDecisionsFiltered)}) || len(snapshot.Recoveries[0].FilteredProposals) != 0 {
t.Fatalf("recovery attribution: %+v", snapshot.Recoveries)
}
for _, document := range []any{snapshot.ErrorsDocument(), snapshot.RecoveriesDocument()} {
encoded, err := reporting.YAML(document)
if err != nil {
t.Fatal(err)
}
var decoded map[string]any
if err := yaml.Unmarshal(encoded, &decoded); err != nil {
t.Fatal(err)
}
if decoded["version"] != 1 || decoded["status"] != "completed" {
t.Fatalf("YAML envelope: %#v", decoded)
}
summary := decoded["summary"].(map[string]any)
if summary["work_units_with_errors"] != 1 || summary["recoveries"] != 1 || summary["proposal_responses_with_filtering"] != 0 {
t.Fatalf("YAML summary: %#v", summary)
}
if errors, ok := decoded["errors"].([]any); ok {
failure := errors[0].(map[string]any)["failures"].([]any)[0].(map[string]any)
if !reflect.DeepEqual(failure["proposal_indices"], []any{1, 3}) {
t.Fatalf("YAML failure indices: %#v", failure)
}
}
if recoveries, ok := decoded["recoveries"].([]any); ok {
recovery := recoveries[0].(map[string]any)
filtered, hasFiltered := recovery["filtered_proposals"]
if !reflect.DeepEqual(recovery["reasons"], []any{string(outcomes.RecoveryValidationDecisionsFiltered)}) || hasFiltered && len(filtered.([]any)) != 0 {
t.Fatalf("YAML recovery: %#v", recovery)
}
}
}
}
func TestBuildInitializationNoWorkAndRunFailure(t *testing.T) {
start := time.Now().UTC()
initial := Build(BuildInput{RunID: "r", Status: reporting.StatusInProgress, StartedAt: start})

34
internal/framework/runner/runner.go
View File

@@ -65,24 +65,26 @@ type WorkUnitOutcome struct {
}
type ValidatorDecisionRecord struct {
ValidatorName string `json:"validator_name"`
ProposalIndex int `json:"proposal_index"`
Approved bool `json:"approved"`
ReasonCode string `json:"reason_code"`
Message string `json:"message"`
DiagnosticArtifactPath string `json:"diagnostic_artifact_path,omitempty"`
ValidatorName string `json:"validator_name"`
ProposalIndex int `json:"proposal_index"`
Approved bool `json:"approved"`
Disposition validators.DecisionDisposition `json:"disposition"`
ReasonCode string `json:"reason_code"`
Message string `json:"message"`
DiagnosticArtifactPath string `json:"diagnostic_artifact_path,omitempty"`
}
type ValidatorRejectedChange struct {
ValidatorName string `json:"validator_name"`
ProposalIndex int `json:"proposal_index"`
ModuleKey string `json:"module_key"`
ModuleInstance string `json:"module_instance"`
TargetSegmentID int `json:"target_segment_id"`
OriginalText string `json:"original_text"`
CorrectedText string `json:"corrected_text"`
ReasonCode string `json:"reason_code"`
Message string `json:"message"`
ValidatorName string `json:"validator_name"`
ProposalIndex int `json:"proposal_index"`
ModuleKey string `json:"module_key"`
ModuleInstance string `json:"module_instance"`
TargetSegmentID int `json:"target_segment_id"`
OriginalText string `json:"original_text"`
CorrectedText string `json:"corrected_text"`
Disposition validators.DecisionDisposition `json:"disposition"`
ReasonCode string `json:"reason_code"`
Message string `json:"message"`
}
// RunInput is the deterministic runner input.
@@ -673,6 +675,7 @@ func validateSectionCandidates(ctx context.Context, input validateSectionCandida
ValidatorName: validator.Name(),
ProposalIndex: d.ProposalIndex,
Approved: d.Approved,
Disposition: d.Disposition,
ReasonCode: d.ReasonCode,
Message: d.Message,
DiagnosticArtifactPath: d.DiagnosticArtifactPath,
@@ -695,6 +698,7 @@ func validateSectionCandidates(ctx context.Context, input validateSectionCandida
TargetSegmentID: p.TargetSegmentID,
OriginalText: p.OriginalText,
CorrectedText: p.CorrectedText,
Disposition: d.Disposition,
ReasonCode: d.ReasonCode,
Message: d.Message,
})

4
internal/framework/runner/runner_test.go
View File

@@ -1067,6 +1067,10 @@ func TestRunnerReconciledSurvivorsCompleteLaterValidation(t *testing.T) {
if unit.CorrectionsApplied != 1 || len(unit.Operations) != 2 || !unit.Operations[0].Succeeded || unit.Operations[1].FailureReason != outcomes.ReasonValidationDecisionsInvalid || len(unit.Operations[1].ProposalIndices) != 1 || unit.Operations[1].ProposalIndices[0] != 1 || len(unit.Operations[1].RecoveryReasons) != 0 {
t.Fatalf("operation accounting: %+v", unit)
}
moduleResult := out.ModuleResults[0]
if len(moduleResult.ValidatorDecisions) != 5 || len(moduleResult.ValidatorRejected) != 2 || moduleResult.ValidatorDecisions[2].Disposition != validators.DispositionUnavailable || moduleResult.ValidatorRejected[0].Disposition != validators.DispositionUnavailable || moduleResult.ValidatorRejected[1].Disposition != validators.DispositionRejected {
t.Fatalf("validator dispositions: %+v", moduleResult)
}
}
func TestRunnerLLMValidatorMissingDecisionRejectsBatch(t *testing.T) {

11
internal/framework/validators/llm_validators_test.go
View File

@@ -25,6 +25,7 @@ type fakeStructuredLLMClient struct {
type captureValidationDiagnosticsWriter struct {
err error
path string
lastRequestMetadata any
lastResponsePayload any
}
@@ -35,7 +36,7 @@ func (w *captureValidationDiagnosticsWriter) WriteInteraction(stage string, requ
_ = errorPayload
w.lastRequestMetadata = requestMetadata
w.lastResponsePayload = responsePayload
return InteractionArtifacts{}, w.err
return InteractionArtifacts{ResponsePayloadPath: w.path}, w.err
}
func (f *fakeStructuredLLMClient) CompleteStructured(ctx context.Context, req StructuredCompletionRequest, out any) (StructuredCompletionResponse, error) {
@@ -233,7 +234,7 @@ func TestLLMBackedValidatorReconcilesDecisionIdentities(t *testing.T) {
} {
t.Run(tc.name, func(t *testing.T) {
client := &fakeStructuredLLMClient{responses: []LLMValidationResponse{{Validations: tc.entries}}}
writer := &captureValidationDiagnosticsWriter{}
writer := &captureValidationDiagnosticsWriter{path: "validation-response.json"}
v, _ := NewLLMBackedValidator("editorial_review", LLMValidatorTypeEditorialReview)
req := makeReq([]proposals.EnrichedCorrectionProposal{mk(7, "gestures", "Jesters"), mk(11, "gestures", "actors"), mk(18, "gestures", "figures")})
req.LLMClient = client
@@ -244,7 +245,7 @@ func TestLLMBackedValidatorReconcilesDecisionIdentities(t *testing.T) {
}
for i, want := range tc.want {
got := result.Decisions[i]
if got.ProposalIndex != want.index || got.Approved != want.approved || got.ReasonCode != want.reason || (got.Disposition == DispositionUnavailable) != (want.reason == ReasonMissingValidationDecision || want.reason == ReasonDuplicateValidationDecision || want.reason == ReasonInvalidValidationDecision) {
if got.ProposalIndex != want.index || got.Approved != want.approved || got.ReasonCode != want.reason || got.DiagnosticArtifactPath != writer.path || (got.Disposition == DispositionUnavailable) != (want.reason == ReasonMissingValidationDecision || want.reason == ReasonDuplicateValidationDecision || want.reason == ReasonInvalidValidationDecision) {
t.Fatalf("decision[%d]=%+v want=%+v", i, got, want)
}
if got.ReasonCode == "llm_rejected" && got.Message != "editorial concern" {
@@ -262,6 +263,10 @@ func TestLLMBackedValidatorReconcilesDecisionIdentities(t *testing.T) {
t.Fatalf("recovery reason=%+v", op)
}
payload := writer.lastResponsePayload.(map[string]any)
decoded := payload["decoded"].(LLMValidationResponse)
if len(decoded.Validations) != len(tc.entries) {
t.Fatalf("supplied model decisions lost from diagnostic: %+v", decoded)
}
reconciliation := payload["reconciliation"].(validationReconciliation)
if !equalIndices(reconciliation.UnavailableIndices, tc.unavailable) || !equalIndices(reconciliation.IgnoredIndices, tc.ignored) || len(reconciliation.Decisions) != len(tc.want) {
t.Fatalf("diagnostic reconciliation=%+v", reconciliation)