audita/docs/roadmap/decision-set-reconciliation.md

Partial Success for Validator Decision Sets

Last updated: 2026-09-25

Status: Implemented. This document owns the target behavior and user intent. implementation.md owns concrete decisions, ordered work, and verification.

Purpose and User Intent

Preserve usable validator decisions when other entries in the same model response have missing, duplicate, or unexpected correction indices. One defective entry should not prevent unrelated candidates from completing their required validation. This extends Audita's existing partial-success approach without another model call.

Implement this simpler improvement before reconsidering guided domain recovery. Missing validation must never authorize applying a correction, but neither should it invalidate a different candidate's unambiguous approval or editorial rejection. The existing resilience design remains the foundation.

Historical Baseline

The reviewed application baseline is d2a6fcf:

• PromptKit validates the complete model response against Audita's embedded JSON Schema and performs its configured structural repairs. Only structurally valid, decoded responses reach domain decision mapping.
• mapLLMResponseToDecisions in llm_validators.go requires exactly one decision for every requested correction index. An empty decision list, missing index, duplicate index, or unexpected index returns an error instead of retaining usable decisions.
• LLMBackedValidator.Validate consequently marks the entire batch unavailable with validation_decisions_invalid. Other batches continue, but usable decisions within that batch are lost.
• The runner's cardinality checks enforce an internal contract: each candidate receives one outcome from each validator it reaches. These checks apply to Audita-owned results, not directly to raw model entries.
• Deterministic validators already filter individual proposals. Unpreviewable candidates and oversized validation inputs are isolated; recoverable provider failures remain batch-local; application skips individual conflicts.
• Ordinary reports distinguish failed operations from successful operations with recoveries. Several failed operations in one stage/chunk share one work-unit error record. The output contract owns the current counts and serialized representation.

The defect addressed here is the all-or-nothing domain mapping boundary, not validation itself or the runner's internal consistency checks.

Target Decision Semantics

Reconcile each structurally valid response against the indices actually requested in that particular validator batch. Correction indices, rather than segment IDs or positions in the response array, identify candidates. Several corrections may legitimately target the same segment.

Response condition	Target behavior
Exactly one valid decision for a requested index	Preserve its approval or rejection and associated decision evidence.
Requested index omitted	Produce an explicit validation-unavailable outcome for that candidate.
Requested index appears more than once	Treat that candidate's validation as unavailable; preserve unrelated decisions.
Index was not requested in this batch	Ignore the extraneous entry and record the handled defect.
Empty decision list for a nonempty request	Mark all requested candidates unavailable; do not fail the run.
Decisions appear in a different order	Match by identity and retain deterministic internal results.

Duplicate handling does not choose the first or last answer, nor infer a verdict by voting. Even identical duplicates follow the agreed duplicate-index handling; accepting them specially is not needed for this feature. Duplicate extraneous indices remain extraneous entries, not reasons to invalidate requested candidates.

An unavailable candidate remains unapplied and does not proceed as approved to later validators. It is distinct from a valid approved: false decision, which is ordinary editorial processing. Valid approvals continue through the remaining required validators and apply-time checks; this feature grants no bypass.

The mapper returns one internal outcome for every requested candidate and no outcome for an extraneous index. Preserve strict runner cardinality checks so application bugs are still detected. Do not convert genuine internal invariant failures, cancellation, or fatal setup failures into tolerated model defects.

Outcomes and Reporting

Keep existing work-unit aggregation and operation accounting:

• If any requested candidate lacks an unambiguous usable decision, record a validation-operation error identifying the affected requested indices. Other candidates' usable decisions from that operation remain effective. An operation error does not mean every candidate in its batch failed.
• If all requested candidates have usable decisions and extra entries were ignored, record a successful operation with a recovery for that handled defect. This remains a recovery even if all legitimate decisions are editorial rejections.
• If missing/duplicate requested decisions and ignored extras coexist, retain the operation error. Follow the existing rule that failed operations do not also produce recovery records; retain extra-entry details in debug evidence.
• A valid complete response with no handled defect adds neither error nor recovery. Valid editorial rejection alone never adds either category.
• Preserve actual structural repair and service-retry counts. On successful operations, consolidate applicable recovery reasons in one recovery record. Do not count reconciliation as another LLM operation or structural repair.

Error attribution must distinguish affected candidates from the rest of the batch. Debug evidence should preserve supplied decisions, ignored entries, and the reason an expected candidate became unavailable. Do not fabricate candidate ledger entries for unknown indices or mislabel missing decisions as deliberate editorial rejection. Ordinary reports must retain safe metadata without raw transcript or response text.

Use a decision-specific recovery reason rather than treating ignored validator entries as defective correction proposals. Concrete reason names and minimal data model changes belong in the implementation plan. Keep YAML, Markdown, and CLI counts consistent, with the existing reporting-disable and independent debug options.

Ownership and Target Architecture

Audita's validator layer reconciles structurally valid model responses into complete internal candidate outcomes. The runner preserves strict internal cardinality and advances only approved candidates. Reporting consumes operation facts without treating partial validation failure as loss of every decision. Debug evidence distinguishes unavailable validation from editorial rejection.

The schema/transport boundary remains with PromptKit; reconciliation needs no additional model call. Concrete helper structures, reason names, mapping changes, and verification steps belong in the implementation plan.

Scope Boundaries

PromptKit's whole-response structural validation and repair remain unchanged. A malformed entry, including an extraneous entry that violates the schema, may still invalidate the response and exhaust structural repairs before reconciliation can run. Ignoring schema-valid extras does not promise salvage from malformed JSON, missing required fields, wrong types, or out-of-range values.

General per-entry structural salvage is a separate design question, not part of this fix. Likewise, guided regeneration remains deferred for reconsideration after this change; no new retries, feedback prompts, budgets, upstream dependency changes, or operator controls are required here.

Preserve module scope, literal replacement behavior, fixed chunks, stage barriers, span-deletion safeguards, backend capacity limits, and fatal cancellation behavior. No additional restrictions on valid corrections or guarantees about model judgment are introduced. This work does not itself authorize tagging or publishing v2.0.0.

Acceptance Criteria

• A schema-valid response can retain usable decisions despite missing, duplicate, or unexpected indices elsewhere in the same batch.
• Only the affected requested candidates become unavailable; extras never create candidates or attach decisions to a different batch's candidates.
• Valid negative decisions remain editorial outcomes, and every applied correction completes all required validation.
• Internal validator results satisfy existing strict cardinality checks.
• Missing/duplicate requested decisions yield accurately attributed errors without losing valid work. Extra-only defects yield a consolidated recovery; mixed defects follow existing failed-operation reporting semantics.
• Final transcript, debug evidence, YAML reports, Markdown counts, and CLI summary agree about applied work and failures/recoveries.
• Reconciliation makes no additional provider calls. Existing structural failures, provider failures, and cancellation retain their documented handling.

Open Questions

No unresolved product-policy questions block implementation. Guided recovery and structural salvage remain separate future decisions.