Define partial success for validator decisions

docs/roadmap/decision-set-reconciliation.md

@@ -0,0 +1,190 @@
+# Partial Success for Validator Decision Sets
+
+**Last updated:** 2026-09-25
+
+**Status:** Accepted feature direction; implementation not started. Intended for
+consideration before v2.0.0. This document defines the target behavior and work
+needed; a staged implementation plan will follow separately.
+
+## Purpose and User Intent
+
+Preserve usable validator decisions when other entries in the same model response
+have missing, duplicate, or unexpected correction indices. One defective entry
+should not prevent unrelated candidates from completing their required validation.
+This extends Audita's existing partial-success approach without another model call.
+
+Implement this simpler improvement before reconsidering guided domain recovery.
+Missing validation must never authorize applying a correction, but neither should
+it invalidate a different candidate's unambiguous approval or editorial rejection.
+The existing [resilience design](resilience.md) remains the foundation.
+
+## Current Baseline
+
+The reviewed application baseline is `d2a6fcf`:
+
+- PromptKit validates the complete model response against Audita's embedded JSON
+  Schema and performs its configured structural repairs. Only structurally valid,
+  decoded responses reach domain decision mapping.
+- `mapLLMResponseToDecisions` in
+  [`llm_validators.go`](../../internal/framework/validators/llm_validators.go)
+  requires exactly one decision for every requested correction index. An empty
+  decision list, missing index, duplicate index, or unexpected index returns an
+  error instead of retaining usable decisions.
+- `LLMBackedValidator.Validate` consequently marks the entire batch unavailable
+  with `validation_decisions_invalid`. Other batches continue, but usable decisions
+  within that batch are lost.
+- The runner's cardinality checks enforce an internal contract: each candidate
+  receives one outcome from each validator it reaches. These checks apply to
+  Audita-owned results, not directly to raw model entries.
+- Deterministic validators already filter individual proposals. Unpreviewable
+  candidates and oversized validation inputs are isolated; recoverable provider
+  failures remain batch-local; application skips individual conflicts.
+- Ordinary reports distinguish failed operations from successful operations with
+  recoveries. Several failed operations in one stage/chunk share one work-unit
+  error record. The [output contract](../integrations/output-files.md) owns the
+  current counts and serialized representation.
+
+The defect addressed here is the all-or-nothing domain mapping boundary, not
+validation itself or the runner's internal consistency checks.
+
+## Target Decision Semantics
+
+Reconcile each structurally valid response against the indices actually requested
+in that particular validator batch. Correction indices, rather than segment IDs
+or positions in the response array, identify candidates. Several corrections may
+legitimately target the same segment.
+
+| Response condition | Target behavior |
+| --- | --- |
+| Exactly one valid decision for a requested index | Preserve its approval or rejection and associated decision evidence. |
+| Requested index omitted | Produce an explicit validation-unavailable outcome for that candidate. |
+| Requested index appears more than once | Treat that candidate's validation as unavailable; preserve unrelated decisions. |
+| Index was not requested in this batch | Ignore the extraneous entry and record the handled defect. |
+| Empty decision list for a nonempty request | Mark all requested candidates unavailable; do not fail the run. |
+| Decisions appear in a different order | Match by identity and retain deterministic internal results. |
+
+Duplicate handling does not choose the first or last answer, nor infer a verdict
+by voting. Even identical duplicates follow the agreed duplicate-index handling;
+accepting them specially is not needed for this feature. Duplicate extraneous
+indices remain extraneous entries, not reasons to invalidate requested candidates.
+
+An unavailable candidate remains unapplied and does not proceed as approved to
+later validators. It is distinct from a valid `approved: false` decision, which
+is ordinary editorial processing. Valid approvals continue through the remaining
+required validators and apply-time checks; this feature grants no bypass.
+
+The mapper returns one internal outcome for every requested candidate and no
+outcome for an extraneous index. Preserve strict runner cardinality checks so
+application bugs are still detected. Do not convert genuine internal invariant
+failures, cancellation, or fatal setup failures into tolerated model defects.
+
+## Outcomes and Reporting
+
+Keep existing work-unit aggregation and operation accounting:
+
+- If any requested candidate lacks an unambiguous usable decision, record a
+  validation-operation error identifying the affected requested indices. Other
+  candidates' usable decisions from that operation remain effective. An operation
+  error does not mean every candidate in its batch failed.
+- If all requested candidates have usable decisions and extra entries were
+  ignored, record a successful operation with a recovery for that handled defect.
+  This remains a recovery even if all legitimate decisions are editorial rejections.
+- If missing/duplicate requested decisions and ignored extras coexist, retain the
+  operation error. Follow the existing rule that failed operations do not also
+  produce recovery records; retain extra-entry details in debug evidence.
+- A valid complete response with no handled defect adds neither error nor recovery.
+  Valid editorial rejection alone never adds either category.
+- Preserve actual structural repair and service-retry counts. On successful
+  operations, consolidate applicable recovery reasons in one recovery record.
+  Do not count reconciliation as another LLM operation or structural repair.
+
+Error attribution must distinguish affected candidates from the rest of the batch.
+Debug evidence should preserve supplied decisions, ignored entries, and the reason
+an expected candidate became unavailable. Do not fabricate candidate ledger entries
+for unknown indices or mislabel missing decisions as deliberate editorial rejection.
+Ordinary reports must retain safe metadata without raw transcript or response text.
+
+Use a decision-specific recovery reason rather than treating ignored validator
+entries as defective correction proposals. Concrete reason names and minimal data
+model changes belong in the implementation plan. Keep YAML, Markdown, and CLI
+counts consistent, with the existing reporting-disable and independent debug options.
+
+## Work Needed
+
+### Decision Mapping and Validator Integration
+
+Replace the fail-fast model-response mapper with reconciliation that preserves
+unambiguous decisions and explicitly represents unavailable candidates and ignored
+entries. Update the LLM-backed validator to return those outcomes and accurate
+operation facts instead of rejecting a whole usable batch. Keep domain ownership
+in Audita and reuse the existing unavailable-candidate path.
+
+Review the confidence check currently present in the mapper: structurally invalid
+confidence is already handled by the schema boundary. Do not introduce a second
+whole-batch failure path for an individual defective entry that reaches domain
+mapping. Preserve valid negative decisions and remaining validator execution.
+
+### Reporting and Debug Evidence
+
+Adapt operation mapping and presentation where needed to express partial batch
+failure and successful extra-entry filtering. Review the correction ledger so it
+records unavailable outcomes distinctly from actual editorial decisions. Preserve
+existing report grouping and avoid a new reporting subsystem or per-entry model
+operation. Add only the attribution needed to explain these outcomes.
+
+### Regression Coverage and Documentation
+
+Replace tests that enforce whole-batch rejection with focused tests of partial
+success. Use mixed responses to prove valid approvals and rejections survive
+alongside missing, duplicate, and extraneous indices. Cover conflicting duplicates,
+empty decision lists, and response-order independence without building an exhaustive
+cross-product suite.
+
+Include representative assembled-path coverage proving that survivors complete
+all remaining validators, unavailable candidates never apply, and reporting reflects
+the correct errors/recoveries and applied corrections. Retain tests for strict
+internal cardinality and genuine operation-level failures. Update canonical
+validator and reporting documentation when implementation lands; do not describe
+this roadmap's target as currently implemented elsewhere.
+
+## Scope Boundaries
+
+PromptKit's whole-response structural validation and repair remain unchanged.
+A malformed entry, including an extraneous entry that violates the schema, may
+still invalidate the response and exhaust structural repairs before reconciliation
+can run. Ignoring schema-valid extras does not promise salvage from malformed JSON,
+missing required fields, wrong types, or out-of-range values.
+
+General per-entry structural salvage is a separate design question, not part of
+this fix. Likewise, guided regeneration remains deferred for reconsideration after
+this change; no new retries, feedback prompts, budgets, upstream dependency changes,
+or operator controls are required here.
+
+Preserve module scope, literal replacement behavior, fixed chunks, stage barriers,
+span-deletion safeguards, backend capacity limits, and fatal cancellation behavior.
+No additional restrictions on valid corrections or guarantees about model judgment
+are introduced. This work does not itself authorize tagging or publishing v2.0.0.
+
+## Acceptance Criteria
+
+- A schema-valid response can retain usable decisions despite missing, duplicate,
+  or unexpected indices elsewhere in the same batch.
+- Only the affected requested candidates become unavailable; extras never create
+  candidates or attach decisions to a different batch's candidates.
+- Valid negative decisions remain editorial outcomes, and every applied correction
+  completes all required validation.
+- Internal validator results satisfy existing strict cardinality checks.
+- Missing/duplicate requested decisions yield accurately attributed errors without
+  losing valid work. Extra-only defects yield a consolidated recovery; mixed
+  defects follow existing failed-operation reporting semantics.
+- Final transcript, debug evidence, YAML reports, Markdown counts, and CLI summary
+  agree about applied work and failures/recoveries.
+- Reconciliation makes no additional provider calls. Existing structural failures,
+  provider failures, and cancellation retain their documented handling.
+
+## Open Questions
+
+No unresolved product-policy questions block the staged implementation plan.
+That plan should select the smallest reporting/model changes and concrete reason
+codes needed to express the behavior above. Guided recovery and structural salvage
+remain separate future decisions.

docs/roadmap/future.md

@@ -9,6 +9,12 @@ intent; inclusion does not commit us to implementing every capability.
 
 ## Domain-Aware Response Recovery
 
+First implement [partial success for validator decision sets](decision-set-reconciliation.md),
+preserving usable decisions without another model call. Reconsider guided recovery
+after that simpler change, based on the failures that remain. General per-entry
+salvage from structurally invalid responses is also a separate design question;
+the accepted reconciliation feature retains whole-response structural validation.
+
 Add bounded feedback and regeneration for model responses that satisfy the
 structural schema but violate Audita's domain response contract, such as missing,
 duplicate, or unexpected validator decision indices. The goal is to recover more