163 lines
9.2 KiB
Markdown
163 lines
9.2 KiB
Markdown
# Partial Success for Validator Decision Sets
|
|
|
|
**Last updated:** 2026-09-25
|
|
|
|
**Status:** Implemented. This document owns the target behavior and user intent.
|
|
[implementation.md](implementation.md) owns concrete decisions, ordered work,
|
|
and verification.
|
|
|
|
## Purpose and User Intent
|
|
|
|
Preserve usable validator decisions when other entries in the same model response
|
|
have missing, duplicate, or unexpected correction indices. One defective entry
|
|
should not prevent unrelated candidates from completing their required validation.
|
|
This extends Audita's existing partial-success approach without another model call.
|
|
|
|
Implement this simpler improvement before reconsidering guided domain recovery.
|
|
Missing validation must never authorize applying a correction, but neither should
|
|
it invalidate a different candidate's unambiguous approval or editorial rejection.
|
|
The existing [resilience design](resilience.md) remains the foundation.
|
|
|
|
## Historical Baseline
|
|
|
|
The reviewed application baseline is `d2a6fcf`:
|
|
|
|
- PromptKit validates the complete model response against Audita's embedded JSON
|
|
Schema and performs its configured structural repairs. Only structurally valid,
|
|
decoded responses reach domain decision mapping.
|
|
- `mapLLMResponseToDecisions` in
|
|
[`llm_validators.go`](../../internal/framework/validators/llm_validators.go)
|
|
requires exactly one decision for every requested correction index. An empty
|
|
decision list, missing index, duplicate index, or unexpected index returns an
|
|
error instead of retaining usable decisions.
|
|
- `LLMBackedValidator.Validate` consequently marks the entire batch unavailable
|
|
with `validation_decisions_invalid`. Other batches continue, but usable decisions
|
|
within that batch are lost.
|
|
- The runner's cardinality checks enforce an internal contract: each candidate
|
|
receives one outcome from each validator it reaches. These checks apply to
|
|
Audita-owned results, not directly to raw model entries.
|
|
- Deterministic validators already filter individual proposals. Unpreviewable
|
|
candidates and oversized validation inputs are isolated; recoverable provider
|
|
failures remain batch-local; application skips individual conflicts.
|
|
- Ordinary reports distinguish failed operations from successful operations with
|
|
recoveries. Several failed operations in one stage/chunk share one work-unit
|
|
error record. The [output contract](../integrations/output-files.md) owns the
|
|
current counts and serialized representation.
|
|
|
|
The defect addressed here is the all-or-nothing domain mapping boundary, not
|
|
validation itself or the runner's internal consistency checks.
|
|
|
|
## Target Decision Semantics
|
|
|
|
Reconcile each structurally valid response against the indices actually requested
|
|
in that particular validator batch. Correction indices, rather than segment IDs
|
|
or positions in the response array, identify candidates. Several corrections may
|
|
legitimately target the same segment.
|
|
|
|
| Response condition | Target behavior |
|
|
| --- | --- |
|
|
| Exactly one valid decision for a requested index | Preserve its approval or rejection and associated decision evidence. |
|
|
| Requested index omitted | Produce an explicit validation-unavailable outcome for that candidate. |
|
|
| Requested index appears more than once | Treat that candidate's validation as unavailable; preserve unrelated decisions. |
|
|
| Index was not requested in this batch | Ignore the extraneous entry and record the handled defect. |
|
|
| Empty decision list for a nonempty request | Mark all requested candidates unavailable; do not fail the run. |
|
|
| Decisions appear in a different order | Match by identity and retain deterministic internal results. |
|
|
|
|
Duplicate handling does not choose the first or last answer, nor infer a verdict
|
|
by voting. Even identical duplicates follow the agreed duplicate-index handling;
|
|
accepting them specially is not needed for this feature. Duplicate extraneous
|
|
indices remain extraneous entries, not reasons to invalidate requested candidates.
|
|
|
|
An unavailable candidate remains unapplied and does not proceed as approved to
|
|
later validators. It is distinct from a valid `approved: false` decision, which
|
|
is ordinary editorial processing. Valid approvals continue through the remaining
|
|
required validators and apply-time checks; this feature grants no bypass.
|
|
|
|
The mapper returns one internal outcome for every requested candidate and no
|
|
outcome for an extraneous index. Preserve strict runner cardinality checks so
|
|
application bugs are still detected. Do not convert genuine internal invariant
|
|
failures, cancellation, or fatal setup failures into tolerated model defects.
|
|
|
|
## Outcomes and Reporting
|
|
|
|
Keep existing work-unit aggregation and operation accounting:
|
|
|
|
- If any requested candidate lacks an unambiguous usable decision, record a
|
|
validation-operation error identifying the affected requested indices. Other
|
|
candidates' usable decisions from that operation remain effective. An operation
|
|
error does not mean every candidate in its batch failed.
|
|
- If all requested candidates have usable decisions and extra entries were
|
|
ignored, record a successful operation with a recovery for that handled defect.
|
|
This remains a recovery even if all legitimate decisions are editorial rejections.
|
|
- If missing/duplicate requested decisions and ignored extras coexist, retain the
|
|
operation error. Follow the existing rule that failed operations do not also
|
|
produce recovery records; retain extra-entry details in debug evidence.
|
|
- A valid complete response with no handled defect adds neither error nor recovery.
|
|
Valid editorial rejection alone never adds either category.
|
|
- Preserve actual structural repair and service-retry counts. On successful
|
|
operations, consolidate applicable recovery reasons in one recovery record.
|
|
Do not count reconciliation as another LLM operation or structural repair.
|
|
|
|
Error attribution must distinguish affected candidates from the rest of the batch.
|
|
Debug evidence should preserve supplied decisions, ignored entries, and the reason
|
|
an expected candidate became unavailable. Do not fabricate candidate ledger entries
|
|
for unknown indices or mislabel missing decisions as deliberate editorial rejection.
|
|
Ordinary reports must retain safe metadata without raw transcript or response text.
|
|
|
|
Use a decision-specific recovery reason rather than treating ignored validator
|
|
entries as defective correction proposals. Concrete reason names and minimal data
|
|
model changes belong in the implementation plan. Keep YAML, Markdown, and CLI
|
|
counts consistent, with the existing reporting-disable and independent debug options.
|
|
|
|
## Ownership and Target Architecture
|
|
|
|
Audita's validator layer reconciles structurally valid model responses into
|
|
complete internal candidate outcomes. The runner preserves strict internal
|
|
cardinality and advances only approved candidates. Reporting consumes operation
|
|
facts without treating partial validation failure as loss of every decision.
|
|
Debug evidence distinguishes unavailable validation from editorial rejection.
|
|
|
|
The schema/transport boundary remains with PromptKit; reconciliation needs no
|
|
additional model call. Concrete helper structures, reason names, mapping changes,
|
|
and verification steps belong in the [implementation plan](implementation.md).
|
|
|
|
## Scope Boundaries
|
|
|
|
PromptKit's whole-response structural validation and repair remain unchanged.
|
|
A malformed entry, including an extraneous entry that violates the schema, may
|
|
still invalidate the response and exhaust structural repairs before reconciliation
|
|
can run. Ignoring schema-valid extras does not promise salvage from malformed JSON,
|
|
missing required fields, wrong types, or out-of-range values.
|
|
|
|
General per-entry structural salvage is a separate design question, not part of
|
|
this fix. Likewise, guided regeneration remains deferred for reconsideration after
|
|
this change; no new retries, feedback prompts, budgets, upstream dependency changes,
|
|
or operator controls are required here.
|
|
|
|
Preserve module scope, literal replacement behavior, fixed chunks, stage barriers,
|
|
span-deletion safeguards, backend capacity limits, and fatal cancellation behavior.
|
|
No additional restrictions on valid corrections or guarantees about model judgment
|
|
are introduced. This work does not itself authorize tagging or publishing v2.0.0.
|
|
|
|
## Acceptance Criteria
|
|
|
|
- A schema-valid response can retain usable decisions despite missing, duplicate,
|
|
or unexpected indices elsewhere in the same batch.
|
|
- Only the affected requested candidates become unavailable; extras never create
|
|
candidates or attach decisions to a different batch's candidates.
|
|
- Valid negative decisions remain editorial outcomes, and every applied correction
|
|
completes all required validation.
|
|
- Internal validator results satisfy existing strict cardinality checks.
|
|
- Missing/duplicate requested decisions yield accurately attributed errors without
|
|
losing valid work. Extra-only defects yield a consolidated recovery; mixed
|
|
defects follow existing failed-operation reporting semantics.
|
|
- Final transcript, debug evidence, YAML reports, Markdown counts, and CLI summary
|
|
agree about applied work and failures/recoveries.
|
|
- Reconciliation makes no additional provider calls. Existing structural failures,
|
|
provider failures, and cancellation retain their documented handling.
|
|
|
|
## Open Questions
|
|
|
|
No unresolved product-policy questions block implementation. Guided recovery and
|
|
structural salvage remain separate future decisions.
|