# Implementation Plan: Partial Validator Decision Sets **Last updated:** 2026-09-25 **Status:** Implemented; all stages completed. **Source baseline:** `60e2fe7`. **Audience:** gpt-5.6-terra coding agent. ## Scope and Execution Implement [partial success for validator decision sets](decision-set-reconciliation.md). That roadmap owns product intent and target semantics; this document selects the implementation and verification work. Follow Stage 1 through Stage 3 in order and record completion evidence beneath each stage. Each stage is a focused agent task. Read [development guidance](../development.md) and all policies under [docs/policy](../policy/): [agents](../policy/agents.md), [architecture](../policy/architecture.md), [documentation](../policy/documentation.md), [testing](../policy/testing.md), and [release](../policy/release.md). Use graph-first code discovery, preserve unrelated work, and commit completed coherent units with plain English messages. Do not introduce approval rounds between stages. Aim directly at the final implementation. No temporary modes, compatibility shims, new configuration, extra model calls, or upstream dependency work is required. Keep PromptKit structural validation/repair and the runner's internal cardinality checks. Implement the agreed outcomes, not extra restrictions suggested by existing tests or incidental internal structure. Internal helper signatures and layout can be adjusted idiomatically while preserving the decisions below. This replaces the completed span-deletion plan, available in Git at `60e2fe7:docs/roadmap/implementation.md`. It does not reopen that feature. The final stage includes correcting historical links that otherwise point to this new plan. No release preparation, tag, push, CI wait, or paid-provider evaluation is included. ## Selected Implementation Decisions ### Reconciliation boundary Keep reconciliation in `internal/framework/validators`, after successful structured completion and before converting the batch into runner-facing results. Replace `mapLLMResponseToDecisions` with a small pure reconciliation helper; use an ordinary private result struct containing: - `Decisions []Decision`: one outcome per requested item, in request order; - `UnavailableIndices []int`: requested indices without a usable verdict; - `IgnoredIndices []int`: unexpected indices encountered in response order, including repeated unexpected entries, for debug attribution. No generic reconciliation framework or new shared abstraction is needed. Requested identities come from `batch.Items`, not transcript segment IDs, the whole section, or any other batch. Expected input identities remain an Audita internal contract. First collect response entries by requested index and count occurrences, separating unexpected indices. Then walk requested items once to construct outcomes. This avoids accepting an early duplicate before seeing a conflicting later entry. Use these candidate-level reason codes in `validators/models.go`: | Condition | Internal outcome/reason | | --- | --- | | Missing requested entry | `DispositionUnavailable`, `missing_validation_decision` | | More than one entry for a requested index | `DispositionUnavailable`, `duplicate_validation_decision` | | Single requested entry failing the existing confidence range check | `DispositionUnavailable`, `invalid_validation_decision` | | Single usable entry | Existing approved or `llm_rejected` decision with its model reason. | Unavailable outcomes have `Approved: false` and fixed safe explanatory messages. Duplicate classification takes precedence over inspecting individual duplicate values; identical duplicates follow the same rule. Unexpected entries are ignored before inspecting confidence. Retain the existing confidence check as a local check for decoded callers/test doubles, scoped to that candidate; do not broaden this into structural salvage, confidence thresholds, or new semantic validation. The production JSON Schema still rejects out-of-range confidence before mapping. An empty response list makes every requested item unavailable. Response ordering must not change verdicts or which candidate is affected. Preserve the existing handling of valid negative decisions, including reason text. No candidate may advance merely because another candidate was approved. ### Operation accounting and ordinary reports Each actual validation batch remains one `outcomes.Operation`. Use existing `ReasonValidationDecisionsInvalid` (`validation_decisions_invalid`) for a batch with unavailable requested candidates. Set `Succeeded: false`, set `ProposalIndices` to only those unavailable indices, retain execution counts, and leave recovery reasons empty. Still return usable decisions from that batch alongside explicit unavailable outcomes, with no Go error for the reconciled model defect. If no requested candidates are unavailable, set `Succeeded: true`. Add `outcomes.RecoveryValidationDecisionsFiltered` with serialized value `validation_decisions_filtered` when extras were ignored. Merge it with actual `output_repaired` and `service_recovered` reasons in the same recovery operation. Valid negative decisions do not prevent operation success. A clean response creates no new recovery reason. When unavailable candidates and extras coexist, record the failed operation only; keep ignored-entry evidence in debug. Do not create both an error and a recovery for that operation. Successful survivors must continue even though the operation itself records a failure. Retain actual execution counts on all paths. Reuse existing ordinary YAML structures and version 1. The only new public report value is the decision-specific recovery reason; affected error indices already fit `Failure.ProposalIndices`. Do not add ignored entries to `Operation.Filtered` or `filtered_proposals`, which describe correction proposals, and do not increment `proposal_responses_with_filtering`. The recovery reason and existing operation identity suffice for ordinary reporting; detailed extra indices belong in debug. Update `processreport.safeMessage` for `validation_decisions_invalid` to explain that some required decisions were unavailable without claiming all batch decisions were discarded. Generalize the recovery message to include response filtering, rather than only proposal filtering. Exact prose is not a test contract. Preserve work-unit error grouping, operation counts, and existing YAML/Markdown/CLI totals. ### Debug evidence and runner records Existing interaction diagnostics already retain the full decoded response and final content. Add the reconciliation result to the response diagnostic payload for a successfully decoded validation response. Perform reconciliation before writing that diagnostic; retain existing call-error diagnostics and diagnostic-write failure handling. All reconciled decisions, including unavailable outcomes, receive the returned response artifact path. This records unexpected indices and duplicates without inventing candidate ledger entries or an additional debug file. Carry `validators.DecisionDisposition` through `runner.ValidatorDecisionRecord` and `runner.ValidatorRejectedChange` in an additive `disposition` field. Populate it from each actual `Decision`, including partial decision evidence returned on fatal paths. Keep the existing collection names and application flow; no parallel unavailable-candidate collection is necessary. In `processreport`, add `disposition` to `LedgerValidatorDecisionRecord` and map unavailable candidates to a top-level ledger disposition `unavailable`, instead of `rejected`. Ordinary editorial/deterministic rejections remain `rejected`. Use the explicit disposition, not a reason-code allowlist, so existing timeout, provider-failure, and oversized-input outcomes retain the same distinction too. Update affected test fixtures to supply dispositions where they represent real validator outputs. Do not invent a fatal check or compatibility subsystem for a zero-value disposition in handcrafted fixtures. Retain candidate identity, literal proposed replacements (including explicit empty strings), and omission of applied replacements for unapplied candidates. Existing fatal/interrupted/module-failure handling and debug-off behavior remain. Supplied model decisions remain visible in interaction evidence even when reconciliation cannot accept them as authoritative verdicts. ## Stage 1 — Reconcile Model Decisions and Preserve Valid Work **Completed:** 2026-09-25. Reconciliation preserves usable decisions, attributes unavailable requested indices and ignored extras, and keeps strict runner cardinality. Validator and runner boundary checks passed with `go test ./internal/framework/validators ./internal/framework/runner`. **Work** 1. Read the validator models, `llm_validators.go`, existing partial-batch tests, `outcomes/outcomes.go`, and `runner.validateSectionCandidates`. 2. Implement the reconciliation result/helper and candidate reason codes selected above. Replace the whole-batch mapping-error branch with outcome construction and correct batch accounting. Add the decision-specific recovery constant. 3. Move successful-response diagnostic assembly after reconciliation and include its result. Preserve raw/decoded response evidence and diagnostic paths; keep recoverable call errors, fatal errors, and cancellation on their existing paths. 4. Keep `EnforceDecisionCardinality` and `EnforcePartialDecisionCardinality` strict. Ensure final validator decisions are complete and unique internally. The runner should continue passing only approved candidates onward without treating an operation's failure metadata as a reason to discard valid decisions. 5. Replace obsolete batch-rejection expectations in `llm_validators_test.go` and relevant runner tests. Preserve cardinality tests that exercise broken internal validators. Update the repaired-but-domain-invalid tests so actual unavailable cases remain errors with counts retained and no recovery record. **Focused verification** Use a small table at the validator boundary for mixed missing/extra indices, conflicting and identical requested duplicates, extra-only responses, empty lists, reordered responses, and a single invalid-confidence decoded test-double entry. Include noncontiguous expected indices and valid negative decisions. Assert semantic outcomes and affected indices, not helper layout or exact messages. Include one controlled case combining successful structural/service recovery facts with ignored extras and one failed reconciliation retaining counts without recovery reasons. At the runner boundary, use the real LLM-backed validator with a controlled client and a later validator: prove a survivor reaches later required validation, a later rejection prevents application, and unavailable candidates never reach that later validator as approved. Use successful proposal operation facts where accounting is asserted; do not check an empty operation list. Reuse existing fixtures/tests rather than creating a second framework or duplicating every table case. Run `go test ./internal/framework/validators ./internal/framework/runner`. **End state:** Usable decisions survive model indexing defects, strict internal cardinality remains satisfied, and operation facts explain partial failure or successful extra filtering. No extra model generation is introduced. ## Stage 2 — Carry Dispositions Through Debug and Report Partial Outcomes **Completed:** 2026-09-25. Runner and ledger records carry explicit dispositions; ordinary reports distinguish unavailable decision errors from extra-index filtering recoveries. Focused report, ledger, runner, and YAML checks passed with `go test ./internal/framework/runner ./internal/framework/processreport ./internal/core/reporting`. **Work** 1. Add and populate disposition fields through the runner and ledger as selected above. Map unavailable candidates distinctly and preserve empty replacement serialization. Check both normal and fatal partial-evidence paths. 2. Update `processreport` messages and review `Build`, `mapFailure`, and `mapRecovery`. Reuse their existing aggregation; change only assumptions that would misreport the new operation facts. Core YAML structures and Markdown table layout remain. 3. Extend existing report/ledger tests for an operation with surviving work plus unavailable indices, extra-only recovery, and a mixed defect operation. Verify error indices exclude survivors, extra filtering does not count as proposal filtering, and failed operations do not also generate recovery records. 4. Verify debug ledger entries distinguish genuine rejection from unavailability, and interaction diagnostics preserve ignored entries and reconciliation reasons. Unknown response indices must not create fake ledger candidates. Retain existing redaction and optional-debug-failure tests rather than reproducing them all. 5. Update `docs/integrations/output-files.md` for the new recovery reason and debug disposition, and `docs/internal/diagnostics-reporting.md` for the mapping changes. Explain that validation failure indices identify unavailable candidates, not necessarily every candidate submitted to that batch. **Validation** Run `go test ./internal/framework/runner ./internal/framework/processreport ./internal/core/reporting`. Use semantic YAML/summary assertions; no whole-report prose golden, exact message snapshot, new schema version, or new rendering mode is needed. **End state:** Reports and debug evidence accurately describe partial validation success and handled extras through existing reporting mechanisms. ## Stage 3 — Verify the Assembled Workflow and Finalize Documentation **Completed:** 2026-09-25. Controlled subprocess runs covered a mixed decision response and an extra-only response through the CLI, transcript, YAML, Markdown, debug ledger, and interaction evidence. Fixture execution facts showed one provider attempt and zero structural repairs per logical call. `go test ./cmd/audita` and `go test ./...` passed. Canonical validator guidance and roadmap status were updated; the two span-deletion plan links now target the historical plan at `60e2fe7`. **Work** 1. Extend the existing local-provider subprocess fixtures in `cmd/audita/main_integration_test.go`. Add a representative mixed-decision run with an applied survivor, a legitimate rejection, a missing requested decision, a duplicate requested decision, and an extra index. Return responses by the requested correction identities; do not freeze incidental validator call order. Ensure the survivor passes all remaining required validation. Assert successful CLI completion, correct transcript, one work-unit error with only unavailable indices, no reconciliation recovery for the failed operation, and debug evidence distinguishing rejection/unavailability without unknown candidates. 2. Add or adapt a compact extra-only case proving usable work survives and one `validation_decisions_filtered` recovery appears. Verify YAML, Markdown, and CLI agree on totals using semantic assertions. Keep proposal-filtering counts separate. Existing reporting-disable/debug selection tests remain the owners of those options. 3. In these fixtures give PromptKit its normal configured structural repair allowance, but supply schema-valid responses. Verify reconciliation adds no generations or structural repairs using fixture requests/execution facts, without encoding today's total validator count as a permanent contract. Keep structural-malformation tests and recoverable batch-failure tests; do not relax the schema to make fixtures pass. 4. Update `docs/internal/validators.md` and any affected canonical descriptions found by searching for whole-batch rejection claims. Explain tolerant external response reconciliation versus strict internal cardinality. The structural-validation boundary remains explicit, and guided recovery remains deferred in `future.md`. 5. Mark the feature roadmap implemented only after the assembled checks pass, and record actual completion evidence in this plan. Repair the two span-deletion roadmap links to the old plan with the historical URL `https://gitea.maximumdirect.net/eric/audita/src/commit/60e2fe7/docs/roadmap/implementation.md`. Preserve historical release notes and unrelated roadmaps. **Validation** Run `go test ./cmd/audita` while developing the fixture, then `go test ./...` once for the completed implementation. Run `git diff --check` and verify affected Markdown links. Do not add live-provider checks, a platform matrix, repeated full suite runs, or release gates solely for this feature. No code is implemented by this planning pass; these commands are instructions for the implementation sprint. **End state:** The accepted partial-success behavior is exercised through the real CLI/library boundary, canonical documentation matches it, and coherent changes are committed for subsequent release consideration. ## Open Questions None required before implementation. The implementation decisions above preserve the agreed product scope. General structural salvage and guided regeneration remain separate future decisions, not unresolved prerequisites for this work.