audita/docs/roadmap/implementation.md

Implementation Plan: Partial Validator Decision Sets

Last updated: 2026-09-25

Status: Implemented; all stages completed. Source baseline: 60e2fe7. Audience: gpt-5.6-terra coding agent.

Scope and Execution

Implement partial success for validator decision sets. That roadmap owns product intent and target semantics; this document selects the implementation and verification work. Follow Stage 1 through Stage 3 in order and record completion evidence beneath each stage. Each stage is a focused agent task.

Read development guidance and all policies under docs/policy: agents, architecture, documentation, testing, and release. Use graph-first code discovery, preserve unrelated work, and commit completed coherent units with plain English messages. Do not introduce approval rounds between stages.

Aim directly at the final implementation. No temporary modes, compatibility shims, new configuration, extra model calls, or upstream dependency work is required. Keep PromptKit structural validation/repair and the runner's internal cardinality checks. Implement the agreed outcomes, not extra restrictions suggested by existing tests or incidental internal structure. Internal helper signatures and layout can be adjusted idiomatically while preserving the decisions below.

This replaces the completed span-deletion plan, available in Git at 60e2fe7:docs/roadmap/implementation.md. It does not reopen that feature. The final stage includes correcting historical links that otherwise point to this new plan. No release preparation, tag, push, CI wait, or paid-provider evaluation is included.

Selected Implementation Decisions

Reconciliation boundary

Keep reconciliation in internal/framework/validators, after successful structured completion and before converting the batch into runner-facing results. Replace mapLLMResponseToDecisions with a small pure reconciliation helper; use an ordinary private result struct containing:

• Decisions []Decision: one outcome per requested item, in request order;
• UnavailableIndices []int: requested indices without a usable verdict;
• IgnoredIndices []int: unexpected indices encountered in response order, including repeated unexpected entries, for debug attribution.

No generic reconciliation framework or new shared abstraction is needed. Requested identities come from batch.Items, not transcript segment IDs, the whole section, or any other batch. Expected input identities remain an Audita internal contract.

First collect response entries by requested index and count occurrences, separating unexpected indices. Then walk requested items once to construct outcomes. This avoids accepting an early duplicate before seeing a conflicting later entry.

Use these candidate-level reason codes in validators/models.go:

Condition	Internal outcome/reason
Missing requested entry	DispositionUnavailable, missing_validation_decision
More than one entry for a requested index	DispositionUnavailable, duplicate_validation_decision
Single requested entry failing the existing confidence range check	DispositionUnavailable, invalid_validation_decision
Single usable entry	Existing approved or llm_rejected decision with its model reason.

Unavailable outcomes have Approved: false and fixed safe explanatory messages. Duplicate classification takes precedence over inspecting individual duplicate values; identical duplicates follow the same rule. Unexpected entries are ignored before inspecting confidence. Retain the existing confidence check as a local check for decoded callers/test doubles, scoped to that candidate; do not broaden this into structural salvage, confidence thresholds, or new semantic validation. The production JSON Schema still rejects out-of-range confidence before mapping.

An empty response list makes every requested item unavailable. Response ordering must not change verdicts or which candidate is affected. Preserve the existing handling of valid negative decisions, including reason text. No candidate may advance merely because another candidate was approved.

Operation accounting and ordinary reports

Each actual validation batch remains one outcomes.Operation. Use existing ReasonValidationDecisionsInvalid (validation_decisions_invalid) for a batch with unavailable requested candidates. Set Succeeded: false, set ProposalIndices to only those unavailable indices, retain execution counts, and leave recovery reasons empty. Still return usable decisions from that batch alongside explicit unavailable outcomes, with no Go error for the reconciled model defect.

If no requested candidates are unavailable, set Succeeded: true. Add outcomes.RecoveryValidationDecisionsFiltered with serialized value validation_decisions_filtered when extras were ignored. Merge it with actual output_repaired and service_recovered reasons in the same recovery operation. Valid negative decisions do not prevent operation success. A clean response creates no new recovery reason.

When unavailable candidates and extras coexist, record the failed operation only; keep ignored-entry evidence in debug. Do not create both an error and a recovery for that operation. Successful survivors must continue even though the operation itself records a failure. Retain actual execution counts on all paths.

Reuse existing ordinary YAML structures and version 1. The only new public report value is the decision-specific recovery reason; affected error indices already fit Failure.ProposalIndices. Do not add ignored entries to Operation.Filtered or filtered_proposals, which describe correction proposals, and do not increment proposal_responses_with_filtering. The recovery reason and existing operation identity suffice for ordinary reporting; detailed extra indices belong in debug.

Update processreport.safeMessage for validation_decisions_invalid to explain that some required decisions were unavailable without claiming all batch decisions were discarded. Generalize the recovery message to include response filtering, rather than only proposal filtering. Exact prose is not a test contract. Preserve work-unit error grouping, operation counts, and existing YAML/Markdown/CLI totals.

Debug evidence and runner records

Existing interaction diagnostics already retain the full decoded response and final content. Add the reconciliation result to the response diagnostic payload for a successfully decoded validation response. Perform reconciliation before writing that diagnostic; retain existing call-error diagnostics and diagnostic-write failure handling. All reconciled decisions, including unavailable outcomes, receive the returned response artifact path. This records unexpected indices and duplicates without inventing candidate ledger entries or an additional debug file.

Carry validators.DecisionDisposition through runner.ValidatorDecisionRecord and runner.ValidatorRejectedChange in an additive disposition field. Populate it from each actual Decision, including partial decision evidence returned on fatal paths. Keep the existing collection names and application flow; no parallel unavailable-candidate collection is necessary.

In processreport, add disposition to LedgerValidatorDecisionRecord and map unavailable candidates to a top-level ledger disposition unavailable, instead of rejected. Ordinary editorial/deterministic rejections remain rejected. Use the explicit disposition, not a reason-code allowlist, so existing timeout, provider-failure, and oversized-input outcomes retain the same distinction too. Update affected test fixtures to supply dispositions where they represent real validator outputs. Do not invent a fatal check or compatibility subsystem for a zero-value disposition in handcrafted fixtures.

Retain candidate identity, literal proposed replacements (including explicit empty strings), and omission of applied replacements for unapplied candidates. Existing fatal/interrupted/module-failure handling and debug-off behavior remain. Supplied model decisions remain visible in interaction evidence even when reconciliation cannot accept them as authoritative verdicts.

Stage 1 — Reconcile Model Decisions and Preserve Valid Work

Completed: 2026-09-25. Reconciliation preserves usable decisions, attributes unavailable requested indices and ignored extras, and keeps strict runner cardinality. Validator and runner boundary checks passed with go test ./internal/framework/validators ./internal/framework/runner.

Work

1. Read the validator models, llm_validators.go, existing partial-batch tests, outcomes/outcomes.go, and runner.validateSectionCandidates.
2. Implement the reconciliation result/helper and candidate reason codes selected above. Replace the whole-batch mapping-error branch with outcome construction and correct batch accounting. Add the decision-specific recovery constant.
3. Move successful-response diagnostic assembly after reconciliation and include its result. Preserve raw/decoded response evidence and diagnostic paths; keep recoverable call errors, fatal errors, and cancellation on their existing paths.
4. Keep EnforceDecisionCardinality and EnforcePartialDecisionCardinality strict. Ensure final validator decisions are complete and unique internally. The runner should continue passing only approved candidates onward without treating an operation's failure metadata as a reason to discard valid decisions.
5. Replace obsolete batch-rejection expectations in llm_validators_test.go and relevant runner tests. Preserve cardinality tests that exercise broken internal validators. Update the repaired-but-domain-invalid tests so actual unavailable cases remain errors with counts retained and no recovery record.

Focused verification

Use a small table at the validator boundary for mixed missing/extra indices, conflicting and identical requested duplicates, extra-only responses, empty lists, reordered responses, and a single invalid-confidence decoded test-double entry. Include noncontiguous expected indices and valid negative decisions. Assert semantic outcomes and affected indices, not helper layout or exact messages. Include one controlled case combining successful structural/service recovery facts with ignored extras and one failed reconciliation retaining counts without recovery reasons.

At the runner boundary, use the real LLM-backed validator with a controlled client and a later validator: prove a survivor reaches later required validation, a later rejection prevents application, and unavailable candidates never reach that later validator as approved. Use successful proposal operation facts where accounting is asserted; do not check an empty operation list. Reuse existing fixtures/tests rather than creating a second framework or duplicating every table case.

Run go test ./internal/framework/validators ./internal/framework/runner.

End state: Usable decisions survive model indexing defects, strict internal cardinality remains satisfied, and operation facts explain partial failure or successful extra filtering. No extra model generation is introduced.

Stage 2 — Carry Dispositions Through Debug and Report Partial Outcomes

Completed: 2026-09-25. Runner and ledger records carry explicit dispositions; ordinary reports distinguish unavailable decision errors from extra-index filtering recoveries. Focused report, ledger, runner, and YAML checks passed with go test ./internal/framework/runner ./internal/framework/processreport ./internal/core/reporting.

Work

1. Add and populate disposition fields through the runner and ledger as selected above. Map unavailable candidates distinctly and preserve empty replacement serialization. Check both normal and fatal partial-evidence paths.
2. Update processreport messages and review Build, mapFailure, and mapRecovery. Reuse their existing aggregation; change only assumptions that would misreport the new operation facts. Core YAML structures and Markdown table layout remain.
3. Extend existing report/ledger tests for an operation with surviving work plus unavailable indices, extra-only recovery, and a mixed defect operation. Verify error indices exclude survivors, extra filtering does not count as proposal filtering, and failed operations do not also generate recovery records.
4. Verify debug ledger entries distinguish genuine rejection from unavailability, and interaction diagnostics preserve ignored entries and reconciliation reasons. Unknown response indices must not create fake ledger candidates. Retain existing redaction and optional-debug-failure tests rather than reproducing them all.
5. Update docs/integrations/output-files.md for the new recovery reason and debug disposition, and docs/internal/diagnostics-reporting.md for the mapping changes. Explain that validation failure indices identify unavailable candidates, not necessarily every candidate submitted to that batch.

Validation

Run go test ./internal/framework/runner ./internal/framework/processreport ./internal/core/reporting. Use semantic YAML/summary assertions; no whole-report prose golden, exact message snapshot, new schema version, or new rendering mode is needed.

End state: Reports and debug evidence accurately describe partial validation success and handled extras through existing reporting mechanisms.

Stage 3 — Verify the Assembled Workflow and Finalize Documentation

Completed: 2026-09-25. Controlled subprocess runs covered a mixed decision response and an extra-only response through the CLI, transcript, YAML, Markdown, debug ledger, and interaction evidence. Fixture execution facts showed one provider attempt and zero structural repairs per logical call. go test ./cmd/audita and go test ./... passed. Canonical validator guidance and roadmap status were updated; the two span-deletion plan links now target the historical plan at 60e2fe7.

Work

1. Extend the existing local-provider subprocess fixtures in cmd/audita/main_integration_test.go. Add a representative mixed-decision run with an applied survivor, a legitimate rejection, a missing requested decision, a duplicate requested decision, and an extra index. Return responses by the requested correction identities; do not freeze incidental validator call order. Ensure the survivor passes all remaining required validation. Assert successful CLI completion, correct transcript, one work-unit error with only unavailable indices, no reconciliation recovery for the failed operation, and debug evidence distinguishing rejection/unavailability without unknown candidates.
2. Add or adapt a compact extra-only case proving usable work survives and one validation_decisions_filtered recovery appears. Verify YAML, Markdown, and CLI agree on totals using semantic assertions. Keep proposal-filtering counts separate. Existing reporting-disable/debug selection tests remain the owners of those options.
3. In these fixtures give PromptKit its normal configured structural repair allowance, but supply schema-valid responses. Verify reconciliation adds no generations or structural repairs using fixture requests/execution facts, without encoding today's total validator count as a permanent contract. Keep structural-malformation tests and recoverable batch-failure tests; do not relax the schema to make fixtures pass.
4. Update docs/internal/validators.md and any affected canonical descriptions found by searching for whole-batch rejection claims. Explain tolerant external response reconciliation versus strict internal cardinality. The structural-validation boundary remains explicit, and guided recovery remains deferred in future.md.
5. Mark the feature roadmap implemented only after the assembled checks pass, and record actual completion evidence in this plan. Repair the two span-deletion roadmap links to the old plan with the historical URL https://gitea.maximumdirect.net/eric/audita/src/commit/60e2fe7/docs/roadmap/implementation.md. Preserve historical release notes and unrelated roadmaps.

Validation

Run go test ./cmd/audita while developing the fixture, then go test ./... once for the completed implementation. Run git diff --check and verify affected Markdown links. Do not add live-provider checks, a platform matrix, repeated full suite runs, or release gates solely for this feature. No code is implemented by this planning pass; these commands are instructions for the implementation sprint.

End state: The accepted partial-success behavior is exercised through the real CLI/library boundary, canonical documentation matches it, and coherent changes are committed for subsequent release consideration.

Open Questions

None required before implementation. The implementation decisions above preserve the agreed product scope. General structural salvage and guided regeneration remain separate future decisions, not unresolved prerequisites for this work.