audita/docs/roadmap/implementation.md

# Implementation Plan: Partial Validator Decision Sets

**Last updated:** 2026-09-25

**Status:** Implemented; all stages completed.
**Source baseline:** `60e2fe7`.
**Audience:** gpt-5.6-terra coding agent.

## Scope and Execution

Implement [partial success for validator decision sets](decision-set-reconciliation.md).
That roadmap owns product intent and target semantics; this document selects the
implementation and verification work. Follow Stage 1 through Stage 3 in order and
record completion evidence beneath each stage. Each stage is a focused agent task.

Read [development guidance](../development.md) and all policies under
[docs/policy](../policy/): [agents](../policy/agents.md),
[architecture](../policy/architecture.md), [documentation](../policy/documentation.md),
[testing](../policy/testing.md), and [release](../policy/release.md). Use graph-first
code discovery, preserve unrelated work, and commit completed coherent units with
plain English messages. Do not introduce approval rounds between stages.

Aim directly at the final implementation. No temporary modes, compatibility shims,
new configuration, extra model calls, or upstream dependency work is required.
Keep PromptKit structural validation/repair and the runner's internal cardinality
checks. Implement the agreed outcomes, not extra restrictions suggested by existing
tests or incidental internal structure. Internal helper signatures and layout can
be adjusted idiomatically while preserving the decisions below.

This replaces the completed span-deletion plan, available in Git at
`60e2fe7:docs/roadmap/implementation.md`. It does not reopen that feature. The final
stage includes correcting historical links that otherwise point to this new plan.
No release preparation, tag, push, CI wait, or paid-provider evaluation is included.

## Selected Implementation Decisions

### Reconciliation boundary

Keep reconciliation in `internal/framework/validators`, after successful structured
completion and before converting the batch into runner-facing results. Replace
`mapLLMResponseToDecisions` with a small pure reconciliation helper; use an ordinary
private result struct containing:

- `Decisions []Decision`: one outcome per requested item, in request order;
- `UnavailableIndices []int`: requested indices without a usable verdict;
- `IgnoredIndices []int`: unexpected indices encountered in response order, including
  repeated unexpected entries, for debug attribution.

No generic reconciliation framework or new shared abstraction is needed. Requested
identities come from `batch.Items`, not transcript segment IDs, the whole section,
or any other batch. Expected input identities remain an Audita internal contract.

First collect response entries by requested index and count occurrences, separating
unexpected indices. Then walk requested items once to construct outcomes. This
avoids accepting an early duplicate before seeing a conflicting later entry.

Use these candidate-level reason codes in `validators/models.go`:

| Condition | Internal outcome/reason |
| --- | --- |
| Missing requested entry | `DispositionUnavailable`, `missing_validation_decision` |
| More than one entry for a requested index | `DispositionUnavailable`, `duplicate_validation_decision` |
| Single requested entry failing the existing confidence range check | `DispositionUnavailable`, `invalid_validation_decision` |
| Single usable entry | Existing approved or `llm_rejected` decision with its model reason. |

Unavailable outcomes have `Approved: false` and fixed safe explanatory messages.
Duplicate classification takes precedence over inspecting individual duplicate
values; identical duplicates follow the same rule. Unexpected entries are ignored
before inspecting confidence. Retain the existing confidence check as a local
check for decoded callers/test doubles, scoped to that candidate; do not broaden
this into structural salvage, confidence thresholds, or new semantic validation.
The production JSON Schema still rejects out-of-range confidence before mapping.

An empty response list makes every requested item unavailable. Response ordering
must not change verdicts or which candidate is affected. Preserve the existing
handling of valid negative decisions, including reason text. No candidate may
advance merely because another candidate was approved.

### Operation accounting and ordinary reports

Each actual validation batch remains one `outcomes.Operation`. Use existing
`ReasonValidationDecisionsInvalid` (`validation_decisions_invalid`) for a batch
with unavailable requested candidates. Set `Succeeded: false`, set `ProposalIndices`
to only those unavailable indices, retain execution counts, and leave recovery
reasons empty. Still return usable decisions from that batch alongside explicit
unavailable outcomes, with no Go error for the reconciled model defect.

If no requested candidates are unavailable, set `Succeeded: true`. Add
`outcomes.RecoveryValidationDecisionsFiltered` with serialized value
`validation_decisions_filtered` when extras were ignored. Merge it with actual
`output_repaired` and `service_recovered` reasons in the same recovery operation.
Valid negative decisions do not prevent operation success. A clean response creates
no new recovery reason.

When unavailable candidates and extras coexist, record the failed operation only;
keep ignored-entry evidence in debug. Do not create both an error and a recovery
for that operation. Successful survivors must continue even though the operation
itself records a failure. Retain actual execution counts on all paths.

Reuse existing ordinary YAML structures and version 1. The only new public report
value is the decision-specific recovery reason; affected error indices already fit
`Failure.ProposalIndices`. Do not add ignored entries to `Operation.Filtered` or
`filtered_proposals`, which describe correction proposals, and do not increment
`proposal_responses_with_filtering`. The recovery reason and existing operation
identity suffice for ordinary reporting; detailed extra indices belong in debug.

Update `processreport.safeMessage` for `validation_decisions_invalid` to explain
that some required decisions were unavailable without claiming all batch decisions
were discarded. Generalize the recovery message to include response filtering,
rather than only proposal filtering. Exact prose is not a test contract. Preserve
work-unit error grouping, operation counts, and existing YAML/Markdown/CLI totals.

### Debug evidence and runner records

Existing interaction diagnostics already retain the full decoded response and final
content. Add the reconciliation result to the response diagnostic payload for a
successfully decoded validation response. Perform reconciliation before writing
that diagnostic; retain existing call-error diagnostics and diagnostic-write failure
handling. All reconciled decisions, including unavailable outcomes, receive the
returned response artifact path. This records unexpected indices and duplicates
without inventing candidate ledger entries or an additional debug file.

Carry `validators.DecisionDisposition` through `runner.ValidatorDecisionRecord`
and `runner.ValidatorRejectedChange` in an additive `disposition` field. Populate
it from each actual `Decision`, including partial decision evidence returned on
fatal paths. Keep the existing collection names and application flow; no parallel
unavailable-candidate collection is necessary.

In `processreport`, add `disposition` to `LedgerValidatorDecisionRecord` and map
unavailable candidates to a top-level ledger disposition `unavailable`, instead
of `rejected`. Ordinary editorial/deterministic rejections remain `rejected`.
Use the explicit disposition, not a reason-code allowlist, so existing timeout,
provider-failure, and oversized-input outcomes retain the same distinction too.
Update affected test fixtures to supply dispositions where they represent real
validator outputs. Do not invent a fatal check or compatibility subsystem for a
zero-value disposition in handcrafted fixtures.

Retain candidate identity, literal proposed replacements (including explicit empty
strings), and omission of applied replacements for unapplied candidates. Existing
fatal/interrupted/module-failure handling and debug-off behavior remain. Supplied
model decisions remain visible in interaction evidence even when reconciliation
cannot accept them as authoritative verdicts.

## Stage 1 — Reconcile Model Decisions and Preserve Valid Work

**Completed:** 2026-09-25. Reconciliation preserves usable decisions, attributes
unavailable requested indices and ignored extras, and keeps strict runner
cardinality. Validator and runner boundary checks passed with
`go test ./internal/framework/validators ./internal/framework/runner`.

**Work**

1. Read the validator models, `llm_validators.go`, existing partial-batch tests,
   `outcomes/outcomes.go`, and `runner.validateSectionCandidates`.
2. Implement the reconciliation result/helper and candidate reason codes selected
   above. Replace the whole-batch mapping-error branch with outcome construction
   and correct batch accounting. Add the decision-specific recovery constant.
3. Move successful-response diagnostic assembly after reconciliation and include
   its result. Preserve raw/decoded response evidence and diagnostic paths; keep
   recoverable call errors, fatal errors, and cancellation on their existing paths.
4. Keep `EnforceDecisionCardinality` and `EnforcePartialDecisionCardinality` strict.
   Ensure final validator decisions are complete and unique internally. The runner
   should continue passing only approved candidates onward without treating an
   operation's failure metadata as a reason to discard valid decisions.
5. Replace obsolete batch-rejection expectations in `llm_validators_test.go` and
   relevant runner tests. Preserve cardinality tests that exercise broken internal
   validators. Update the repaired-but-domain-invalid tests so actual unavailable
   cases remain errors with counts retained and no recovery record.

**Focused verification**

Use a small table at the validator boundary for mixed missing/extra indices,
conflicting and identical requested duplicates, extra-only responses, empty lists,
reordered responses, and a single invalid-confidence decoded test-double entry.
Include noncontiguous expected indices and valid negative decisions. Assert semantic
outcomes and affected indices, not helper layout or exact messages. Include one
controlled case combining successful structural/service recovery facts with ignored
extras and one failed reconciliation retaining counts without recovery reasons.

At the runner boundary, use the real LLM-backed validator with a controlled client
and a later validator: prove a survivor reaches later required validation, a later
rejection prevents application, and unavailable candidates never reach that later
validator as approved. Use successful proposal operation facts where accounting is
asserted; do not check an empty operation list. Reuse existing fixtures/tests rather
than creating a second framework or duplicating every table case.

Run `go test ./internal/framework/validators ./internal/framework/runner`.

**End state:** Usable decisions survive model indexing defects, strict internal
cardinality remains satisfied, and operation facts explain partial failure or
successful extra filtering. No extra model generation is introduced.

## Stage 2 — Carry Dispositions Through Debug and Report Partial Outcomes

**Completed:** 2026-09-25. Runner and ledger records carry explicit
dispositions; ordinary reports distinguish unavailable decision errors from
extra-index filtering recoveries. Focused report, ledger, runner, and YAML
checks passed with `go test ./internal/framework/runner
./internal/framework/processreport ./internal/core/reporting`.

**Work**

1. Add and populate disposition fields through the runner and ledger as selected
   above. Map unavailable candidates distinctly and preserve empty replacement
   serialization. Check both normal and fatal partial-evidence paths.
2. Update `processreport` messages and review `Build`, `mapFailure`, and `mapRecovery`.
   Reuse their existing aggregation; change only assumptions that would misreport
   the new operation facts. Core YAML structures and Markdown table layout remain.
3. Extend existing report/ledger tests for an operation with surviving work plus
   unavailable indices, extra-only recovery, and a mixed defect operation. Verify
   error indices exclude survivors, extra filtering does not count as proposal
   filtering, and failed operations do not also generate recovery records.
4. Verify debug ledger entries distinguish genuine rejection from unavailability,
   and interaction diagnostics preserve ignored entries and reconciliation reasons.
   Unknown response indices must not create fake ledger candidates. Retain existing
   redaction and optional-debug-failure tests rather than reproducing them all.
5. Update `docs/integrations/output-files.md` for the new recovery reason and debug
   disposition, and `docs/internal/diagnostics-reporting.md` for the mapping changes.
   Explain that validation failure indices identify unavailable candidates, not
   necessarily every candidate submitted to that batch.

**Validation**

Run `go test ./internal/framework/runner ./internal/framework/processreport ./internal/core/reporting`.
Use semantic YAML/summary assertions; no whole-report prose golden, exact message
snapshot, new schema version, or new rendering mode is needed.

**End state:** Reports and debug evidence accurately describe partial validation
success and handled extras through existing reporting mechanisms.

## Stage 3 — Verify the Assembled Workflow and Finalize Documentation

**Completed:** 2026-09-25. Controlled subprocess runs covered a mixed decision
response and an extra-only response through the CLI, transcript, YAML, Markdown,
debug ledger, and interaction evidence. Fixture execution facts showed one
provider attempt and zero structural repairs per logical call. `go test
./cmd/audita` and `go test ./...` passed. Canonical validator guidance and
roadmap status were updated; the two span-deletion plan links now target the
historical plan at `60e2fe7`.

**Work**

1. Extend the existing local-provider subprocess fixtures in
   `cmd/audita/main_integration_test.go`. Add a representative mixed-decision run
   with an applied survivor, a legitimate rejection, a missing requested decision,
   a duplicate requested decision, and an extra index. Return responses by the
   requested correction identities; do not freeze incidental validator call order.
   Ensure the survivor passes all remaining required validation. Assert successful
   CLI completion, correct transcript, one work-unit error with only unavailable
   indices, no reconciliation recovery for the failed operation, and debug
   evidence distinguishing rejection/unavailability without unknown candidates.
2. Add or adapt a compact extra-only case proving usable work survives and one
   `validation_decisions_filtered` recovery appears. Verify YAML, Markdown, and CLI
   agree on totals using semantic assertions. Keep proposal-filtering counts separate.
   Existing reporting-disable/debug selection tests remain the owners of those options.
3. In these fixtures give PromptKit its normal configured structural repair allowance,
   but supply schema-valid responses. Verify reconciliation adds no generations or
   structural repairs using fixture requests/execution facts, without encoding today's
   total validator count as a permanent contract. Keep structural-malformation tests
   and recoverable batch-failure tests; do not relax the schema to make fixtures pass.
4. Update `docs/internal/validators.md` and any affected canonical descriptions found
   by searching for whole-batch rejection claims. Explain tolerant external response
   reconciliation versus strict internal cardinality. The structural-validation
   boundary remains explicit, and guided recovery remains deferred in `future.md`.
5. Mark the feature roadmap implemented only after the assembled checks pass, and
   record actual completion evidence in this plan. Repair the two span-deletion
   roadmap links to the old plan with the historical URL
   `https://gitea.maximumdirect.net/eric/audita/src/commit/60e2fe7/docs/roadmap/implementation.md`.
   Preserve historical release notes and unrelated roadmaps.

**Validation**

Run `go test ./cmd/audita` while developing the fixture, then `go test ./...` once
for the completed implementation. Run `git diff --check` and verify affected
Markdown links. Do not add live-provider checks, a platform matrix, repeated full
suite runs, or release gates solely for this feature. No code is implemented by
this planning pass; these commands are instructions for the implementation sprint.

**End state:** The accepted partial-success behavior is exercised through the real
CLI/library boundary, canonical documentation matches it, and coherent changes
are committed for subsequent release consideration.

## Open Questions

None required before implementation. The implementation decisions above preserve
the agreed product scope. General structural salvage and guided regeneration remain
separate future decisions, not unresolved prerequisites for this work.