eric/audita
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 6 Wiki Activity

Plan partial validator decision handling

Browse Source
This commit is contained in:
Eric Rakestraw
2026-09-25 03:27:41 +00:00
parent 60e2fe74b5
commit 7199cb565f
2 changed files with 232 additions and 359 deletions
Download Patch File Download Diff File Expand all files Collapse all files

55
docs/roadmap/decision-set-reconciliation.md
View File

@@ -3,8 +3,9 @@
**Last updated:** 2026-09-25
**Status:** Accepted feature direction; implementation not started. Intended for
consideration before v2.0.0. This document defines the target behavior and work
needed; a staged implementation plan will follow separately.
consideration before v2.0.0. This document owns the target behavior and user intent.
[implementation.md](implementation.md) owns concrete decisions, ordered work,
and verification.
## Purpose and User Intent
@@ -109,43 +110,17 @@ entries as defective correction proposals. Concrete reason names and minimal dat
model changes belong in the implementation plan. Keep YAML, Markdown, and CLI
counts consistent, with the existing reporting-disable and independent debug options.
## Work Needed
## Ownership and Target Architecture
### Decision Mapping and Validator Integration
Audita's validator layer reconciles structurally valid model responses into
complete internal candidate outcomes. The runner preserves strict internal
cardinality and advances only approved candidates. Reporting consumes operation
facts without treating partial validation failure as loss of every decision.
Debug evidence distinguishes unavailable validation from editorial rejection.
Replace the fail-fast model-response mapper with reconciliation that preserves
unambiguous decisions and explicitly represents unavailable candidates and ignored
entries. Update the LLM-backed validator to return those outcomes and accurate
operation facts instead of rejecting a whole usable batch. Keep domain ownership
in Audita and reuse the existing unavailable-candidate path.
Review the confidence check currently present in the mapper: structurally invalid
confidence is already handled by the schema boundary. Do not introduce a second
whole-batch failure path for an individual defective entry that reaches domain
mapping. Preserve valid negative decisions and remaining validator execution.
### Reporting and Debug Evidence
Adapt operation mapping and presentation where needed to express partial batch
failure and successful extra-entry filtering. Review the correction ledger so it
records unavailable outcomes distinctly from actual editorial decisions. Preserve
existing report grouping and avoid a new reporting subsystem or per-entry model
operation. Add only the attribution needed to explain these outcomes.
### Regression Coverage and Documentation
Replace tests that enforce whole-batch rejection with focused tests of partial
success. Use mixed responses to prove valid approvals and rejections survive
alongside missing, duplicate, and extraneous indices. Cover conflicting duplicates,
empty decision lists, and response-order independence without building an exhaustive
cross-product suite.
Include representative assembled-path coverage proving that survivors complete
all remaining validators, unavailable candidates never apply, and reporting reflects
the correct errors/recoveries and applied corrections. Retain tests for strict
internal cardinality and genuine operation-level failures. Update canonical
validator and reporting documentation when implementation lands; do not describe
this roadmap's target as currently implemented elsewhere.
The schema/transport boundary remains with PromptKit; reconciliation needs no
additional model call. Concrete helper structures, reason names, mapping changes,
and verification steps belong in the [implementation plan](implementation.md).
## Scope Boundaries
@@ -184,7 +159,5 @@ are introduced. This work does not itself authorize tagging or publishing v2.0.0
## Open Questions
No unresolved product-policy questions block the staged implementation plan.
That plan should select the smallest reporting/model changes and concrete reason
codes needed to express the behavior above. Guided recovery and structural salvage
remain separate future decisions.
No unresolved product-policy questions block implementation. Guided recovery and
structural salvage remain separate future decisions.