eric/audita
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 2 Wiki Activity

Make module-stage LLM handling resilient and report warnings

Browse Source
This commit is contained in:
Eric Rakestraw
2026-05-23 10:07:06 -05:00
parent a84941d681
commit a3655f5540
43 changed files with 856 additions and 217 deletions
Download Patch File Download Diff File Expand all files Collapse all files

72
docs/roadmap/publish.md Normal file
View File

@@ -0,0 +1,72 @@
# Hard-Cutover Roadmap for Module-Stage LLM Resilience
## Summary
This roadmap captures the module-stage resilience work for Audita:
- fail fast on initialization, configuration, schema, and other pre-module setup errors;
- remain resilient during module execution when LLM payloads are malformed or individual proposed corrections are invalid;
- reject bad corrections through validator/reporting paths instead of aborting the module or process;
- keep success stderr quiet; surface warnings only through report and diagnostics artifacts;
- use a hard cutover only, with no compatibility aliases or transitional code.
Locked decisions:
- keep the stable public validator key `non_empty_corrected_text`;
- change that validator’s behavior to mean “the resulting segment text must not be empty/whitespace-only after applying the proposal preview”;
- malformed LLM-validator batch payloads reject the entire affected batch under that validator and continue;
- proposal/validator transport failures, timeouts, and provider/runtime call failures remain fatal;
- malformed structured payloads are downgraded; non-malformed runtime call failures are not.
## Stage 1: Proposal Intake Hardening
- Stop treating invalid individual structured corrections as fatal during proposal generation.
- Preserve returned correction ordering and proposal-index assignment even when individual corrections are malformed.
- Allow `corrected_text == ""` when the resulting segment remains non-empty after previewed application.
- Downgrade malformed proposal-generation structured payloads into section-scoped warnings with zero proposals for that section.
- Keep proposal-generation transport/provider/runtime call failures fatal.
Deterministic validation changes:
- Add `proposal_shape` as a built-in deterministic validator and run it first in every built-in module chain.
- Reject malformed proposal fields with stable reason codes:
- `invalid_target_segment_id`
- `empty_original_text`
- `invalid_confidence`
- Keep `non_empty_corrected_text` as the stable validator key, but change its semantics to reject only `empty_resulting_segment`.
- Keep validator rejection and apply-time skip as distinct outcomes.
## Stage 2: LLM Validator Resilience
- Keep validator transport/provider/runtime call failures fatal.
- Downgrade malformed validator structured payloads into batch-scoped validator rejections plus module warnings.
- Downgrade oversized single-proposal validator inputs into per-proposal validator rejections plus module warnings.
- Preserve decision-cardinality enforcement as an internal invariant after malformed-payload degradation has synthesized complete decision sets.
Stable reason codes introduced or relied upon by this cutover:
- `empty_resulting_segment`
- `invalid_target_segment_id`
- `empty_original_text`
- `invalid_confidence`
- `validator_response_malformed`
- `validator_input_too_large`
- `proposal_response_malformed`
## Stage 3: Reporting, Diagnostics, and Docs
- Add module warning records to runner results and process reports.
- Keep correction-ledger entries per-correction only; do not add standalone warning rows.
- Record malformed proposal-generation and validator-batch warnings through report and diagnostics artifacts only.
- Keep successful runs quiet on stderr even when warnings are present.
- Update README and architecture/public-contract/diagnostics/validators/release-checklist docs to reflect the new behavior.
## Acceptance Criteria
- Empty `corrected_text` may delete words, but proposals that would blank the whole segment are rejected or skipped safely.
- Invalid proposal shape is rejected by validators, not by proposal generation.
- Malformed proposal-generation payloads succeed with warnings and zero proposals for the affected section.
- Malformed validator payloads reject only the affected validator batch and do not fail the module.
- Oversized single validator inputs reject only the affected proposal.
- Transport/provider/runtime LLM failures still fail the module and process.
- Successful runs with warnings still exit `0`, emit transcript output normally, keep stderr empty, and expose warnings in report/diagnostics artifacts.