73 lines
4.1 KiB
Markdown
73 lines
4.1 KiB
Markdown
# Hard-Cutover Roadmap for Module-Stage LLM Resilience
|
||
|
||
## Summary
|
||
|
||
This roadmap captures the module-stage resilience work for Audita:
|
||
|
||
- fail fast on initialization, configuration, schema, and other pre-module setup errors;
|
||
- remain resilient during module execution when LLM payloads are malformed or individual proposed corrections are invalid;
|
||
- reject bad corrections through validator/reporting paths instead of aborting the module or process;
|
||
- keep success stderr quiet; surface warnings only through report and diagnostics artifacts;
|
||
- use a hard cutover only, with no compatibility aliases or transitional code.
|
||
|
||
Locked decisions:
|
||
|
||
- keep the stable public validator key `non_empty_corrected_text`;
|
||
- change that validator’s behavior to mean “the resulting segment text must not be empty/whitespace-only after applying the proposal preview”;
|
||
- malformed LLM-validator batch payloads reject the entire affected batch under that validator and continue;
|
||
- proposal/validator transport failures, timeouts, and provider/runtime call failures remain fatal;
|
||
- malformed structured payloads are downgraded; non-malformed runtime call failures are not.
|
||
|
||
## Stage 1: Proposal Intake Hardening
|
||
|
||
- Stop treating invalid individual structured corrections as fatal during proposal generation.
|
||
- Preserve returned correction ordering and proposal-index assignment even when individual corrections are malformed.
|
||
- Allow `corrected_text == ""` when the resulting segment remains non-empty after previewed application.
|
||
- Downgrade malformed proposal-generation structured payloads into section-scoped warnings with zero proposals for that section.
|
||
- Keep proposal-generation transport/provider/runtime call failures fatal.
|
||
|
||
Deterministic validation changes:
|
||
|
||
- Add `proposal_shape` as a built-in deterministic validator and run it first in every built-in module chain.
|
||
- Reject malformed proposal fields with stable reason codes:
|
||
- `invalid_target_segment_id`
|
||
- `empty_original_text`
|
||
- `invalid_confidence`
|
||
- Keep `non_empty_corrected_text` as the stable validator key, but change its semantics to reject only `empty_resulting_segment`.
|
||
- Keep validator rejection and apply-time skip as distinct outcomes.
|
||
|
||
## Stage 2: LLM Validator Resilience
|
||
|
||
- Keep validator transport/provider/runtime call failures fatal.
|
||
- Downgrade malformed validator structured payloads into batch-scoped validator rejections plus module warnings.
|
||
- Downgrade oversized single-proposal validator inputs into per-proposal validator rejections plus module warnings.
|
||
- Preserve decision-cardinality enforcement as an internal invariant after malformed-payload degradation has synthesized complete decision sets.
|
||
|
||
Stable reason codes introduced or relied upon by this cutover:
|
||
|
||
- `empty_resulting_segment`
|
||
- `invalid_target_segment_id`
|
||
- `empty_original_text`
|
||
- `invalid_confidence`
|
||
- `validator_response_malformed`
|
||
- `validator_input_too_large`
|
||
- `proposal_response_malformed`
|
||
|
||
## Stage 3: Reporting, Diagnostics, and Docs
|
||
|
||
- Add module warning records to runner results and process reports.
|
||
- Keep correction-ledger entries per-correction only; do not add standalone warning rows.
|
||
- Record malformed proposal-generation and validator-batch warnings through report and diagnostics artifacts only.
|
||
- Keep successful runs quiet on stderr even when warnings are present.
|
||
- Update README and architecture/public-contract/diagnostics/validators/release-checklist docs to reflect the new behavior.
|
||
|
||
## Acceptance Criteria
|
||
|
||
- Empty `corrected_text` may delete words, but proposals that would blank the whole segment are rejected or skipped safely.
|
||
- Invalid proposal shape is rejected by validators, not by proposal generation.
|
||
- Malformed proposal-generation payloads succeed with warnings and zero proposals for the affected section.
|
||
- Malformed validator payloads reject only the affected validator batch and do not fail the module.
|
||
- Oversized single validator inputs reject only the affected proposal.
|
||
- Transport/provider/runtime LLM failures still fail the module and process.
|
||
- Successful runs with warnings still exit `0`, emit transcript output normally, keep stderr empty, and expose warnings in report/diagnostics artifacts.
|