eric/audita
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 2 Wiki Activity

Make module-stage LLM handling resilient and report warnings

Browse Source
This commit is contained in:
Eric Rakestraw
2026-05-23 10:07:06 -05:00
parent a84941d681
commit a3655f5540
43 changed files with 856 additions and 217 deletions
Download Patch File Download Diff File Expand all files Collapse all files

32
docs/architecture/architecture.md
View File

@@ -133,11 +133,16 @@ internal/framework/validators/
llm_batching.go
llm_validators.go
internal/framework/warnings/
warnings.go
internal/validators/
metadata/
metadata.go
registry.go
chains.go
proposal_shape/
validator.go
confidence_threshold/
validator.go
original_text_presence/
@@ -242,6 +247,7 @@ Important behavior details:
- Explicit `--modules grammar`, `--modules glossary`, `--modules homophones`, and `--modules spoken_word` continue to run production module paths with LLM-backed proposal generation and validator-chain execution.
- Default runs (without explicit module selection) perform LLM calls through production module and validator paths.
- Success path is generally quiet on stderr.
- Malformed module-stage LLM payloads degrade to validator rejections and module warnings instead of aborting the run.
- Source IDs are preserved into a canonical transcript before normalization; normalization then reassigns output IDs sequentially from `1`.
## Implemented data contracts
@@ -539,20 +545,20 @@ Validator execution classification metadata:
Stable built-in validator keys:
- deterministic:
- `proposal_shape`
- `confidence_threshold`
- `original_text_presence`
- `non_empty_corrected_text`
- `non_empty_corrected_text` (historical key name; current semantics reject empty resulting segment text)
- `no_effect`
- `protected_terms`
- LLM-backed:
- `spoken_form_plausibility`
- `meaning_reversal_review`
- `editorial_review`
- `grammar_review`
- `spoken_word_review`
Built-in module chains:
- `glossary`:
- `proposal_shape`
- `no_effect`
- `original_text_presence`
- `confidence_threshold`
@@ -561,6 +567,7 @@ Built-in module chains:
- `spoken_form_plausibility`
- `meaning_reversal_review`
- `homophones`:
- `proposal_shape`
- `no_effect`
- `original_text_presence`
- `confidence_threshold`
@@ -569,20 +576,22 @@ Built-in module chains:
- `spoken_form_plausibility`
- `meaning_reversal_review`
- `spoken_word`:
- `proposal_shape`
- `no_effect`
- `original_text_presence`
- `confidence_threshold`
- `protected_terms`
- `non_empty_corrected_text`
- `spoken_word_review`
- `editorial_review`
- `meaning_reversal_review`
- `grammar`:
- `proposal_shape`
- `no_effect`
- `original_text_presence`
- `confidence_threshold`
- `protected_terms`
- `non_empty_corrected_text`
- `grammar_review`
- `editorial_review`
- `meaning_reversal_review`
1.0 boundary:
@@ -595,12 +604,14 @@ Built-in module chains:
- prompt builders for:
- spoken-form plausibility
- meaning reversal detection
- editorial review
- grammar review
- spoken-word review
- editorial review
- grammar review
- spoken-word review
- deterministic batching by `validation_max_prompt_tokens`;
- strict cardinality validation of structured LLM decisions (missing/duplicate/unknown indexes fail);
- safe failure behavior for malformed/invalid structured responses.
- strict cardinality validation of synthesized validator decision sets;
- malformed validator payloads reject only the affected batch with warnings;
- oversized single-proposal validator inputs reject only the affected proposal;
- transport/provider/runtime LLM call failures remain fatal.
`internal/framework/runner` wires LLM validators into existing validator chains using:
- the internal structured LLM client abstraction (`contracts.StructuredLLMClient`);
@@ -726,6 +737,7 @@ Current process reports include diagnostics metadata references for:
Current process reports also include:
- module-level results (when runner modules execute), including applied/skipped proposal changes;
- run-level module summary totals and failed module instance metadata.
- module-level warning records for malformed proposal-generation payloads and malformed validator batches.
- module-level validator decisions and validator rejections.
- optional decision-level diagnostic artifact paths for validator LLM interactions when available.
- stable validator keys in `validator_name` fields for validator decisions/rejections.

6
docs/architecture/diagnostics.md
View File

@@ -63,6 +63,7 @@ Ledger records are flattened review entries derived from module results and incl
- deterministic and LLM validator decision snapshots using stable validator keys.
Validator rejection and proposal-application skip are distinct dispositions.
Module warnings are reported in module results and diagnostics metadata, but do not create standalone correction-ledger rows.
## Report references
@@ -71,6 +72,8 @@ Validator rejection and proposal-application skip are distinct dispositions.
- correction ledger artifact;
- existing transcript/normalization/chunking/invocation/effective-config artifacts.
Module report entries also include warning records for malformed proposal-generation payloads and malformed validator batches.
## Retention behavior
Run-directory retention follows configured policy:
@@ -93,6 +96,9 @@ When debugging:
- validator rejections:
- inspect `correction-ledger.json` rejected entries and matching validator decisions;
- inspect validator response diagnostics payloads.
- module warnings:
- inspect module `warnings` entries in `report.json` or `--report-json`;
- follow any diagnostic artifact path on the warning to the recorded error/response payload.
- application skips:
- inspect `correction-ledger.json` skipped entries and skip reason codes;
- compare with validator decisions to distinguish validation rejection vs apply-time skip.

3
docs/architecture/public-contract.md
View File

@@ -93,6 +93,7 @@ Current values:
`--report-json` output and diagnostics run-dir `report.json` use the same report schema metadata.
Validator decision/rejection records in reports use stable validator keys in `validator_name`.
Module results may also include warning records for malformed module-stage LLM payloads.
Report diagnostics metadata includes artifact-path fields for utilization diagnostics and correction ledger when diagnostics initialization succeeds.
## Diagnostics directory behavior
@@ -123,6 +124,7 @@ Success behavior:
- with `--output`, stdout is empty
- without `--output`, stdout contains only transcript JSON in selected output schema
- report JSON is not written to stdout
- success stderr remains empty even when reports/diagnostics contain module warnings
Failure behavior:
- stderr contains human-readable error summary
@@ -149,6 +151,7 @@ Config files should reference secrets via environment variable names (`api_key_e
## Compatibility and deprecation policy
- Existing stable schema names, report metadata keys, and top-level command behavior are treated as public contract.
- Existing stable validator keys remain public contract values even when validator semantics are refined.
- Compatibility inputs (legacy flags/env aliases) may remain during transition windows.
- Any planned removal or behavior change should include clear compatibility notes and migration guidance.

16
docs/architecture/validators.md
View File

@@ -8,6 +8,7 @@ For LLM-backed validator prompt asset details, see [`docs/prompts.md`](prompts.m
Built-in validator construction is package-owned under `internal/validators/<validator_key>`:
- `internal/validators/confidence_threshold`
- `internal/validators/proposal_shape`
- `internal/validators/original_text_presence`
- `internal/validators/non_empty_corrected_text`
- `internal/validators/no_effect`
@@ -45,12 +46,14 @@ Current 1.0 boundary:
### Deterministic validators
- `proposal_shape`
- rejects malformed proposal fields before other validators run.
- `confidence_threshold`
- checks proposal confidence against module-specific configured threshold.
- `original_text_presence`
- ensures target segment exists and `original_text` exists in current working segment text.
- `non_empty_corrected_text`
- rejects blank/whitespace-only `corrected_text`.
- rejects proposals whose previewed resulting segment text would be empty or whitespace-only.
- `no_effect`
- rejects proposals where `original_text == corrected_text`.
- `protected_terms`
@@ -71,6 +74,7 @@ Current 1.0 boundary:
Current built-in chains resolved from `internal/validators/chains.go`:
- `glossary`
- `proposal_shape`
- `no_effect`
- `original_text_presence`
- `confidence_threshold`
@@ -80,6 +84,7 @@ Current built-in chains resolved from `internal/validators/chains.go`:
- `meaning_reversal_review`
- `homophones`
- `proposal_shape`
- `no_effect`
- `original_text_presence`
- `confidence_threshold`
@@ -89,6 +94,7 @@ Current built-in chains resolved from `internal/validators/chains.go`:
- `meaning_reversal_review`
- `spoken_word`
- `proposal_shape`
- `no_effect`
- `original_text_presence`
- `confidence_threshold`
@@ -98,6 +104,7 @@ Current built-in chains resolved from `internal/validators/chains.go`:
- `meaning_reversal_review`
- `grammar`
- `proposal_shape`
- `no_effect`
- `original_text_presence`
- `confidence_threshold`
@@ -119,7 +126,9 @@ Both variants preserve existing behavior and report the stable key `protected_te
- modules execute serially;
- section proposal work can run concurrently within a module;
- deterministic validators run before LLM-backed validators;
- malformed/missing/duplicate/unknown LLM validator decisions fail safely;
- malformed module proposal payloads are downgraded to section-scoped module warnings with zero proposals for the affected section rather than module failure;
- malformed/missing/duplicate/unknown LLM validator decisions reject the affected validator batch with warnings instead of failing the module;
- oversized single-proposal validator inputs reject only the affected proposal under that validator;
- approved proposals are applied once per module after section work settles.
## Validator rejections vs proposal-application skips
@@ -128,12 +137,15 @@ Both variants preserve existing behavior and report the stable key `protected_te
- proposal is denied by validator-chain review and appears in validator rejection reporting with validator key and reason code.
- proposal-application skip:
- proposal passed validators but could not be applied under replacement-policy semantics (for example no matching span at apply time).
- module warning:
- malformed proposal-generation payloads and malformed validator batches are recorded in module warning records and diagnostics without writing success stderr.
These are separate outcomes and are reported separately.
## Reporting and diagnostics identity
- report validator decision/rejection entries use stable validator keys in `validator_name`.
- report module results include warning records for malformed module-stage LLM payloads.
- validator LLM diagnostics include validator identity in interaction metadata and structured response schema metadata.
- correction ledger entries include deterministic and LLM validator decision snapshots keyed by the same stable validator keys, and keep validator rejection distinct from application-level skip.

4
docs/release-checklist.md
View File

@@ -40,6 +40,7 @@ Use this checklist before cutting a pre-1.0 or 1.0 release candidate.
- Verify structured response schemas are attached via `response_format.type=json_schema`.
- Verify diagnostics metadata includes structured schema `id/version/name/sha256`.
- Verify provider output is still locally decoded/validated before use.
- Verify malformed module-stage structured payloads degrade to warnings/rejections instead of failing the run.
## Report and diagnostics schema checks
@@ -67,6 +68,7 @@ Use this checklist before cutting a pre-1.0 or 1.0 release candidate.
- Verify prompt metadata appears in LLM request metadata diagnostics:
- `prompt_id`, `prompt_version`, `prompt_source`, `embedded_path`, `sha256`.
- Verify stable validator keys appear in report decisions/rejections.
- Verify module warning records appear in reports for malformed proposal-generation payloads and malformed validator batches.
- Verify built-in validator chains resolve and execute for default and explicit module runs.
## Utilization diagnostics checks
@@ -96,6 +98,8 @@ Use this checklist before cutting a pre-1.0 or 1.0 release candidate.
## Failure and cancellation checks
- Verify controlled failure paths retain diagnostics and produce best-effort failure reports.
- Verify malformed proposal-generation payloads keep exit code `0`, keep stderr empty on success, and record warnings in reports/diagnostics.
- Verify malformed validator payloads reject only the affected batch and do not fail the module.
- Verify timeout/cancellation paths exit nonzero, do not hang, and retain failure diagnostics when initialized.
## Release fixture/idempotence checks

72
docs/roadmap/publish.md Normal file
View File

@@ -0,0 +1,72 @@
# Hard-Cutover Roadmap for Module-Stage LLM Resilience
## Summary
This roadmap captures the module-stage resilience work for Audita:
- fail fast on initialization, configuration, schema, and other pre-module setup errors;
- remain resilient during module execution when LLM payloads are malformed or individual proposed corrections are invalid;
- reject bad corrections through validator/reporting paths instead of aborting the module or process;
- keep success stderr quiet; surface warnings only through report and diagnostics artifacts;
- use a hard cutover only, with no compatibility aliases or transitional code.
Locked decisions:
- keep the stable public validator key `non_empty_corrected_text`;
- change that validator’s behavior to mean “the resulting segment text must not be empty/whitespace-only after applying the proposal preview”;
- malformed LLM-validator batch payloads reject the entire affected batch under that validator and continue;
- proposal/validator transport failures, timeouts, and provider/runtime call failures remain fatal;
- malformed structured payloads are downgraded; non-malformed runtime call failures are not.
## Stage 1: Proposal Intake Hardening
- Stop treating invalid individual structured corrections as fatal during proposal generation.
- Preserve returned correction ordering and proposal-index assignment even when individual corrections are malformed.
- Allow `corrected_text == ""` when the resulting segment remains non-empty after previewed application.
- Downgrade malformed proposal-generation structured payloads into section-scoped warnings with zero proposals for that section.
- Keep proposal-generation transport/provider/runtime call failures fatal.
Deterministic validation changes:
- Add `proposal_shape` as a built-in deterministic validator and run it first in every built-in module chain.
- Reject malformed proposal fields with stable reason codes:
- `invalid_target_segment_id`
- `empty_original_text`
- `invalid_confidence`
- Keep `non_empty_corrected_text` as the stable validator key, but change its semantics to reject only `empty_resulting_segment`.
- Keep validator rejection and apply-time skip as distinct outcomes.
## Stage 2: LLM Validator Resilience
- Keep validator transport/provider/runtime call failures fatal.
- Downgrade malformed validator structured payloads into batch-scoped validator rejections plus module warnings.
- Downgrade oversized single-proposal validator inputs into per-proposal validator rejections plus module warnings.
- Preserve decision-cardinality enforcement as an internal invariant after malformed-payload degradation has synthesized complete decision sets.
Stable reason codes introduced or relied upon by this cutover:
- `empty_resulting_segment`
- `invalid_target_segment_id`
- `empty_original_text`
- `invalid_confidence`
- `validator_response_malformed`
- `validator_input_too_large`
- `proposal_response_malformed`
## Stage 3: Reporting, Diagnostics, and Docs
- Add module warning records to runner results and process reports.
- Keep correction-ledger entries per-correction only; do not add standalone warning rows.
- Record malformed proposal-generation and validator-batch warnings through report and diagnostics artifacts only.
- Keep successful runs quiet on stderr even when warnings are present.
- Update README and architecture/public-contract/diagnostics/validators/release-checklist docs to reflect the new behavior.
## Acceptance Criteria
- Empty `corrected_text` may delete words, but proposals that would blank the whole segment are rejected or skipped safely.
- Invalid proposal shape is rejected by validators, not by proposal generation.
- Malformed proposal-generation payloads succeed with warnings and zero proposals for the affected section.
- Malformed validator payloads reject only the affected validator batch and do not fail the module.
- Oversized single validator inputs reject only the affected proposal.
- Transport/provider/runtime LLM failures still fail the module and process.
- Successful runs with warnings still exit `0`, emit transcript output normally, keep stderr empty, and expose warnings in report/diagnostics artifacts.