Make module-stage LLM handling resilient and report warnings

2026-05-23 10:07:06 -05:00
parent a84941d681
commit a3655f5540
43 changed files with 856 additions and 217 deletions
--- a/docs/architecture/architecture.md
+++ b/docs/architecture/architecture.md
@@ -133,11 +133,16 @@ internal/framework/validators/
  llm_batching.go
  llm_validators.go

+internal/framework/warnings/
+  warnings.go
+
 internal/validators/
  metadata/
    metadata.go
  registry.go
  chains.go
+  proposal_shape/
+    validator.go
  confidence_threshold/
    validator.go
  original_text_presence/
@@ -242,6 +247,7 @@ Important behavior details:
 - Explicit `--modules grammar`, `--modules glossary`, `--modules homophones`, and `--modules spoken_word` continue to run production module paths with LLM-backed proposal generation and validator-chain execution.
 - Default runs (without explicit module selection) perform LLM calls through production module and validator paths.
 - Success path is generally quiet on stderr.
+- Malformed module-stage LLM payloads degrade to validator rejections and module warnings instead of aborting the run.
 - Source IDs are preserved into a canonical transcript before normalization; normalization then reassigns output IDs sequentially from `1`.

 ## Implemented data contracts
@@ -539,20 +545,20 @@ Validator execution classification metadata:

 Stable built-in validator keys:
 - deterministic:
+  - `proposal_shape`
  - `confidence_threshold`
  - `original_text_presence`
-  - `non_empty_corrected_text`
+  - `non_empty_corrected_text` (historical key name; current semantics reject empty resulting segment text)
  - `no_effect`
  - `protected_terms`
 - LLM-backed:
  - `spoken_form_plausibility`
  - `meaning_reversal_review`
  - `editorial_review`
-  - `grammar_review`
-  - `spoken_word_review`

 Built-in module chains:
 - `glossary`:
+  - `proposal_shape`
  - `no_effect`
  - `original_text_presence`
  - `confidence_threshold`
@@ -561,6 +567,7 @@ Built-in module chains:
  - `spoken_form_plausibility`
  - `meaning_reversal_review`
 - `homophones`:
+  - `proposal_shape`
  - `no_effect`
  - `original_text_presence`
  - `confidence_threshold`
@@ -569,20 +576,22 @@ Built-in module chains:
  - `spoken_form_plausibility`
  - `meaning_reversal_review`
 - `spoken_word`:
+  - `proposal_shape`
  - `no_effect`
  - `original_text_presence`
  - `confidence_threshold`
  - `protected_terms`
  - `non_empty_corrected_text`
-  - `spoken_word_review`
+  - `editorial_review`
  - `meaning_reversal_review`
 - `grammar`:
+  - `proposal_shape`
  - `no_effect`
  - `original_text_presence`
  - `confidence_threshold`
  - `protected_terms`
  - `non_empty_corrected_text`
-  - `grammar_review`
+  - `editorial_review`
  - `meaning_reversal_review`

 1.0 boundary:
@@ -595,12 +604,14 @@ Built-in module chains:
 - prompt builders for:
  - spoken-form plausibility
  - meaning reversal detection
-  - editorial review
-  - grammar review
-  - spoken-word review
+- editorial review
+- grammar review
+- spoken-word review
 - deterministic batching by `validation_max_prompt_tokens`;
- strict cardinality validation of structured LLM decisions (missing/duplicate/unknown indexes fail);
- safe failure behavior for malformed/invalid structured responses.
+- strict cardinality validation of synthesized validator decision sets;
+- malformed validator payloads reject only the affected batch with warnings;
+- oversized single-proposal validator inputs reject only the affected proposal;
+- transport/provider/runtime LLM call failures remain fatal.

 `internal/framework/runner` wires LLM validators into existing validator chains using:
 - the internal structured LLM client abstraction (`contracts.StructuredLLMClient`);
@@ -726,6 +737,7 @@ Current process reports include diagnostics metadata references for:
 Current process reports also include:
 - module-level results (when runner modules execute), including applied/skipped proposal changes;
 - run-level module summary totals and failed module instance metadata.
+- module-level warning records for malformed proposal-generation payloads and malformed validator batches.
 - module-level validator decisions and validator rejections.
 - optional decision-level diagnostic artifact paths for validator LLM interactions when available.
 - stable validator keys in `validator_name` fields for validator decisions/rejections.
--- a/docs/architecture/diagnostics.md
+++ b/docs/architecture/diagnostics.md
@@ -63,6 +63,7 @@ Ledger records are flattened review entries derived from module results and incl
 - deterministic and LLM validator decision snapshots using stable validator keys.

 Validator rejection and proposal-application skip are distinct dispositions.
+Module warnings are reported in module results and diagnostics metadata, but do not create standalone correction-ledger rows.

 ## Report references

@@ -71,6 +72,8 @@ Validator rejection and proposal-application skip are distinct dispositions.
 - correction ledger artifact;
 - existing transcript/normalization/chunking/invocation/effective-config artifacts.

+Module report entries also include warning records for malformed proposal-generation payloads and malformed validator batches.
+
 ## Retention behavior

 Run-directory retention follows configured policy:
@@ -93,6 +96,9 @@ When debugging:
 - validator rejections:
  - inspect `correction-ledger.json` rejected entries and matching validator decisions;
  - inspect validator response diagnostics payloads.
+- module warnings:
+  - inspect module `warnings` entries in `report.json` or `--report-json`;
+  - follow any diagnostic artifact path on the warning to the recorded error/response payload.
 - application skips:
  - inspect `correction-ledger.json` skipped entries and skip reason codes;
  - compare with validator decisions to distinguish validation rejection vs apply-time skip.
--- a/docs/architecture/public-contract.md
+++ b/docs/architecture/public-contract.md
@@ -93,6 +93,7 @@ Current values:
 `--report-json` output and diagnostics run-dir `report.json` use the same report schema metadata.

 Validator decision/rejection records in reports use stable validator keys in `validator_name`.
+Module results may also include warning records for malformed module-stage LLM payloads.
 Report diagnostics metadata includes artifact-path fields for utilization diagnostics and correction ledger when diagnostics initialization succeeds.

 ## Diagnostics directory behavior
@@ -123,6 +124,7 @@ Success behavior:
 - with `--output`, stdout is empty
 - without `--output`, stdout contains only transcript JSON in selected output schema
 - report JSON is not written to stdout
+- success stderr remains empty even when reports/diagnostics contain module warnings

 Failure behavior:
 - stderr contains human-readable error summary
@@ -149,6 +151,7 @@ Config files should reference secrets via environment variable names (`api_key_e
 ## Compatibility and deprecation policy

 - Existing stable schema names, report metadata keys, and top-level command behavior are treated as public contract.
+- Existing stable validator keys remain public contract values even when validator semantics are refined.
 - Compatibility inputs (legacy flags/env aliases) may remain during transition windows.
 - Any planned removal or behavior change should include clear compatibility notes and migration guidance.

--- a/docs/architecture/validators.md
+++ b/docs/architecture/validators.md
@@ -8,6 +8,7 @@ For LLM-backed validator prompt asset details, see [`docs/prompts.md`](prompts.m

 Built-in validator construction is package-owned under `internal/validators/<validator_key>`:
 - `internal/validators/confidence_threshold`
+- `internal/validators/proposal_shape`
 - `internal/validators/original_text_presence`
 - `internal/validators/non_empty_corrected_text`
 - `internal/validators/no_effect`
@@ -45,12 +46,14 @@ Current 1.0 boundary:

 ### Deterministic validators

+- `proposal_shape`
+  - rejects malformed proposal fields before other validators run.
 - `confidence_threshold`
  - checks proposal confidence against module-specific configured threshold.
 - `original_text_presence`
  - ensures target segment exists and `original_text` exists in current working segment text.
 - `non_empty_corrected_text`
-  - rejects blank/whitespace-only `corrected_text`.
+  - rejects proposals whose previewed resulting segment text would be empty or whitespace-only.
 - `no_effect`
  - rejects proposals where `original_text == corrected_text`.
 - `protected_terms`
@@ -71,6 +74,7 @@ Current 1.0 boundary:
 Current built-in chains resolved from `internal/validators/chains.go`:

 - `glossary`
+  - `proposal_shape`
  - `no_effect`
  - `original_text_presence`
  - `confidence_threshold`
@@ -80,6 +84,7 @@ Current built-in chains resolved from `internal/validators/chains.go`:
  - `meaning_reversal_review`

 - `homophones`
+  - `proposal_shape`
  - `no_effect`
  - `original_text_presence`
  - `confidence_threshold`
@@ -89,6 +94,7 @@ Current built-in chains resolved from `internal/validators/chains.go`:
  - `meaning_reversal_review`

 - `spoken_word`
+  - `proposal_shape`
  - `no_effect`
  - `original_text_presence`
  - `confidence_threshold`
@@ -98,6 +104,7 @@ Current built-in chains resolved from `internal/validators/chains.go`:
  - `meaning_reversal_review`

 - `grammar`
+  - `proposal_shape`
  - `no_effect`
  - `original_text_presence`
  - `confidence_threshold`
@@ -119,7 +126,9 @@ Both variants preserve existing behavior and report the stable key `protected_te
 - modules execute serially;
 - section proposal work can run concurrently within a module;
 - deterministic validators run before LLM-backed validators;
- malformed/missing/duplicate/unknown LLM validator decisions fail safely;
+- malformed module proposal payloads are downgraded to section-scoped module warnings with zero proposals for the affected section rather than module failure;
+- malformed/missing/duplicate/unknown LLM validator decisions reject the affected validator batch with warnings instead of failing the module;
+- oversized single-proposal validator inputs reject only the affected proposal under that validator;
 - approved proposals are applied once per module after section work settles.

 ## Validator rejections vs proposal-application skips
@@ -128,12 +137,15 @@ Both variants preserve existing behavior and report the stable key `protected_te
  - proposal is denied by validator-chain review and appears in validator rejection reporting with validator key and reason code.
 - proposal-application skip:
  - proposal passed validators but could not be applied under replacement-policy semantics (for example no matching span at apply time).
+- module warning:
+  - malformed proposal-generation payloads and malformed validator batches are recorded in module warning records and diagnostics without writing success stderr.

 These are separate outcomes and are reported separately.

 ## Reporting and diagnostics identity

 - report validator decision/rejection entries use stable validator keys in `validator_name`.
+- report module results include warning records for malformed module-stage LLM payloads.
 - validator LLM diagnostics include validator identity in interaction metadata and structured response schema metadata.
 - correction ledger entries include deterministic and LLM validator decision snapshots keyed by the same stable validator keys, and keep validator rejection distinct from application-level skip.