eric/audita
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 6 Wiki Activity

Align prompts and strengthen validation context

Browse Source
This commit is contained in:
Eric Rakestraw
2026-09-25 12:23:10 +00:00
parent 3eb73e70fc
commit 325c4ea7d3
23 changed files with 329 additions and 133 deletions
Download Patch File Download Diff File Expand all files Collapse all files

33
docs/roadmap/remediation.md
View File

@@ -4,7 +4,7 @@
**Scope:** All four proposal prompts, three active LLM-validator prompts, two
inactive validator definitions, shared hardening text, response schemas, supplied
context, and downstream validation/application behavior.
**Status:** R16–R26 open. R1–R15 are resolved; their findings and resolution records
**Status:** R16–R26 addressed; see the resolution record below. R1–R15 are resolved; their findings and resolution records
remain in Git history through `b8089f4`.
## Architecture Assessment
@@ -180,8 +180,8 @@ a new literal-token rule or a guarantee of model accuracy.
**Category:** Optional maintenance cleanup.
**Evidence:** Registered [grammar_review](../../internal/prompts/assets/prompts/validators.grammar_review.yaml)
and [spoken_word_review](../../internal/prompts/assets/prompts/validators.spoken_word_review.yaml)
**Evidence:** Registered `grammar_review`
and `spoken_word_review`
definitions largely duplicate editorial review and have already diverged from its
updated deletion guidance. Neither is used by the [built-in chains](../../internal/validators/chains.go).
@@ -244,3 +244,30 @@ introduced.
framework solely to implement wording improvements.
- This document catalogs observations and recommendations only; no prompts,
application behavior, or release artifacts were changed by this update.
## Resolution Record
Implemented after the maintainer requested addressing all findings. The maintainer
explicitly selected allowing protected-name repetition cleanup with validation for
R21. The original evidence above describes the audited baseline, not current defects.
| Finding | Resolution and evidence |
| --- | --- |
| R16 | Grammar system and detailed rules consistently state the whole-word a/an exception. |
| R17 | Glossary guidance distinguishes correct protected spelling from a supported mishearing of another recognized term. |
| R18 | All proposal prompts explain literal case-sensitive matching and preserving disambiguating context; glossary explicitly describes replacing every matching occurrence. |
| R19 | Proposal prompts describe multiple source-based corrections per segment and the corrections envelope. Active validators require unchanged correction indices, including negative decisions, and the validations envelope. |
| R20 | Validators receive adjacent in-chunk source segments and mentioned glossary definitions. Optional context is reduced to fit the existing payload budget before required candidate data is considered oversized. Context selection and limits are documented in the validator internals. |
| R21 | Spoken-word protected-term checking permits reduced repetition only with surviving protected identities and existing capitalization checks. The user-approved exception leaves subsequent editorial and meaning validation in place; focused tests cover phrase replacement, literal deletion, last-reference removal, identity substitution, spelling degradation, and unchanged grammar protection. |
| R22 | Spoken-form guidance explicitly evaluates full before/after text for every correction. |
| R23 | Meaning-reversal guidance includes deletion of negation, with contextual judgment rather than a token-based prohibition. |
| R24 | Consumer inspection found only unused internal selections and their tests. Removed both inactive definitions, type selections, metadata entries, and associated cases; updated the seven-prompt inventory. No operator-selectable setting or built-in chain changed. |
| R25 | Proposal confidence describes whether the correction is warranted; validator confidence describes confidence in either verdict. Runtime thresholds are unchanged. |
| R26 | Hardening describes evidence-supported correction of source material, and grammar no longer promises a later pipeline stage. |
Validation: the complete `go test ./...` suite passed, including embedded prompt
preparation through PromptKit. New context tests verify supplied neighboring evidence,
relevant glossary summaries, chunk boundaries, serialized payloads, and budget effects.
The existing oversized-input regression confirms that a large neighbor does not
invalidate another otherwise usable candidate. Documentation links and whitespace
checks passed. No live model quality claims or release publication are implied.