audita/docs/roadmap/remediation.md

Prompt Audit: Findings and Recommended Changes

Reviewed: 2026-09-25, source commit b8089f4. Scope: All four proposal prompts, three active LLM-validator prompts, two inactive validator definitions, shared hardening text, response schemas, supplied context, and downstream validation/application behavior. Status: R16–R26 addressed; see the resolution record below. R1–R15 are resolved; their findings and resolution records remain in Git history through b8089f4.

Architecture Assessment

PromptKit owns structural response validation and repair. Audita owns domain validation, decision reconciliation, and application. The recent span-deletion and partial-decision changes are aligned across these boundaries. This audit found prompt contradictions, incomplete explanations of application semantics, and opportunities to improve instructions and supplied context.

R16–R19 and R22–R23/R25–R26 are focused wording improvements suitable for consideration before v2.0.0. R20–R21 require separate design or policy decisions; R24 is optional cleanup after checking consumers. Their inclusion here does not make every item a release gate or authorize an unreviewed behavioral expansion. The shared editorial validator's broad remit is intentional and needs no new module-scope enforcement as part of this audit.

R16 — P2: Make the Grammar Article Exception Consistent

Category: Conflicting instructions.

Evidence: The grammar prompt allows whole-word a/an substitution, while its system instruction prohibits word substitution and a later rule permits only capitalization changes to written words. A model can follow those prohibitions and omit the expressly permitted correction.

Recommended change: State the existing article exception consistently in the system message and word-preservation rule. Consolidate overlapping prohibitions where helpful without expanding grammar into spelling, homophone, or content edits.

Acceptance: The prompt unambiguously permits a/an correction under the existing conditions while retaining the module's conservative formatting remit.

R17 — P2: Clarify the Glossary Mishearing Exception

Category: Prompt conflict with intended glossary correction behavior.

Evidence: The glossary prompt permits replacing an existing glossary term with another for a plausible mishearing, then treats present glossary names as protected and prohibits altering them. The glossary-specific protected-term validation deliberately allows protected-term swaps, subject to subsequent validation.

Recommended change: Distinguish preserving correct glossary spellings from correcting a contextually supported misidentification. Make the plausible-mishearing exception explicit while retaining acoustic plausibility and context requirements.

Acceptance: Recognizing a source word as a glossary term does not itself imply that the prompt forbids a supported correction to another term. Unrelated name substitution and gratuitous spelling/capitalization normalization remain outside scope; no validator relaxation is implied by this wording change.

R18 — P2: Explain Literal Replacement and Context Spans

Category: Incomplete application contract in proposal instructions.

Evidence: Glossary uses replace_all without explaining that every matching occurrence in the target segment changes. Grammar, homophones, and spoken-word require unique matches, but all proposal prompts ask for only the exact span needing replacement, which can discourage including unchanged context. Preview/application uses literal, case-sensitive matching.

Recommended change: Explain glossary's replace-all behavior explicitly. Allow unchanged surrounding text in a source span when needed to identify the intended edit, preserving that context in the replacement. Explain literal matching without imposing uniqueness on glossary or adding automatic whitespace cleanup.

Acceptance: Guidance explains how to target one intended occurrence when a shorter span repeats, and that a glossary proposal using a repeated span affects all matches. Existing module replacement policies and eligible whitespace spans remain unchanged.

R19 — P2: Align Output Instructions with Correction and Decision Identity

Category: Ambiguous response shape and identity guidance.

Evidence: Proposal prompts request only changed “segments,” although their schema contains correction proposals and multiple proposals can target one segment. Validator payloads contain both segment id and correction_index; the validator prompts request a decision for each correction index without expressly warning against substituting segment IDs or renumbering. Both schemas require a top-level object containing a named list, which prose currently leaves largely to the schema.

Recommended change: Request proposed corrections rather than changed segments, allow multiple proposals per segment, and explain that their source spans refer to the supplied text rather than imagined results of other proposals. Tell validators to copy each supplied correction_index exactly, including negative decisions, without renumbering or substituting segment IDs. Show the top-level {"corrections":[...]} and {"validations":[...]} shapes concisely.

Acceptance: Instructions match the supplied schemas and identity model without requiring one proposal per segment. Validators are still asked for complete decision sets; tolerant reconciliation remains a fallback, not the preferred output.

R20 — P2: Evaluate Context Parity for LLM Validation

Category: Input-context limitation; separate design work.

Evidence: Proposal generation supplies section text and glossary data. The validator variable builder supplies candidate before/after segments, categories, and transcript description, but not neighboring transcript segments or glossary definitions. Spoken-form and editorial prompts ask for contextual judgments; evidence supporting the proposal may be absent during validation. Other batched candidates are not reliable context.

Recommended change: Evaluate supplying relevant glossary information and neighboring transcript context to validators, including its effect on token estimation and batching. Choose the context scope in a separate design pass. Avoid asking models to rely on evidence they do not receive; do not silently weaken validation or imply access to audio.

Acceptance: A selected design states which evidence each validator receives and how it fits existing budgets. Representative examples distinguish cases needing additional context from cases already decidable from the segment. This is not a requirement to copy the entire section or glossary into every request.

R21 — P2: Decide How Protected-Term Repetition Should Be Handled

Category: Prompt/validator policy tension; separate decision required.

Evidence: The spoken-word prompt permits accidental repetition cleanup while preserving named entities. The protected-term check requires each protected identity's occurrence count to be preserved. Thus Jesters Jesters arrived to Jesters arrived can be rejected even when repetition is accidental and the named entity remains.

Recommended change: Explicitly decide whether preservation of identity should permit removing a redundant occurrence. Evaluate a narrow allowance for supported repetition cleanup, retaining ordinary editorial and meaning checks. If current count preservation is retained, document that as a deliberate policy choice rather than silently making the prompt more restrictive to fit existing code.

Acceptance: Prompt and validation expectations agree after the policy decision. Any allowance preserves substantive references and intentional emphasis; any restriction is explicitly selected. This audit does not authorize a general relaxation of protected-term validation.

R22 — P3: Clarify Full-Segment Spoken-Form Assessment

Category: Validator wording improvement.

Evidence: The spoken-form validator explicitly directs full before/after comparison for deletions, while its general criteria discuss source and replacement spans and surrounding segment context. Full segment text is available for every candidate.

Recommended change: Explicitly direct full before/after assessment for all corrections, retaining the narrow acoustic/mistranscription remit and the distinction from style or readability review.

Acceptance: Guidance consistently uses the available segment evidence without turning this validator into a general editorial or semantic-equivalence check.

R23 — P3: Add a Deletion Example to Meaning-Reversal Guidance

Category: Optional illustrative improvement.

Evidence: The meaning-reversal prompt correctly explains empty replacement and full-segment comparison, but its concrete examples focus on word substitutions such as visible/invisible and up/down.

Recommended change: Add a concise example where deleting a negation reverses meaning. Preserve context-sensitive judgment and the validator's narrow scope.

Acceptance: The example illustrates why a deletion can reverse meaning without prohibiting empty replacements or making every word deletion a reversal. It is not a new literal-token rule or a guarantee of model accuracy.

R24 — P3: Review the Two Inactive Validator Definitions

Category: Optional maintenance cleanup.

Evidence: Registered grammar_review and spoken_word_review definitions largely duplicate editorial review and have already diverged from its updated deletion guidance. Neither is used by the built-in chains.

Recommended change: Check consumers and retained API uses, then retire the unused definitions and associated registrations if nothing needs them. If retained, document their purpose and keep relevant guidance consistent. Do not activate them as extra validation stages merely because they exist.

Acceptance: Each retained definition has a clear purpose; removal, if selected, updates actual consumers, tests, and prompt inventory, with any compatibility implications considered explicitly.

R25 — P3: Define What Confidence Means

Category: Underspecified output semantics.

Evidence: Proposal and validator prompts specify the confidence range but not its interpretation. Proposal confidence drives threshold filtering. Validator confidence is range-checked but does not determine the verdict, which comes from approved.

Recommended change: Describe proposal confidence as confidence that the proposed correction is warranted, and validator confidence as confidence in the supplied verdict, including a negative verdict. Do not introduce a validator confidence threshold or request inflated scores to pass existing thresholds.

Acceptance: A confident rejection can legitimately have high confidence; approved remains authoritative for the validator verdict. No promise of numerical calibration or new runtime filtering behavior is added.

R26 — P3: Clarify Hardening and Remove Assumed Pipeline Ordering

Category: Shared wording conflicts with the correction task or configuration.

Evidence: The hardening fragment calls the transcript the “source of truth for what was said,” although correcting transcription errors is the task. The grammar prompt claims other corrections occur in a later review stage, although pipeline selection/order is configurable.

Recommended change: Describe the transcript as source material and require only evidence-supported corrections while retaining the data-versus-instructions boundary. Replace the grammar scheduling claim with a statement that homophone and mistranscription corrections are outside this module's scope.

Acceptance: Hardening discourages invention without implying the transcript cannot contain errors. Module guidance remains true regardless of configured stage order or whether another module is enabled. No pipeline ordering restriction is introduced.

Review Verification and Limits

• This is a source/contract audit, not evidence of the frequency of any particular model failure in production.
• go test ./internal/prompts ./internal/framework/validators ./internal/framework/responseschema passed during the audit. Those checks verify existing deterministic behavior, not the effectiveness of proposed language changes.
• No live-provider evaluation was performed and none is made a release gate here.
• Remediation should use focused tests for changed contracts or payloads and retain prompt parsing/rendering checks. Avoid exact-prose snapshots and a new evaluation framework solely to implement wording improvements.
• This document catalogs observations and recommendations only; no prompts, application behavior, or release artifacts were changed by this update.

Resolution Record

Implemented after the maintainer requested addressing all findings. The maintainer explicitly selected allowing protected-name repetition cleanup with validation for R21. The original evidence above describes the audited baseline, not current defects.

Finding	Resolution and evidence
R16	Grammar system and detailed rules consistently state the whole-word a/an exception.
R17	Glossary guidance distinguishes correct protected spelling from a supported mishearing of another recognized term.
R18	All proposal prompts explain literal case-sensitive matching and preserving disambiguating context; glossary explicitly describes replacing every matching occurrence.
R19	Proposal prompts describe multiple source-based corrections per segment and the corrections envelope. Active validators require unchanged correction indices, including negative decisions, and the validations envelope.
R20	Validators receive adjacent in-chunk source segments and mentioned glossary definitions. Optional context is reduced to fit the existing payload budget before required candidate data is considered oversized. Context selection and limits are documented in the validator internals.
R21	Spoken-word protected-term checking permits reduced repetition only with surviving protected identities and existing capitalization checks. The user-approved exception leaves subsequent editorial and meaning validation in place; focused tests cover phrase replacement, literal deletion, last-reference removal, identity substitution, spelling degradation, and unchanged grammar protection.
R22	Spoken-form guidance explicitly evaluates full before/after text for every correction.
R23	Meaning-reversal guidance includes deletion of negation, with contextual judgment rather than a token-based prohibition.
R24	Consumer inspection found only unused internal selections and their tests. Removed both inactive definitions, type selections, metadata entries, and associated cases; updated the seven-prompt inventory. No operator-selectable setting or built-in chain changed.
R25	Proposal confidence describes whether the correction is warranted; validator confidence describes confidence in either verdict. Runtime thresholds are unchanged.
R26	Hardening describes evidence-supported correction of source material, and grammar no longer promises a later pipeline stage.

Validation: the complete go test ./... suite passed, including embedded prompt preparation through PromptKit. New context tests verify supplied neighboring evidence, relevant glossary summaries, chunk boundaries, serialized payloads, and budget effects. The existing oversized-input regression confirms that a large neighbor does not invalidate another otherwise usable candidate. Documentation links and whitespace checks passed. No live model quality claims or release publication are implied.