audita/docs/roadmap/remediation.md

# Prompt Audit: Findings and Recommended Changes

**Reviewed:** 2026-09-25, source commit `b8089f4`.
**Scope:** All four proposal prompts, three active LLM-validator prompts, two
inactive validator definitions, shared hardening text, response schemas, supplied
context, and downstream validation/application behavior.
**Status:** R16–R26 open. R1–R15 are resolved; their findings and resolution records
remain in Git history through `b8089f4`.

## Architecture Assessment

PromptKit owns structural response validation and repair. Audita owns domain
validation, decision reconciliation, and application. The recent span-deletion
and partial-decision changes are aligned across these boundaries. This audit found
prompt contradictions, incomplete explanations of application semantics, and
opportunities to improve instructions and supplied context.

R16–R19 and R22–R23/R25–R26 are focused wording improvements suitable for
consideration before v2.0.0. R20–R21 require separate design or policy decisions;
R24 is optional cleanup after checking consumers. Their inclusion here does not
make every item a release gate or authorize an unreviewed behavioral expansion.
The shared editorial validator's broad remit is intentional and needs no new
module-scope enforcement as part of this audit.

## R16 — P2: Make the Grammar Article Exception Consistent

**Category:** Conflicting instructions.

**Evidence:** The [grammar prompt](../../internal/prompts/assets/prompts/modules.grammar.proposal.yaml)
allows whole-word `a`/`an` substitution, while its system instruction prohibits word
substitution and a later rule permits only capitalization changes to written words.
A model can follow those prohibitions and omit the expressly permitted correction.

**Recommended change:** State the existing article exception consistently in the
system message and word-preservation rule. Consolidate overlapping prohibitions
where helpful without expanding grammar into spelling, homophone, or content edits.

**Acceptance:** The prompt unambiguously permits `a`/`an` correction under the
existing conditions while retaining the module's conservative formatting remit.

## R17 — P2: Clarify the Glossary Mishearing Exception

**Category:** Prompt conflict with intended glossary correction behavior.

**Evidence:** The [glossary prompt](../../internal/prompts/assets/prompts/modules.glossary.proposal.yaml)
permits replacing an existing glossary term with another for a plausible mishearing,
then treats present glossary names as protected and prohibits altering them.
The glossary-specific [protected-term validation](../../internal/framework/validators/protected_terms.go)
deliberately allows protected-term swaps, subject to subsequent validation.

**Recommended change:** Distinguish preserving correct glossary spellings from
correcting a contextually supported misidentification. Make the plausible-mishearing
exception explicit while retaining acoustic plausibility and context requirements.

**Acceptance:** Recognizing a source word as a glossary term does not itself imply
that the prompt forbids a supported correction to another term. Unrelated name
substitution and gratuitous spelling/capitalization normalization remain outside
scope; no validator relaxation is implied by this wording change.

## R18 — P2: Explain Literal Replacement and Context Spans

**Category:** Incomplete application contract in proposal instructions.

**Evidence:** [Glossary](../../internal/prompts/assets/prompts/modules.glossary.proposal.yaml)
uses `replace_all` without explaining that every matching occurrence in the target
segment changes. Grammar, homophones, and spoken-word require unique matches, but
all proposal prompts ask for only the exact span needing replacement, which can
discourage including unchanged context. [Preview/application](../../internal/framework/proposals/preview.go)
uses literal, case-sensitive matching.

**Recommended change:** Explain glossary's replace-all behavior explicitly. Allow
unchanged surrounding text in a source span when needed to identify the intended
edit, preserving that context in the replacement. Explain literal matching without
imposing uniqueness on glossary or adding automatic whitespace cleanup.

**Acceptance:** Guidance explains how to target one intended occurrence when a
shorter span repeats, and that a glossary proposal using a repeated span affects
all matches. Existing module replacement policies and eligible whitespace spans
remain unchanged.

## R19 — P2: Align Output Instructions with Correction and Decision Identity

**Category:** Ambiguous response shape and identity guidance.

**Evidence:** Proposal prompts request only changed “segments,” although their
[schema](../../internal/framework/responseschema/assets/correction_set.v1.json)
contains correction proposals and multiple proposals can target one segment.
Validator payloads contain both segment `id` and `correction_index`; the validator
prompts request a decision for each correction index without expressly warning
against substituting segment IDs or renumbering. Both schemas require a top-level
object containing a named list, which prose currently leaves largely to the schema.

**Recommended change:** Request proposed corrections rather than changed segments,
allow multiple proposals per segment, and explain that their source spans refer to
the supplied text rather than imagined results of other proposals. Tell validators
to copy each supplied `correction_index` exactly, including negative decisions,
without renumbering or substituting segment IDs. Show the top-level
`{"corrections":[...]}` and `{"validations":[...]}` shapes concisely.

**Acceptance:** Instructions match the supplied schemas and identity model without
requiring one proposal per segment. Validators are still asked for complete decision
sets; tolerant reconciliation remains a fallback, not the preferred output.

## R20 — P2: Evaluate Context Parity for LLM Validation

**Category:** Input-context limitation; separate design work.

**Evidence:** Proposal generation supplies section text and glossary data. The
[validator variable builder](../../internal/framework/validators/llm_prompt_variables.go)
supplies candidate before/after segments, categories, and transcript description,
but not neighboring transcript segments or glossary definitions. Spoken-form and
editorial prompts ask for contextual judgments; evidence supporting the proposal
may be absent during validation. Other batched candidates are not reliable context.

**Recommended change:** Evaluate supplying relevant glossary information and
neighboring transcript context to validators, including its effect on token
estimation and batching. Choose the context scope in a separate design pass.
Avoid asking models to rely on evidence they do not receive; do not silently weaken
validation or imply access to audio.

**Acceptance:** A selected design states which evidence each validator receives
and how it fits existing budgets. Representative examples distinguish cases needing
additional context from cases already decidable from the segment. This is not a
requirement to copy the entire section or glossary into every request.

## R21 — P2: Decide How Protected-Term Repetition Should Be Handled

**Category:** Prompt/validator policy tension; separate decision required.

**Evidence:** The [spoken-word prompt](../../internal/prompts/assets/prompts/modules.spoken_word.proposal.yaml)
permits accidental repetition cleanup while preserving named entities. The
[protected-term check](../../internal/framework/validators/protected_terms.go)
requires each protected identity's occurrence count to be preserved. Thus
`Jesters Jesters arrived` to `Jesters arrived` can be rejected even when repetition
is accidental and the named entity remains.

**Recommended change:** Explicitly decide whether preservation of identity should
permit removing a redundant occurrence. Evaluate a narrow allowance for supported
repetition cleanup, retaining ordinary editorial and meaning checks. If current
count preservation is retained, document that as a deliberate policy choice rather
than silently making the prompt more restrictive to fit existing code.

**Acceptance:** Prompt and validation expectations agree after the policy decision.
Any allowance preserves substantive references and intentional emphasis; any
restriction is explicitly selected. This audit does not authorize a general
relaxation of protected-term validation.

## R22 — P3: Clarify Full-Segment Spoken-Form Assessment

**Category:** Validator wording improvement.

**Evidence:** The [spoken-form validator](../../internal/prompts/assets/prompts/validators.spoken_form_plausibility.yaml)
explicitly directs full before/after comparison for deletions, while its general
criteria discuss source and replacement spans and surrounding segment context.
Full segment text is available for every candidate.

**Recommended change:** Explicitly direct full before/after assessment for all
corrections, retaining the narrow acoustic/mistranscription remit and the distinction
from style or readability review.

**Acceptance:** Guidance consistently uses the available segment evidence without
turning this validator into a general editorial or semantic-equivalence check.

## R23 — P3: Add a Deletion Example to Meaning-Reversal Guidance

**Category:** Optional illustrative improvement.

**Evidence:** The [meaning-reversal prompt](../../internal/prompts/assets/prompts/validators.meaning_reversal_review.yaml)
correctly explains empty replacement and full-segment comparison, but its concrete
examples focus on word substitutions such as visible/invisible and up/down.

**Recommended change:** Add a concise example where deleting a negation reverses
meaning. Preserve context-sensitive judgment and the validator's narrow scope.

**Acceptance:** The example illustrates why a deletion can reverse meaning without
prohibiting empty replacements or making every word deletion a reversal. It is not
a new literal-token rule or a guarantee of model accuracy.

## R24 — P3: Review the Two Inactive Validator Definitions

**Category:** Optional maintenance cleanup.

**Evidence:** Registered [grammar_review](../../internal/prompts/assets/prompts/validators.grammar_review.yaml)
and [spoken_word_review](../../internal/prompts/assets/prompts/validators.spoken_word_review.yaml)
definitions largely duplicate editorial review and have already diverged from its
updated deletion guidance. Neither is used by the [built-in chains](../../internal/validators/chains.go).

**Recommended change:** Check consumers and retained API uses, then retire the
unused definitions and associated registrations if nothing needs them. If retained,
document their purpose and keep relevant guidance consistent. Do not activate them
as extra validation stages merely because they exist.

**Acceptance:** Each retained definition has a clear purpose; removal, if selected,
updates actual consumers, tests, and prompt inventory, with any compatibility
implications considered explicitly.

## R25 — P3: Define What Confidence Means

**Category:** Underspecified output semantics.

**Evidence:** Proposal and validator prompts specify the confidence range but not
its interpretation. Proposal confidence drives threshold filtering. Validator
confidence is range-checked but does not determine the verdict, which comes from
`approved`.

**Recommended change:** Describe proposal confidence as confidence that the proposed
correction is warranted, and validator confidence as confidence in the supplied
verdict, including a negative verdict. Do not introduce a validator confidence
threshold or request inflated scores to pass existing thresholds.

**Acceptance:** A confident rejection can legitimately have high confidence;
`approved` remains authoritative for the validator verdict. No promise of numerical
calibration or new runtime filtering behavior is added.

## R26 — P3: Clarify Hardening and Remove Assumed Pipeline Ordering

**Category:** Shared wording conflicts with the correction task or configuration.

**Evidence:** The [hardening fragment](../../internal/prompts/assets/shared/prompt_hardening.md)
calls the transcript the “source of truth for what was said,” although correcting
transcription errors is the task. The grammar prompt claims other corrections occur
in a later review stage, although pipeline selection/order is configurable.

**Recommended change:** Describe the transcript as source material and require
only evidence-supported corrections while retaining the data-versus-instructions
boundary. Replace the grammar scheduling claim with a statement that homophone and
mistranscription corrections are outside this module's scope.

**Acceptance:** Hardening discourages invention without implying the transcript
cannot contain errors. Module guidance remains true regardless of configured stage
order or whether another module is enabled. No pipeline ordering restriction is
introduced.

## Review Verification and Limits

- This is a source/contract audit, not evidence of the frequency of any particular
  model failure in production.
- `go test ./internal/prompts ./internal/framework/validators ./internal/framework/responseschema`
  passed during the audit. Those checks verify existing deterministic behavior,
  not the effectiveness of proposed language changes.
- No live-provider evaluation was performed and none is made a release gate here.
- Remediation should use focused tests for changed contracts or payloads and retain
  prompt parsing/rendering checks. Avoid exact-prose snapshots and a new evaluation
  framework solely to implement wording improvements.
- This document catalogs observations and recommendations only; no prompts,
  application behavior, or release artifacts were changed by this update.