eric/audita
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 6 Wiki Activity

Record maintainability audit recommendations

Browse Source
This commit is contained in:
Eric Rakestraw
2026-09-25 13:23:12 +00:00
parent 011b021d47
commit 5feec1bf14
1 changed files with 112 additions and 243 deletions
Download Patch File Download Diff File Expand all files Collapse all files

355
docs/roadmap/remediation.md
View File

@@ -1,273 +1,142 @@
# Prompt Audit: Findings and Recommended Changes
# Maintainability Audit: Findings and Recommended Changes
**Reviewed:** 2026-09-25, source commit `b8089f4`.
**Scope:** All four proposal prompts, three active LLM-validator prompts, two
inactive validator definitions, shared hardening text, response schemas, supplied
context, and downstream validation/application behavior.
**Status:** R16–R26 addressed; see the resolution record below. R1–R15 are resolved; their findings and resolution records
remain in Git history through `b8089f4`.
**Reviewed:** 2026-09-25, source commit `011b021`.
**Scope:** Recent partial-decision reconciliation, validation context, prompt
alignment, and request-local reference changes, including related existing code.
**Status:** R27–R30 open. Previous findings and resolution records remain in Git
history through `011b021`.
## Architecture Assessment
PromptKit owns structural response validation and repair. Audita owns domain
validation, decision reconciliation, and application. The recent span-deletion
and partial-decision changes are aligned across these boundaries. This audit found
prompt contradictions, incomplete explanations of application semantics, and
opportunities to improve instructions and supplied context.
The recent changes fit Audita's existing boundaries. Proposal generation,
validation, prompt assets, and outcome reporting already have suitable owners;
this audit does not recommend a new common package or a broad refactor.
R16–R19 and R22–R23/R25–R26 are focused wording improvements suitable for
consideration before v2.0.0. R20–R21 require separate design or policy decisions;
R24 is optional cleanup after checking consumers. Their inclusion here does not
make every item a release gate or authorize an unreviewed behavioral expansion.
The shared editorial validator's broad remit is intentional and needs no new
module-scope enforcement as part of this audit.
The strongest opportunities are shared prompt instructions and request-scoped
validation context preparation. Two smaller improvements concern recovery-reason
classification and partial-result assembly; both predate the latest commits.
These are bounded maintenance recommendations, not new release gates or changes
to correction policy, response acceptance, or reporting semantics.
## R16 — P2: Make the Grammar Article Exception Consistent
## R27 — P2: Consolidate Shared Prompt Instructions
**Category:** Conflicting instructions.
**Category:** Repeated behavioral-contract text; risk of future drift.
**Evidence:** The [grammar prompt](../../internal/prompts/assets/prompts/modules.grammar.proposal.yaml)
allows whole-word `a`/`an` substitution, while its system instruction prohibits word
substitution and a later rule permits only capitalization changes to written words.
A model can follow those prohibitions and omit the expressly permitted correction.
**Evidence:** The four [proposal prompts](../../internal/prompts/assets/prompts/modules.grammar.proposal.yaml)
repeat literal source-span matching, request-local identity, replacement-text,
source-based proposal, and confidence instructions. The three active
[validator prompts](../../internal/prompts/assets/prompts/validators.editorial_review.yaml)
repeat candidate-reference, supplemental-context, and verdict-confidence guidance.
Recent contract changes required coordinated edits across these assets. The
[prompt registry](../../internal/prompts/registry.go) already provides a shared
hardening fragment supplied through the role-bound adapter.
**Recommended change:** State the existing article exception consistently in the
system message and word-preservation rule. Consolidate overlapping prohibitions
where helpful without expanding grammar into spelling, homophone, or content edits.
**Recommended change:** Extract genuinely common passages into embedded fragments
under `internal/prompts/assets/shared`, following the existing fragment-supply
approach. Keep the fragments owned by `internal/prompts` and use the existing
PromptKit rendering path. Keep module-specific examples, replacement-policy
instructions, protected-name exceptions, and semantic criteria in their individual
prompt definitions. Do not merge similar-looking rules with different meanings.
**Acceptance:** The prompt unambiguously permits `a`/`an` correction under the
existing conditions while retaining the module's conservative formatting remit.
**Acceptance:** Common instructions have one maintained source and appear in all
intended rendered prompts. The existing module and validator distinctions remain
intact, including glossary replace-all behavior and the spoken-word protected-name
exception. Focused rendering checks cover fragment inclusion and representative
module-specific guidance without requiring exact-prose snapshots or introducing
a separate templating framework.
## R17 — P2: Clarify the Glossary Mishearing Exception
## R28 — P2: Prepare Validation Context Once Per Request
**Category:** Prompt conflict with intended glossary correction behavior.
**Category:** Repeated computation and overlapping segment lookup.
**Evidence:** The [glossary prompt](../../internal/prompts/assets/prompts/modules.glossary.proposal.yaml)
permits replacing an existing glossary term with another for a plausible mishearing,
then treats present glossary names as protected and prohibits altering them.
The glossary-specific [protected-term validation](../../internal/framework/validators/protected_terms.go)
deliberately allows protected-term swaps, subject to subsequent validation.
**Evidence:** [BuildLLMValidationRequest](../../internal/framework/validators/llm_validators.go)
already indexes transcript segments, but
[validationContext](../../internal/framework/validators/llm_context.go) scans the
transcript again for each candidate. It also calls
[NewProtectedVocabulary](../../internal/framework/validators/protected_terms.go)
for each candidate, rebuilding the same vocabulary and compiling its matching
expression repeatedly. This is duplicated preparation rather than copied code.
**Recommended change:** Distinguish preserving correct glossary spellings from
correcting a contextually supported misidentification. Make the plausible-mishearing
exception explicit while retaining acoustic plausibility and context requirements.
**Recommended change:** Prepare a segment-position lookup and protected vocabulary
once per validation request, then pass that prepared context into the existing
context-selection helper. Keep the implementation within `framework/validators`.
Continue selecting glossary evidence for each candidate individually because its
proposed replacement can mention different terms. A process-wide cache or new
context-management package is unnecessary for this cleanup.
**Acceptance:** Recognizing a source word as a glossary term does not itself imply
that the prompt forbids a supported correction to another term. Unrelated name
substitution and gratuitous spelling/capitalization normalization remain outside
scope; no validator relaxation is implied by this wording change.
**Acceptance:** Candidates in one validation request reuse the prepared lookup and
vocabulary. Context still consists of the available immediate neighbors within the
current chunk and relevant glossary entries in glossary order. Candidate-specific
replacement text still participates in glossary selection. Existing optional-context
trimming, token estimation, and oversized-input handling retain their behavior.
Verify these outcomes with focused behavioral tests rather than assertions about
private helper call counts or unmeasured performance guarantees.
## R18 — P2: Explain Literal Replacement and Context Spans
## R29 — P3: Share Execution-Recovery Classification
**Category:** Incomplete application contract in proposal instructions.
**Category:** Duplicate reporting rule; predates the latest commits.
**Evidence:** [Glossary](../../internal/prompts/assets/prompts/modules.glossary.proposal.yaml)
uses `replace_all` without explaining that every matching occurrence in the target
segment changes. Grammar, homophones, and spoken-word require unique matches, but
all proposal prompts ask for only the exact span needing replacement, which can
discourage including unchanged context. [Preview/application](../../internal/framework/proposals/preview.go)
uses literal, case-sensitive matching.
**Evidence:** [Proposal generation](../../internal/framework/proposal_generation/generate.go)
and [LLM validation](../../internal/framework/validators/llm_validators.go)
independently translate structural-repair and transient-retry counts into
`output_repaired` and `service_recovered` reasons. The conversion is the same in
both locations; the surrounding success and domain-outcome decisions differ.
**Recommended change:** Explain glossary's replace-all behavior explicitly. Allow
unchanged surrounding text in a source span when needed to identify the intended
edit, preserving that context in the replacement. Explain literal matching without
imposing uniqueness on glossary or adding automatic whitespace cleanup.
**Recommended change:** Place a small conversion helper in the existing
`framework/outcomes` package and use it from both callers. Keep the decision to
record recoveries at the call sites, along with domain-specific reasons such as
filtered validator decisions. Do not combine proposal and validation orchestration
or move domain-success classification into this helper.
**Acceptance:** Guidance explains how to target one intended occurrence when a
shorter span repeats, and that a glossary proposal using a repeated span affects
all matches. Existing module replacement policies and eligible whitespace spans
remain unchanged.
**Acceptance:** Both callers use the same execution-count conversion. Existing
reason ordering, successful-operation reporting, and domain-specific additions
remain intact. A partially unavailable validation batch continues to record its
failure and execution counts without acquiring recovery records merely because a
repair or retry occurred. Existing outcome tests should continue to protect these
distinctions.
## R19 — P2: Align Output Instructions with Correction and Decision Identity
## R30 — P3: Centralize Partial Validator Result Assembly
**Category:** Ambiguous response shape and identity guidance.
**Category:** Repeated early-return construction; predates the latest commits.
**Evidence:** Proposal prompts request only changed “segments,” although their
[schema](../../internal/framework/responseschema/assets/correction_set.v1.json)
contains correction proposals and multiple proposals can target one segment.
Validator payloads contain both segment `id` and `correction_index`; the validator
prompts request a decision for each correction index without expressly warning
against substituting segment IDs or renumbering. Both schemas require a top-level
object containing a named list, which prose currently leaves largely to the schema.
**Evidence:** Several early returns in
[LLMBackedValidator.Validate](../../internal/framework/validators/llm_validators.go)
repeat nested slice appends to combine immediate decisions, oversized-input
decisions, and decisions from completed model batches, together with accumulated
operation outcomes. This makes it easier for a future failure path to omit facts
already collected by the validator.
**Recommended change:** Request proposed corrections rather than changed segments,
allow multiple proposals per segment, and explain that their source spans refer to
the supplied text rather than imagined results of other proposals. Tell validators
to copy each supplied `correction_index` exactly, including negative decisions,
without renumbering or substituting segment IDs. Show the top-level
`{"corrections":[...]}` and `{"validations":[...]}` shapes concisely.
**Recommended change:** Use a small local result-building helper to assemble the
partial result consistently. Keep error handling, batching, success-path ordering,
and cardinality enforcement explicit. A general-purpose result-aggregation package
is not needed.
**Acceptance:** Instructions match the supplied schemas and identity model without
requiring one proposal per segment. Validators are still asked for complete decision
sets; tolerant reconciliation remains a fallback, not the preferred output.
**Acceptance:** Cancellation and failure paths retain the same previously collected
decisions and operations. Result ordering and cardinality behavior remain unchanged;
the helper does not enforce complete cardinality on interrupted work. Retain the
existing partial-progress and fatal-failure coverage, adding tests only for a
material gap exposed by the refactor.
## R20 — P2: Evaluate Context Parity for LLM Validation
## Boundaries to Preserve
**Category:** Input-context limitation; separate design work.
**Evidence:** Proposal generation supplies section text and glossary data. The
[validator variable builder](../../internal/framework/validators/llm_prompt_variables.go)
supplies candidate before/after segments, categories, and transcript description,
but not neighboring transcript segments or glossary definitions. Spoken-form and
editorial prompts ask for contextual judgments; evidence supporting the proposal
may be absent during validation. Other batched candidates are not reliable context.
**Recommended change:** Evaluate supplying relevant glossary information and
neighboring transcript context to validators, including its effect on token
estimation and batching. Choose the context scope in a separate design pass.
Avoid asking models to rely on evidence they do not receive; do not silently weaken
validation or imply access to audio.
**Acceptance:** A selected design states which evidence each validator receives
and how it fits existing budgets. Representative examples distinguish cases needing
additional context from cases already decidable from the segment. This is not a
requirement to copy the entire section or glossary into every request.
## R21 — P2: Decide How Protected-Term Repetition Should Be Handled
**Category:** Prompt/validator policy tension; separate decision required.
**Evidence:** The [spoken-word prompt](../../internal/prompts/assets/prompts/modules.spoken_word.proposal.yaml)
permits accidental repetition cleanup while preserving named entities. The
[protected-term check](../../internal/framework/validators/protected_terms.go)
requires each protected identity's occurrence count to be preserved. Thus
`Jesters Jesters arrived` to `Jesters arrived` can be rejected even when repetition
is accidental and the named entity remains.
**Recommended change:** Explicitly decide whether preservation of identity should
permit removing a redundant occurrence. Evaluate a narrow allowance for supported
repetition cleanup, retaining ordinary editorial and meaning checks. If current
count preservation is retained, document that as a deliberate policy choice rather
than silently making the prompt more restrictive to fit existing code.
**Acceptance:** Prompt and validation expectations agree after the policy decision.
Any allowance preserves substantive references and intentional emphasis; any
restriction is explicitly selected. This audit does not authorize a general
relaxation of protected-term validation.
## R22 — P3: Clarify Full-Segment Spoken-Form Assessment
**Category:** Validator wording improvement.
**Evidence:** The [spoken-form validator](../../internal/prompts/assets/prompts/validators.spoken_form_plausibility.yaml)
explicitly directs full before/after comparison for deletions, while its general
criteria discuss source and replacement spans and surrounding segment context.
Full segment text is available for every candidate.
**Recommended change:** Explicitly direct full before/after assessment for all
corrections, retaining the narrow acoustic/mistranscription remit and the distinction
from style or readability review.
**Acceptance:** Guidance consistently uses the available segment evidence without
turning this validator into a general editorial or semantic-equivalence check.
## R23 — P3: Add a Deletion Example to Meaning-Reversal Guidance
**Category:** Optional illustrative improvement.
**Evidence:** The [meaning-reversal prompt](../../internal/prompts/assets/prompts/validators.meaning_reversal_review.yaml)
correctly explains empty replacement and full-segment comparison, but its concrete
examples focus on word substitutions such as visible/invisible and up/down.
**Recommended change:** Add a concise example where deleting a negation reverses
meaning. Preserve context-sensitive judgment and the validator's narrow scope.
**Acceptance:** The example illustrates why a deletion can reverse meaning without
prohibiting empty replacements or making every word deletion a reversal. It is not
a new literal-token rule or a guarantee of model accuracy.
## R24 — P3: Review the Two Inactive Validator Definitions
**Category:** Optional maintenance cleanup.
**Evidence:** Registered `grammar_review`
and `spoken_word_review`
definitions largely duplicate editorial review and have already diverged from its
updated deletion guidance. Neither is used by the [built-in chains](../../internal/validators/chains.go).
**Recommended change:** Check consumers and retained API uses, then retire the
unused definitions and associated registrations if nothing needs them. If retained,
document their purpose and keep relevant guidance consistent. Do not activate them
as extra validation stages merely because they exist.
**Acceptance:** Each retained definition has a clear purpose; removal, if selected,
updates actual consumers, tests, and prompt inventory, with any compatibility
implications considered explicitly.
## R25 — P3: Define What Confidence Means
**Category:** Underspecified output semantics.
**Evidence:** Proposal and validator prompts specify the confidence range but not
its interpretation. Proposal confidence drives threshold filtering. Validator
confidence is range-checked but does not determine the verdict, which comes from
`approved`.
**Recommended change:** Describe proposal confidence as confidence that the proposed
correction is warranted, and validator confidence as confidence in the supplied
verdict, including a negative verdict. Do not introduce a validator confidence
threshold or request inflated scores to pass existing thresholds.
**Acceptance:** A confident rejection can legitimately have high confidence;
`approved` remains authoritative for the validator verdict. No promise of numerical
calibration or new runtime filtering behavior is added.
## R26 — P3: Clarify Hardening and Remove Assumed Pipeline Ordering
**Category:** Shared wording conflicts with the correction task or configuration.
**Evidence:** The [hardening fragment](../../internal/prompts/assets/shared/prompt_hardening.md)
calls the transcript the “source of truth for what was said,” although correcting
transcription errors is the task. The grammar prompt claims other corrections occur
in a later review stage, although pipeline selection/order is configurable.
**Recommended change:** Describe the transcript as source material and require
only evidence-supported corrections while retaining the data-versus-instructions
boundary. Replace the grammar scheduling claim with a statement that homophone and
mistranscription corrections are outside this module's scope.
**Acceptance:** Hardening discourages invention without implying the transcript
cannot contain errors. Module guidance remains true regardless of configured stage
order or whether another module is enabled. No pipeline ordering restriction is
introduced.
- Keep proposal and validator reference mapping separate. Proposals can target the
same segment repeatedly; duplicate validator decisions make that candidate
unavailable. Similar indexing arithmetic alone does not justify a generic
reference-management abstraction.
- Keep distinct protected-term policy checks separate. They already share the
vocabulary implementation and intentionally apply different policies.
- Preserve exact source-span matching, request-local references, batching, and
partial-success behavior. Consolidation is not a reason to strengthen contracts
or broaden the accepted correction scope.
## Review Verification and Limits
- This is a source/contract audit, not evidence of the frequency of any particular
model failure in production.
- `go test ./internal/prompts ./internal/framework/validators ./internal/framework/responseschema`
passed during the audit. Those checks verify existing deterministic behavior,
not the effectiveness of proposed language changes.
- No live-provider evaluation was performed and none is made a release gate here.
- Remediation should use focused tests for changed contracts or payloads and retain
prompt parsing/rendering checks. Avoid exact-prose snapshots and a new evaluation
framework solely to implement wording improvements.
- This document catalogs observations and recommendations only; no prompts,
application behavior, or release artifacts were changed by this update.
## Resolution Record
Implemented after the maintainer requested addressing all findings. The maintainer
explicitly selected allowing protected-name repetition cleanup with validation for
R21. The original evidence above describes the audited baseline, not current defects.
| Finding | Resolution and evidence |
| --- | --- |
| R16 | Grammar system and detailed rules consistently state the whole-word a/an exception. |
| R17 | Glossary guidance distinguishes correct protected spelling from a supported mishearing of another recognized term. |
| R18 | All proposal prompts explain literal case-sensitive matching and preserving disambiguating context; glossary explicitly describes replacing every matching occurrence. |
| R19 | Proposal prompts describe multiple source-based corrections per segment and the corrections envelope. Active validators require unchanged correction indices, including negative decisions, and the validations envelope. |
| R20 | Validators receive adjacent in-chunk source segments and mentioned glossary definitions. Optional context is reduced to fit the existing payload budget before required candidate data is considered oversized. Context selection and limits are documented in the validator internals. |
| R21 | Spoken-word protected-term checking permits reduced repetition only with surviving protected identities and existing capitalization checks. The user-approved exception leaves subsequent editorial and meaning validation in place; focused tests cover phrase replacement, literal deletion, last-reference removal, identity substitution, spelling degradation, and unchanged grammar protection. |
| R22 | Spoken-form guidance explicitly evaluates full before/after text for every correction. |
| R23 | Meaning-reversal guidance includes deletion of negation, with contextual judgment rather than a token-based prohibition. |
| R24 | Consumer inspection found only unused internal selections and their tests. Removed both inactive definitions, type selections, metadata entries, and associated cases; updated the seven-prompt inventory. No operator-selectable setting or built-in chain changed. |
| R25 | Proposal confidence describes whether the correction is warranted; validator confidence describes confidence in either verdict. Runtime thresholds are unchanged. |
| R26 | Hardening describes evidence-supported correction of source material, and grammar no longer promises a later pipeline stage. |
Validation: the complete `go test ./...` suite passed, including embedded prompt
preparation through PromptKit. New context tests verify supplied neighboring evidence,
relevant glossary summaries, chunk boundaries, serialized payloads, and budget effects.
The existing oversized-input regression confirms that a large neighbor does not
invalidate another otherwise usable candidate. Documentation links and whitespace
checks passed. No live model quality claims or release publication are implied.
- This is a source-level maintainability audit, not a finding of demonstrated
production failures or a measured performance assessment.
- No application tests were rerun for the read-only audit. This roadmap update
requires documentation link and whitespace checks only.
- Future remediation should run focused behavioral tests and the broader suite
appropriate to the actual change, following the testing policy. No live-provider
evaluation or new evaluation framework is required by these recommendations.
- This document records recommendations only; no implementation, prompt, or
release changes are included in this update.