Record maintainability audit recommendations

This commit is contained in:
2026-09-25 13:23:12 +00:00
parent 011b021d47
commit 5feec1bf14

View File

@@ -1,273 +1,142 @@
# Prompt Audit: Findings and Recommended Changes # Maintainability Audit: Findings and Recommended Changes
**Reviewed:** 2026-09-25, source commit `b8089f4`. **Reviewed:** 2026-09-25, source commit `011b021`.
**Scope:** All four proposal prompts, three active LLM-validator prompts, two **Scope:** Recent partial-decision reconciliation, validation context, prompt
inactive validator definitions, shared hardening text, response schemas, supplied alignment, and request-local reference changes, including related existing code.
context, and downstream validation/application behavior. **Status:** R27–R30 open. Previous findings and resolution records remain in Git
**Status:** R16–R26 addressed; see the resolution record below. R1–R15 are resolved; their findings and resolution records history through `011b021`.
remain in Git history through `b8089f4`.
## Architecture Assessment ## Architecture Assessment
PromptKit owns structural response validation and repair. Audita owns domain The recent changes fit Audita's existing boundaries. Proposal generation,
validation, decision reconciliation, and application. The recent span-deletion validation, prompt assets, and outcome reporting already have suitable owners;
and partial-decision changes are aligned across these boundaries. This audit found this audit does not recommend a new common package or a broad refactor.
prompt contradictions, incomplete explanations of application semantics, and
opportunities to improve instructions and supplied context.
R16–R19 and R22–R23/R25–R26 are focused wording improvements suitable for The strongest opportunities are shared prompt instructions and request-scoped
consideration before v2.0.0. R20–R21 require separate design or policy decisions; validation context preparation. Two smaller improvements concern recovery-reason
R24 is optional cleanup after checking consumers. Their inclusion here does not classification and partial-result assembly; both predate the latest commits.
make every item a release gate or authorize an unreviewed behavioral expansion. These are bounded maintenance recommendations, not new release gates or changes
The shared editorial validator's broad remit is intentional and needs no new to correction policy, response acceptance, or reporting semantics.
module-scope enforcement as part of this audit.
## R16 — P2: Make the Grammar Article Exception Consistent ## R27 — P2: Consolidate Shared Prompt Instructions
**Category:** Conflicting instructions. **Category:** Repeated behavioral-contract text; risk of future drift.
**Evidence:** The [grammar prompt](../../internal/prompts/assets/prompts/modules.grammar.proposal.yaml) **Evidence:** The four [proposal prompts](../../internal/prompts/assets/prompts/modules.grammar.proposal.yaml)
allows whole-word `a`/`an` substitution, while its system instruction prohibits word repeat literal source-span matching, request-local identity, replacement-text,
substitution and a later rule permits only capitalization changes to written words. source-based proposal, and confidence instructions. The three active
A model can follow those prohibitions and omit the expressly permitted correction. [validator prompts](../../internal/prompts/assets/prompts/validators.editorial_review.yaml)
repeat candidate-reference, supplemental-context, and verdict-confidence guidance.
Recent contract changes required coordinated edits across these assets. The
[prompt registry](../../internal/prompts/registry.go) already provides a shared
hardening fragment supplied through the role-bound adapter.
**Recommended change:** State the existing article exception consistently in the **Recommended change:** Extract genuinely common passages into embedded fragments
system message and word-preservation rule. Consolidate overlapping prohibitions under `internal/prompts/assets/shared`, following the existing fragment-supply
where helpful without expanding grammar into spelling, homophone, or content edits. approach. Keep the fragments owned by `internal/prompts` and use the existing
PromptKit rendering path. Keep module-specific examples, replacement-policy
instructions, protected-name exceptions, and semantic criteria in their individual
prompt definitions. Do not merge similar-looking rules with different meanings.
**Acceptance:** The prompt unambiguously permits `a`/`an` correction under the **Acceptance:** Common instructions have one maintained source and appear in all
existing conditions while retaining the module's conservative formatting remit. intended rendered prompts. The existing module and validator distinctions remain
intact, including glossary replace-all behavior and the spoken-word protected-name
exception. Focused rendering checks cover fragment inclusion and representative
module-specific guidance without requiring exact-prose snapshots or introducing
a separate templating framework.
## R17 — P2: Clarify the Glossary Mishearing Exception ## R28 — P2: Prepare Validation Context Once Per Request
**Category:** Prompt conflict with intended glossary correction behavior. **Category:** Repeated computation and overlapping segment lookup.
**Evidence:** The [glossary prompt](../../internal/prompts/assets/prompts/modules.glossary.proposal.yaml) **Evidence:** [BuildLLMValidationRequest](../../internal/framework/validators/llm_validators.go)
permits replacing an existing glossary term with another for a plausible mishearing, already indexes transcript segments, but
then treats present glossary names as protected and prohibits altering them. [validationContext](../../internal/framework/validators/llm_context.go) scans the
The glossary-specific [protected-term validation](../../internal/framework/validators/protected_terms.go) transcript again for each candidate. It also calls
deliberately allows protected-term swaps, subject to subsequent validation. [NewProtectedVocabulary](../../internal/framework/validators/protected_terms.go)
for each candidate, rebuilding the same vocabulary and compiling its matching
expression repeatedly. This is duplicated preparation rather than copied code.
**Recommended change:** Distinguish preserving correct glossary spellings from **Recommended change:** Prepare a segment-position lookup and protected vocabulary
correcting a contextually supported misidentification. Make the plausible-mishearing once per validation request, then pass that prepared context into the existing
exception explicit while retaining acoustic plausibility and context requirements. context-selection helper. Keep the implementation within `framework/validators`.
Continue selecting glossary evidence for each candidate individually because its
proposed replacement can mention different terms. A process-wide cache or new
context-management package is unnecessary for this cleanup.
**Acceptance:** Recognizing a source word as a glossary term does not itself imply **Acceptance:** Candidates in one validation request reuse the prepared lookup and
that the prompt forbids a supported correction to another term. Unrelated name vocabulary. Context still consists of the available immediate neighbors within the
substitution and gratuitous spelling/capitalization normalization remain outside current chunk and relevant glossary entries in glossary order. Candidate-specific
scope; no validator relaxation is implied by this wording change. replacement text still participates in glossary selection. Existing optional-context
trimming, token estimation, and oversized-input handling retain their behavior.
Verify these outcomes with focused behavioral tests rather than assertions about
private helper call counts or unmeasured performance guarantees.
## R18 — P2: Explain Literal Replacement and Context Spans ## R29 — P3: Share Execution-Recovery Classification
**Category:** Incomplete application contract in proposal instructions. **Category:** Duplicate reporting rule; predates the latest commits.
**Evidence:** [Glossary](../../internal/prompts/assets/prompts/modules.glossary.proposal.yaml) **Evidence:** [Proposal generation](../../internal/framework/proposal_generation/generate.go)
uses `replace_all` without explaining that every matching occurrence in the target and [LLM validation](../../internal/framework/validators/llm_validators.go)
segment changes. Grammar, homophones, and spoken-word require unique matches, but independently translate structural-repair and transient-retry counts into
all proposal prompts ask for only the exact span needing replacement, which can `output_repaired` and `service_recovered` reasons. The conversion is the same in
discourage including unchanged context. [Preview/application](../../internal/framework/proposals/preview.go) both locations; the surrounding success and domain-outcome decisions differ.
uses literal, case-sensitive matching.
**Recommended change:** Explain glossary's replace-all behavior explicitly. Allow **Recommended change:** Place a small conversion helper in the existing
unchanged surrounding text in a source span when needed to identify the intended `framework/outcomes` package and use it from both callers. Keep the decision to
edit, preserving that context in the replacement. Explain literal matching without record recoveries at the call sites, along with domain-specific reasons such as
imposing uniqueness on glossary or adding automatic whitespace cleanup. filtered validator decisions. Do not combine proposal and validation orchestration
or move domain-success classification into this helper.
**Acceptance:** Guidance explains how to target one intended occurrence when a **Acceptance:** Both callers use the same execution-count conversion. Existing
shorter span repeats, and that a glossary proposal using a repeated span affects reason ordering, successful-operation reporting, and domain-specific additions
all matches. Existing module replacement policies and eligible whitespace spans remain intact. A partially unavailable validation batch continues to record its
remain unchanged. failure and execution counts without acquiring recovery records merely because a
repair or retry occurred. Existing outcome tests should continue to protect these
distinctions.
## R19 — P2: Align Output Instructions with Correction and Decision Identity ## R30 — P3: Centralize Partial Validator Result Assembly
**Category:** Ambiguous response shape and identity guidance. **Category:** Repeated early-return construction; predates the latest commits.
**Evidence:** Proposal prompts request only changed “segments,” although their **Evidence:** Several early returns in
[schema](../../internal/framework/responseschema/assets/correction_set.v1.json) [LLMBackedValidator.Validate](../../internal/framework/validators/llm_validators.go)
contains correction proposals and multiple proposals can target one segment. repeat nested slice appends to combine immediate decisions, oversized-input
Validator payloads contain both segment `id` and `correction_index`; the validator decisions, and decisions from completed model batches, together with accumulated
prompts request a decision for each correction index without expressly warning operation outcomes. This makes it easier for a future failure path to omit facts
against substituting segment IDs or renumbering. Both schemas require a top-level already collected by the validator.
object containing a named list, which prose currently leaves largely to the schema.
**Recommended change:** Request proposed corrections rather than changed segments, **Recommended change:** Use a small local result-building helper to assemble the
allow multiple proposals per segment, and explain that their source spans refer to partial result consistently. Keep error handling, batching, success-path ordering,
the supplied text rather than imagined results of other proposals. Tell validators and cardinality enforcement explicit. A general-purpose result-aggregation package
to copy each supplied `correction_index` exactly, including negative decisions, is not needed.
without renumbering or substituting segment IDs. Show the top-level
`{"corrections":[...]}` and `{"validations":[...]}` shapes concisely.
**Acceptance:** Instructions match the supplied schemas and identity model without **Acceptance:** Cancellation and failure paths retain the same previously collected
requiring one proposal per segment. Validators are still asked for complete decision decisions and operations. Result ordering and cardinality behavior remain unchanged;
sets; tolerant reconciliation remains a fallback, not the preferred output. the helper does not enforce complete cardinality on interrupted work. Retain the
existing partial-progress and fatal-failure coverage, adding tests only for a
material gap exposed by the refactor.
## R20 — P2: Evaluate Context Parity for LLM Validation ## Boundaries to Preserve
**Category:** Input-context limitation; separate design work. - Keep proposal and validator reference mapping separate. Proposals can target the
same segment repeatedly; duplicate validator decisions make that candidate
**Evidence:** Proposal generation supplies section text and glossary data. The unavailable. Similar indexing arithmetic alone does not justify a generic
[validator variable builder](../../internal/framework/validators/llm_prompt_variables.go) reference-management abstraction.
supplies candidate before/after segments, categories, and transcript description, - Keep distinct protected-term policy checks separate. They already share the
but not neighboring transcript segments or glossary definitions. Spoken-form and vocabulary implementation and intentionally apply different policies.
editorial prompts ask for contextual judgments; evidence supporting the proposal - Preserve exact source-span matching, request-local references, batching, and
may be absent during validation. Other batched candidates are not reliable context. partial-success behavior. Consolidation is not a reason to strengthen contracts
or broaden the accepted correction scope.
**Recommended change:** Evaluate supplying relevant glossary information and
neighboring transcript context to validators, including its effect on token
estimation and batching. Choose the context scope in a separate design pass.
Avoid asking models to rely on evidence they do not receive; do not silently weaken
validation or imply access to audio.
**Acceptance:** A selected design states which evidence each validator receives
and how it fits existing budgets. Representative examples distinguish cases needing
additional context from cases already decidable from the segment. This is not a
requirement to copy the entire section or glossary into every request.
## R21 — P2: Decide How Protected-Term Repetition Should Be Handled
**Category:** Prompt/validator policy tension; separate decision required.
**Evidence:** The [spoken-word prompt](../../internal/prompts/assets/prompts/modules.spoken_word.proposal.yaml)
permits accidental repetition cleanup while preserving named entities. The
[protected-term check](../../internal/framework/validators/protected_terms.go)
requires each protected identity's occurrence count to be preserved. Thus
`Jesters Jesters arrived` to `Jesters arrived` can be rejected even when repetition
is accidental and the named entity remains.
**Recommended change:** Explicitly decide whether preservation of identity should
permit removing a redundant occurrence. Evaluate a narrow allowance for supported
repetition cleanup, retaining ordinary editorial and meaning checks. If current
count preservation is retained, document that as a deliberate policy choice rather
than silently making the prompt more restrictive to fit existing code.
**Acceptance:** Prompt and validation expectations agree after the policy decision.
Any allowance preserves substantive references and intentional emphasis; any
restriction is explicitly selected. This audit does not authorize a general
relaxation of protected-term validation.
## R22 — P3: Clarify Full-Segment Spoken-Form Assessment
**Category:** Validator wording improvement.
**Evidence:** The [spoken-form validator](../../internal/prompts/assets/prompts/validators.spoken_form_plausibility.yaml)
explicitly directs full before/after comparison for deletions, while its general
criteria discuss source and replacement spans and surrounding segment context.
Full segment text is available for every candidate.
**Recommended change:** Explicitly direct full before/after assessment for all
corrections, retaining the narrow acoustic/mistranscription remit and the distinction
from style or readability review.
**Acceptance:** Guidance consistently uses the available segment evidence without
turning this validator into a general editorial or semantic-equivalence check.
## R23 — P3: Add a Deletion Example to Meaning-Reversal Guidance
**Category:** Optional illustrative improvement.
**Evidence:** The [meaning-reversal prompt](../../internal/prompts/assets/prompts/validators.meaning_reversal_review.yaml)
correctly explains empty replacement and full-segment comparison, but its concrete
examples focus on word substitutions such as visible/invisible and up/down.
**Recommended change:** Add a concise example where deleting a negation reverses
meaning. Preserve context-sensitive judgment and the validator's narrow scope.
**Acceptance:** The example illustrates why a deletion can reverse meaning without
prohibiting empty replacements or making every word deletion a reversal. It is not
a new literal-token rule or a guarantee of model accuracy.
## R24 — P3: Review the Two Inactive Validator Definitions
**Category:** Optional maintenance cleanup.
**Evidence:** Registered `grammar_review`
and `spoken_word_review`
definitions largely duplicate editorial review and have already diverged from its
updated deletion guidance. Neither is used by the [built-in chains](../../internal/validators/chains.go).
**Recommended change:** Check consumers and retained API uses, then retire the
unused definitions and associated registrations if nothing needs them. If retained,
document their purpose and keep relevant guidance consistent. Do not activate them
as extra validation stages merely because they exist.
**Acceptance:** Each retained definition has a clear purpose; removal, if selected,
updates actual consumers, tests, and prompt inventory, with any compatibility
implications considered explicitly.
## R25 — P3: Define What Confidence Means
**Category:** Underspecified output semantics.
**Evidence:** Proposal and validator prompts specify the confidence range but not
its interpretation. Proposal confidence drives threshold filtering. Validator
confidence is range-checked but does not determine the verdict, which comes from
`approved`.
**Recommended change:** Describe proposal confidence as confidence that the proposed
correction is warranted, and validator confidence as confidence in the supplied
verdict, including a negative verdict. Do not introduce a validator confidence
threshold or request inflated scores to pass existing thresholds.
**Acceptance:** A confident rejection can legitimately have high confidence;
`approved` remains authoritative for the validator verdict. No promise of numerical
calibration or new runtime filtering behavior is added.
## R26 — P3: Clarify Hardening and Remove Assumed Pipeline Ordering
**Category:** Shared wording conflicts with the correction task or configuration.
**Evidence:** The [hardening fragment](../../internal/prompts/assets/shared/prompt_hardening.md)
calls the transcript the “source of truth for what was said,” although correcting
transcription errors is the task. The grammar prompt claims other corrections occur
in a later review stage, although pipeline selection/order is configurable.
**Recommended change:** Describe the transcript as source material and require
only evidence-supported corrections while retaining the data-versus-instructions
boundary. Replace the grammar scheduling claim with a statement that homophone and
mistranscription corrections are outside this module's scope.
**Acceptance:** Hardening discourages invention without implying the transcript
cannot contain errors. Module guidance remains true regardless of configured stage
order or whether another module is enabled. No pipeline ordering restriction is
introduced.
## Review Verification and Limits ## Review Verification and Limits
- This is a source/contract audit, not evidence of the frequency of any particular - This is a source-level maintainability audit, not a finding of demonstrated
model failure in production. production failures or a measured performance assessment.
- `go test ./internal/prompts ./internal/framework/validators ./internal/framework/responseschema` - No application tests were rerun for the read-only audit. This roadmap update
passed during the audit. Those checks verify existing deterministic behavior, requires documentation link and whitespace checks only.
not the effectiveness of proposed language changes. - Future remediation should run focused behavioral tests and the broader suite
- No live-provider evaluation was performed and none is made a release gate here. appropriate to the actual change, following the testing policy. No live-provider
- Remediation should use focused tests for changed contracts or payloads and retain evaluation or new evaluation framework is required by these recommendations.
prompt parsing/rendering checks. Avoid exact-prose snapshots and a new evaluation - This document records recommendations only; no implementation, prompt, or
framework solely to implement wording improvements. release changes are included in this update.
- This document catalogs observations and recommendations only; no prompts,
application behavior, or release artifacts were changed by this update.
## Resolution Record
Implemented after the maintainer requested addressing all findings. The maintainer
explicitly selected allowing protected-name repetition cleanup with validation for
R21. The original evidence above describes the audited baseline, not current defects.
| Finding | Resolution and evidence |
| --- | --- |
| R16 | Grammar system and detailed rules consistently state the whole-word a/an exception. |
| R17 | Glossary guidance distinguishes correct protected spelling from a supported mishearing of another recognized term. |
| R18 | All proposal prompts explain literal case-sensitive matching and preserving disambiguating context; glossary explicitly describes replacing every matching occurrence. |
| R19 | Proposal prompts describe multiple source-based corrections per segment and the corrections envelope. Active validators require unchanged correction indices, including negative decisions, and the validations envelope. |
| R20 | Validators receive adjacent in-chunk source segments and mentioned glossary definitions. Optional context is reduced to fit the existing payload budget before required candidate data is considered oversized. Context selection and limits are documented in the validator internals. |
| R21 | Spoken-word protected-term checking permits reduced repetition only with surviving protected identities and existing capitalization checks. The user-approved exception leaves subsequent editorial and meaning validation in place; focused tests cover phrase replacement, literal deletion, last-reference removal, identity substitution, spelling degradation, and unchanged grammar protection. |
| R22 | Spoken-form guidance explicitly evaluates full before/after text for every correction. |
| R23 | Meaning-reversal guidance includes deletion of negation, with contextual judgment rather than a token-based prohibition. |
| R24 | Consumer inspection found only unused internal selections and their tests. Removed both inactive definitions, type selections, metadata entries, and associated cases; updated the seven-prompt inventory. No operator-selectable setting or built-in chain changed. |
| R25 | Proposal confidence describes whether the correction is warranted; validator confidence describes confidence in either verdict. Runtime thresholds are unchanged. |
| R26 | Hardening describes evidence-supported correction of source material, and grammar no longer promises a later pipeline stage. |
Validation: the complete `go test ./...` suite passed, including embedded prompt
preparation through PromptKit. New context tests verify supplied neighboring evidence,
relevant glossary summaries, chunk boundaries, serialized payloads, and budget effects.
The existing oversized-input regression confirms that a large neighbor does not
invalidate another otherwise usable candidate. Documentation links and whitespace
checks passed. No live model quality claims or release publication are implied.