191 lines
11 KiB
Markdown
191 lines
11 KiB
Markdown
# Consistent Span Deletion
|
|
|
|
**Last updated:** 2026-09-25
|
|
|
|
**Status:** Accepted feature direction; not implemented. This document defines
|
|
the target behavior and work required. A staged implementation plan will be
|
|
prepared separately; the existing implementation plan covers the completed
|
|
resilience work.
|
|
|
|
## Purpose and User Intent
|
|
|
|
Align proposal prompts, the LLM response schema, internal validation, and
|
|
application behavior so legitimate text-span deletions can be considered and
|
|
applied through the ordinary correction workflow. An empty replacement should
|
|
not trigger structural repair merely because it is empty, nor invalidate other
|
|
usable corrections returned alongside it.
|
|
|
|
This is a consistency fix within Audita's current editing scope. It does not
|
|
authorize whole-segment removal, broaden module responsibilities, or introduce
|
|
new recovery machinery. The accepted [resilience behavior](resilience.md) remains
|
|
in effect, including partial success, fixed chunks, and validation before mutation.
|
|
|
|
## Current Baseline
|
|
|
|
The source reviewed at `0b6cd98` contains a mismatch:
|
|
|
|
- The [spoken-word prompt](../../internal/prompts/assets/prompts/modules.spoken_word.proposal.yaml)
|
|
permits removing fillers and asks for the exact affected span and replacement,
|
|
without explaining how to represent deletion. Grammar permits punctuation and
|
|
spacing changes, which can also naturally involve deletion.
|
|
- The shared [correction response schema](../../internal/framework/responseschema/assets/correction_set.v1.json)
|
|
requires `corrected_text` to have at least one character. Every built-in proposal
|
|
module uses that schema. An empty replacement invalidates the response before
|
|
Audita evaluates individual proposals.
|
|
- PromptKit performs bounded structural repair. If repair does not produce a
|
|
valid response, the proposal operation fails and yields no candidates, including
|
|
otherwise valid proposals in the same response. The current resilience path
|
|
records the error and continues other work.
|
|
- [CorrectionProposal](../../internal/framework/proposals/proposal.go) already
|
|
allows empty replacement text. The
|
|
[preview/application logic](../../internal/framework/proposals/preview.go)
|
|
supports literal span deletion when the resulting segment remains nonblank.
|
|
- The validator named `non_empty_corrected_text` actually checks the resulting
|
|
segment, not the replacement string. Both this check and application prevent
|
|
a segment from becoming empty or whitespace-only. Application changes segment
|
|
text; it does not remove segment records.
|
|
|
|
A temporary controlled-provider review reproduced whole-response rejection for
|
|
an empty replacement alongside a valid correction, and immediate schema acceptance
|
|
for a space-only replacement. Existing proposal tests confirm internal support for
|
|
partial deletion and rejection of blank resulting segments. These observations
|
|
explain a plausible production failure mechanism; they do not establish the cause
|
|
of any particular historical incident without its payload and running version.
|
|
|
|
## Target Correction Contract
|
|
|
|
- `original_text` identifies an existing, nonblank literal span in the current
|
|
target segment. Preserve the existing matching and module replacement policies.
|
|
- `corrected_text` remains a required string. An empty string means delete the
|
|
identified span; it does not mean remove the segment record or skip validation.
|
|
Missing fields and non-string values remain structurally invalid.
|
|
- After applying a correction, the segment must contain non-whitespace text.
|
|
Enforce this using the source segment and replacement result in Audita, not a
|
|
minimum replacement length in the structural schema.
|
|
- Every accepted deletion passes the same required validators as other edits in
|
|
its module, including confidence, protected terms, editorial review, and meaning
|
|
preservation where applicable. Empty replacement is neither automatic approval
|
|
nor an automatic rejection.
|
|
- Application preserves segment identity, speaker, timing, membership, ordering,
|
|
and fixed chunk boundaries. Recheck applicability against text produced by
|
|
earlier accepted corrections, retaining existing conflict handling.
|
|
- Preserve literal replacement behavior. Prompts should select spans and
|
|
replacements that handle adjacent whitespace and punctuation appropriately;
|
|
this feature does not add automatic trimming or a second normalization pass.
|
|
Whitespace-only replacements are not categorically prohibited: their validity
|
|
depends on the resulting text and the ordinary module/domain checks.
|
|
|
|
Illustrative cases:
|
|
|
|
| Segment | Original span | Replacement | Intended handling |
|
|
| --- | --- | --- | --- |
|
|
| `uh hello` | `uh ` | empty string | Structurally valid; may produce `hello` after required validation. |
|
|
| `uh hello` | `uh hello` | `hello` | Remains a valid representation of the same cleanup, subject to validation. |
|
|
| `uh` | `uh` | empty string | Structurally valid proposal, then filtered because the resulting segment would be blank. |
|
|
| `hello world` | `hello world` | spaces only | Filtered because the resulting segment would be blank. |
|
|
|
|
These examples describe eligibility and mechanical results, not unconditional
|
|
editorial approval. An unchanged segment or valid empty correction list remains
|
|
a successful no-change outcome.
|
|
|
|
## Prompt and Module Alignment
|
|
|
|
Proposal instructions should explicitly describe empty-string deletion and its
|
|
resulting-segment limit. Include a concise example where it helps the module's
|
|
task. Tell the model to omit a correction that would leave a blank segment rather
|
|
than fabricate replacement content or padding to satisfy a schema restriction.
|
|
|
|
Retain each module's existing scope:
|
|
|
|
- Spoken-word cleanup can propose appropriate filler or dysfluency deletion while
|
|
preserving substantive content and intentional emphasis.
|
|
- Grammar can propose permitted punctuation or spacing deletion; allowing an
|
|
empty replacement does not authorize word removal or spoken-word cleanup.
|
|
- Glossary and homophone modules retain their existing correction purposes.
|
|
Shared deletion syntax does not grant them new editorial responsibilities.
|
|
|
|
Review the active LLM-validator prompts and their input construction as well as
|
|
proposal prompts. Validators must understand the before/after effect of deletion
|
|
and assess it under their existing criteria. A correct negative editorial decision
|
|
remains ordinary processing, not a defect that should be regenerated until approved.
|
|
|
|
## Outcomes and Reporting
|
|
|
|
A legitimate deletion is an ordinary correction. Empty replacement alone must
|
|
produce neither structural repair nor a recovery classification. If another
|
|
problem required successful repair or filtering, retain that actual recovery.
|
|
|
|
A schema-valid proposal that would blank a segment is a defective proposal handled
|
|
through existing safe filtering. Preserve the `empty_resulting_segment` reason
|
|
and consolidated `proposals_filtered` recovery behavior; other usable proposals
|
|
in the response continue through required validation. Ordinary editorial rejection
|
|
remains debug evidence rather than a model error or filtering recovery.
|
|
|
|
Debug evidence must accurately retain the empty replacement and the corresponding
|
|
validation/application outcome. Ordinary reports retain their existing counts and
|
|
purpose; no new report schema, incident category, or deletion-specific counter is
|
|
needed. Unrelated structural defects may still exhaust PromptKit repair and produce
|
|
an error for the operation, as today.
|
|
|
|
## Work Required
|
|
|
|
This is a description of the change surface, not an ordered implementation plan.
|
|
|
|
- Relax the shared correction schema's replacement-length constraint while
|
|
retaining its required string field and the other structural checks.
|
|
- Align proposal prompts, relevant validator prompts, and validation payloads with
|
|
the selected semantics. Reuse the existing source/result preview behavior.
|
|
- Rename the misleading resulting-segment validator and associated package,
|
|
registry, chain, metadata, test, and documentation references consistently.
|
|
Choose a name that states the resulting-segment requirement. Treat any exposed
|
|
validator identity change deliberately; do not rename the existing failure
|
|
reason merely to match the new validator name.
|
|
- Review preview and application paths for consistency, including successive edits
|
|
to the same segment. Preserve existing safeguards without adding new deletion
|
|
abstractions, configuration controls, or independent retry loops.
|
|
- Replace tests that enforce nonempty replacement strings with meaningful contract
|
|
tests, and add representative integration coverage through the real schema,
|
|
PromptKit adapter, validation chain, application, and outcome mapping.
|
|
- Update canonical module, validator, prompt, and affected output documentation
|
|
when implementation lands. Keep current behavior references accurate until then.
|
|
|
|
## Acceptance Criteria
|
|
|
|
- A controlled valid response containing an empty replacement reaches Audita's
|
|
domain validation without structural repair solely due to that empty string.
|
|
- A permitted span deletion can pass the required validators and reach the final
|
|
transcript while preserving segment metadata and fixed chunk membership.
|
|
- A mixed response can apply an approved deletion and another approved correction;
|
|
an empty replacement does not invalidate the whole response.
|
|
- A proposal that would leave a blank segment is filtered independently, retains
|
|
the existing reason, and does not prevent unrelated valid candidates from
|
|
succeeding. Application still prevents a blank result if earlier edits change
|
|
what a later proposal would do.
|
|
- Protected-term and meaning-changing deletions remain subject to their normal
|
|
rejection rules. No candidate bypasses required validation.
|
|
- A deletion needing no recovery counts as an ordinary applied correction; actual
|
|
repair/filtering and editorial rejection retain their existing classifications.
|
|
Debug evidence represents the empty replacement faithfully.
|
|
- Schema and domain tests distinguish an empty string from missing/non-string
|
|
replacement values and distinguish empty replacement from blank resulting text.
|
|
- Prompt guidance and validator naming express the same contract as the code.
|
|
Tests protect outcomes rather than exact prompt prose or incidental formatting.
|
|
|
|
Use deterministic, offline controlled responses for behavioral integration tests;
|
|
no live-provider or paid evaluation is a prerequisite. Representative production
|
|
examples can inform prompt quality without expanding this into an evaluation
|
|
framework or guaranteeing that models always follow the instructions.
|
|
|
|
## Deferred Work and Planning Boundary
|
|
|
|
[Whole-segment removal](future.md#whole-segment-removal) is only potential future
|
|
work and first requires deciding whether its benefit warrants implementation.
|
|
Domain-aware regeneration and independent whole-pipeline traversal also remain
|
|
in [future.md](future.md). This feature changes neither retry budgets nor the
|
|
Audita/PromptKit ownership boundary and requires no upstream library feature.
|
|
|
|
The product choices above are settled. The next planning pass should select
|
|
concrete names and integration-test placement and produce a staged implementation
|
|
plan. No implementation stages or application changes are introduced by this
|
|
roadmap-writing pass.
|