eric/audita
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 6 Wiki Activity

Define consistent span deletion and defer whole-segment removal

Browse Source
This commit is contained in:
Eric Rakestraw
2026-09-25 01:16:17 +00:00
parent 0b6cd98262
commit a83a51848b
2 changed files with 215 additions and 22 deletions
Download Patch File Download Diff File Expand all files Collapse all files

47
docs/roadmap/future.md
View File

@@ -1,10 +1,11 @@
# Deferred Feature Work
**Last updated:** 2026-09-24
**Last updated:** 2026-09-25
**Status:** Intended future work; not implemented or scheduled. These entries
preserve scope deferred from the [resilience roadmap](resilience.md). They are
not prerequisites for completing that feature or staged implementation plans.
**Status:** Deferred or exploratory work; not implemented or scheduled. These
entries preserve scope outside the [resilience roadmap](resilience.md) and
[span-deletion consistency roadmap](span-deletion.md). Each entry states its
intent; inclusion does not commit us to implementing every capability.
## Domain-Aware Response Recovery
@@ -50,24 +51,26 @@ still stop and join active work. Assess whether the scheduling change materially
improves real workloads before introducing additional orchestration machinery.
No new scheduler, global limiter, or execution mode is selected by this entry.
## Empty Replacement Text and Segment Removal
## Whole-Segment Removal
Address a frequent proposal failure reported in production: the LLM supplies an
empty string as `corrected_text`, possibly because it intends to remove the
entire segment. That interpretation is a hypothesis, not a reliable signal of
intent. Improve both prompting and handling so this case has a clear, consistent
outcome rather than repeatedly producing unusable proposals.
Potential future work, subject first to evaluating whether it is worth implementing
at all. Empty replacement text and deletion of a span within a surviving segment
are covered by the accepted [span-deletion consistency roadmap](span-deletion.md).
That feature does not remove segment records or permit blank resulting segments.
Review representative examples and distinguish deletion of a text span from
removal of an entire segment. Define whether either operation is supported, how
it is represented, and what validation is required. Align proposal prompts,
response contracts, validators, application behavior, and error/recovery reporting
with the selected semantics. If an operation is unsupported, prompts should give
clear guidance on what the model should return instead.
Before designing segment removal, review representative production examples to
establish whether whole-segment deletion is a meaningful unmet need. An empty
replacement alone does not establish that intent: it can express ordinary filler
or punctuation deletion within a larger segment. Assess how often genuinely
expendable whole segments occur after deterministic normalization, the transcript
quality benefit of removing them, and the risk of losing substantive speech.
Compare those benefits with the implementation and ongoing maintenance costs.
Retaining the segment and declining the edit is a valid outcome if the benefit is
small or uncertain. No numerical adoption threshold is selected.
Do not infer authorization to delete a segment from an empty replacement alone.
Any future support for segment removal must address meaning preservation and the
effect on segment identity, timestamps, fixed chunks, later stages, and output.
Coordinate with domain-aware response recovery where feedback could resolve an
invalid proposal. Exact deletion semantics and recovery behavior remain undecided;
this entry does not relax current validation or expand the resilience workstream.
Only if that evaluation supports proceeding should a separate feature design
choose a representation, authorization and validation rules, and reporting
semantics. It must address meaning preservation and the effects on segment
identity, timestamps, fixed chunks, subsequent stages, recombination, and output
consumers. Do not infer authorization to remove a segment from an empty string
or introduce segment-removal machinery as part of span-deletion consistency.

190
docs/roadmap/span-deletion.md Normal file
View File

@@ -0,0 +1,190 @@
# Consistent Span Deletion
**Last updated:** 2026-09-25
**Status:** Accepted feature direction; not implemented. This document defines
the target behavior and work required. A staged implementation plan will be
prepared separately; the existing implementation plan covers the completed
resilience work.
## Purpose and User Intent
Align proposal prompts, the LLM response schema, internal validation, and
application behavior so legitimate text-span deletions can be considered and
applied through the ordinary correction workflow. An empty replacement should
not trigger structural repair merely because it is empty, nor invalidate other
usable corrections returned alongside it.
This is a consistency fix within Audita's current editing scope. It does not
authorize whole-segment removal, broaden module responsibilities, or introduce
new recovery machinery. The accepted [resilience behavior](resilience.md) remains
in effect, including partial success, fixed chunks, and validation before mutation.
## Current Baseline
The source reviewed at `0b6cd98` contains a mismatch:
- The [spoken-word prompt](../../internal/prompts/assets/prompts/modules.spoken_word.proposal.yaml)
permits removing fillers and asks for the exact affected span and replacement,
without explaining how to represent deletion. Grammar permits punctuation and
spacing changes, which can also naturally involve deletion.
- The shared [correction response schema](../../internal/framework/responseschema/assets/correction_set.v1.json)
requires `corrected_text` to have at least one character. Every built-in proposal
module uses that schema. An empty replacement invalidates the response before
Audita evaluates individual proposals.
- PromptKit performs bounded structural repair. If repair does not produce a
valid response, the proposal operation fails and yields no candidates, including
otherwise valid proposals in the same response. The current resilience path
records the error and continues other work.
- [CorrectionProposal](../../internal/framework/proposals/proposal.go) already
allows empty replacement text. The
[preview/application logic](../../internal/framework/proposals/preview.go)
supports literal span deletion when the resulting segment remains nonblank.
- The validator named `non_empty_corrected_text` actually checks the resulting
segment, not the replacement string. Both this check and application prevent
a segment from becoming empty or whitespace-only. Application changes segment
text; it does not remove segment records.
A temporary controlled-provider review reproduced whole-response rejection for
an empty replacement alongside a valid correction, and immediate schema acceptance
for a space-only replacement. Existing proposal tests confirm internal support for
partial deletion and rejection of blank resulting segments. These observations
explain a plausible production failure mechanism; they do not establish the cause
of any particular historical incident without its payload and running version.
## Target Correction Contract
- `original_text` identifies an existing, nonblank literal span in the current
target segment. Preserve the existing matching and module replacement policies.
- `corrected_text` remains a required string. An empty string means delete the
identified span; it does not mean remove the segment record or skip validation.
Missing fields and non-string values remain structurally invalid.
- After applying a correction, the segment must contain non-whitespace text.
Enforce this using the source segment and replacement result in Audita, not a
minimum replacement length in the structural schema.
- Every accepted deletion passes the same required validators as other edits in
its module, including confidence, protected terms, editorial review, and meaning
preservation where applicable. Empty replacement is neither automatic approval
nor an automatic rejection.
- Application preserves segment identity, speaker, timing, membership, ordering,
and fixed chunk boundaries. Recheck applicability against text produced by
earlier accepted corrections, retaining existing conflict handling.
- Preserve literal replacement behavior. Prompts should select spans and
replacements that handle adjacent whitespace and punctuation appropriately;
this feature does not add automatic trimming or a second normalization pass.
Whitespace-only replacements are not categorically prohibited: their validity
depends on the resulting text and the ordinary module/domain checks.
Illustrative cases:
| Segment | Original span | Replacement | Intended handling |
| --- | --- | --- | --- |
| `uh hello` | `uh ` | empty string | Structurally valid; may produce `hello` after required validation. |
| `uh hello` | `uh hello` | `hello` | Remains a valid representation of the same cleanup, subject to validation. |
| `uh` | `uh` | empty string | Structurally valid proposal, then filtered because the resulting segment would be blank. |
| `hello world` | `hello world` | spaces only | Filtered because the resulting segment would be blank. |
These examples describe eligibility and mechanical results, not unconditional
editorial approval. An unchanged segment or valid empty correction list remains
a successful no-change outcome.
## Prompt and Module Alignment
Proposal instructions should explicitly describe empty-string deletion and its
resulting-segment limit. Include a concise example where it helps the module's
task. Tell the model to omit a correction that would leave a blank segment rather
than fabricate replacement content or padding to satisfy a schema restriction.
Retain each module's existing scope:
- Spoken-word cleanup can propose appropriate filler or dysfluency deletion while
preserving substantive content and intentional emphasis.
- Grammar can propose permitted punctuation or spacing deletion; allowing an
empty replacement does not authorize word removal or spoken-word cleanup.
- Glossary and homophone modules retain their existing correction purposes.
Shared deletion syntax does not grant them new editorial responsibilities.
Review the active LLM-validator prompts and their input construction as well as
proposal prompts. Validators must understand the before/after effect of deletion
and assess it under their existing criteria. A correct negative editorial decision
remains ordinary processing, not a defect that should be regenerated until approved.
## Outcomes and Reporting
A legitimate deletion is an ordinary correction. Empty replacement alone must
produce neither structural repair nor a recovery classification. If another
problem required successful repair or filtering, retain that actual recovery.
A schema-valid proposal that would blank a segment is a defective proposal handled
through existing safe filtering. Preserve the `empty_resulting_segment` reason
and consolidated `proposals_filtered` recovery behavior; other usable proposals
in the response continue through required validation. Ordinary editorial rejection
remains debug evidence rather than a model error or filtering recovery.
Debug evidence must accurately retain the empty replacement and the corresponding
validation/application outcome. Ordinary reports retain their existing counts and
purpose; no new report schema, incident category, or deletion-specific counter is
needed. Unrelated structural defects may still exhaust PromptKit repair and produce
an error for the operation, as today.
## Work Required
This is a description of the change surface, not an ordered implementation plan.
- Relax the shared correction schema's replacement-length constraint while
retaining its required string field and the other structural checks.
- Align proposal prompts, relevant validator prompts, and validation payloads with
the selected semantics. Reuse the existing source/result preview behavior.
- Rename the misleading resulting-segment validator and associated package,
registry, chain, metadata, test, and documentation references consistently.
Choose a name that states the resulting-segment requirement. Treat any exposed
validator identity change deliberately; do not rename the existing failure
reason merely to match the new validator name.
- Review preview and application paths for consistency, including successive edits
to the same segment. Preserve existing safeguards without adding new deletion
abstractions, configuration controls, or independent retry loops.
- Replace tests that enforce nonempty replacement strings with meaningful contract
tests, and add representative integration coverage through the real schema,
PromptKit adapter, validation chain, application, and outcome mapping.
- Update canonical module, validator, prompt, and affected output documentation
when implementation lands. Keep current behavior references accurate until then.
## Acceptance Criteria
- A controlled valid response containing an empty replacement reaches Audita's
domain validation without structural repair solely due to that empty string.
- A permitted span deletion can pass the required validators and reach the final
transcript while preserving segment metadata and fixed chunk membership.
- A mixed response can apply an approved deletion and another approved correction;
an empty replacement does not invalidate the whole response.
- A proposal that would leave a blank segment is filtered independently, retains
the existing reason, and does not prevent unrelated valid candidates from
succeeding. Application still prevents a blank result if earlier edits change
what a later proposal would do.
- Protected-term and meaning-changing deletions remain subject to their normal
rejection rules. No candidate bypasses required validation.
- A deletion needing no recovery counts as an ordinary applied correction; actual
repair/filtering and editorial rejection retain their existing classifications.
Debug evidence represents the empty replacement faithfully.
- Schema and domain tests distinguish an empty string from missing/non-string
replacement values and distinguish empty replacement from blank resulting text.
- Prompt guidance and validator naming express the same contract as the code.
Tests protect outcomes rather than exact prompt prose or incidental formatting.
Use deterministic, offline controlled responses for behavioral integration tests;
no live-provider or paid evaluation is a prerequisite. Representative production
examples can inform prompt quality without expanding this into an evaluation
framework or guaranteeing that models always follow the instructions.
## Deferred Work and Planning Boundary
[Whole-segment removal](future.md#whole-segment-removal) is only potential future
work and first requires deciding whether its benefit warrants implementation.
Domain-aware regeneration and independent whole-pipeline traversal also remain
in [future.md](future.md). This feature changes neither retry budgets nor the
Audita/PromptKit ownership boundary and requires no upstream library feature.
The product choices above are settled. The next planning pass should select
concrete names and integration-test placement and produce a staged implementation
plan. No implementation stages or application changes are introduced by this
roadmap-writing pass.