diff --git a/docs/roadmap/future.md b/docs/roadmap/future.md index 39d3ab8..b2f06ee 100644 --- a/docs/roadmap/future.md +++ b/docs/roadmap/future.md @@ -1,10 +1,11 @@ # Deferred Feature Work -**Last updated:** 2026-09-24 +**Last updated:** 2026-09-25 -**Status:** Intended future work; not implemented or scheduled. These entries -preserve scope deferred from the [resilience roadmap](resilience.md). They are -not prerequisites for completing that feature or staged implementation plans. +**Status:** Deferred or exploratory work; not implemented or scheduled. These +entries preserve scope outside the [resilience roadmap](resilience.md) and +[span-deletion consistency roadmap](span-deletion.md). Each entry states its +intent; inclusion does not commit us to implementing every capability. ## Domain-Aware Response Recovery @@ -50,24 +51,26 @@ still stop and join active work. Assess whether the scheduling change materially improves real workloads before introducing additional orchestration machinery. No new scheduler, global limiter, or execution mode is selected by this entry. -## Empty Replacement Text and Segment Removal +## Whole-Segment Removal -Address a frequent proposal failure reported in production: the LLM supplies an -empty string as `corrected_text`, possibly because it intends to remove the -entire segment. That interpretation is a hypothesis, not a reliable signal of -intent. Improve both prompting and handling so this case has a clear, consistent -outcome rather than repeatedly producing unusable proposals. +Potential future work, subject first to evaluating whether it is worth implementing +at all. Empty replacement text and deletion of a span within a surviving segment +are covered by the accepted [span-deletion consistency roadmap](span-deletion.md). +That feature does not remove segment records or permit blank resulting segments. -Review representative examples and distinguish deletion of a text span from -removal of an entire segment. Define whether either operation is supported, how -it is represented, and what validation is required. Align proposal prompts, -response contracts, validators, application behavior, and error/recovery reporting -with the selected semantics. If an operation is unsupported, prompts should give -clear guidance on what the model should return instead. +Before designing segment removal, review representative production examples to +establish whether whole-segment deletion is a meaningful unmet need. An empty +replacement alone does not establish that intent: it can express ordinary filler +or punctuation deletion within a larger segment. Assess how often genuinely +expendable whole segments occur after deterministic normalization, the transcript +quality benefit of removing them, and the risk of losing substantive speech. +Compare those benefits with the implementation and ongoing maintenance costs. +Retaining the segment and declining the edit is a valid outcome if the benefit is +small or uncertain. No numerical adoption threshold is selected. -Do not infer authorization to delete a segment from an empty replacement alone. -Any future support for segment removal must address meaning preservation and the -effect on segment identity, timestamps, fixed chunks, later stages, and output. -Coordinate with domain-aware response recovery where feedback could resolve an -invalid proposal. Exact deletion semantics and recovery behavior remain undecided; -this entry does not relax current validation or expand the resilience workstream. +Only if that evaluation supports proceeding should a separate feature design +choose a representation, authorization and validation rules, and reporting +semantics. It must address meaning preservation and the effects on segment +identity, timestamps, fixed chunks, subsequent stages, recombination, and output +consumers. Do not infer authorization to remove a segment from an empty string +or introduce segment-removal machinery as part of span-deletion consistency. diff --git a/docs/roadmap/span-deletion.md b/docs/roadmap/span-deletion.md new file mode 100644 index 0000000..3023be5 --- /dev/null +++ b/docs/roadmap/span-deletion.md @@ -0,0 +1,190 @@ +# Consistent Span Deletion + +**Last updated:** 2026-09-25 + +**Status:** Accepted feature direction; not implemented. This document defines +the target behavior and work required. A staged implementation plan will be +prepared separately; the existing implementation plan covers the completed +resilience work. + +## Purpose and User Intent + +Align proposal prompts, the LLM response schema, internal validation, and +application behavior so legitimate text-span deletions can be considered and +applied through the ordinary correction workflow. An empty replacement should +not trigger structural repair merely because it is empty, nor invalidate other +usable corrections returned alongside it. + +This is a consistency fix within Audita's current editing scope. It does not +authorize whole-segment removal, broaden module responsibilities, or introduce +new recovery machinery. The accepted [resilience behavior](resilience.md) remains +in effect, including partial success, fixed chunks, and validation before mutation. + +## Current Baseline + +The source reviewed at `0b6cd98` contains a mismatch: + +- The [spoken-word prompt](../../internal/prompts/assets/prompts/modules.spoken_word.proposal.yaml) + permits removing fillers and asks for the exact affected span and replacement, + without explaining how to represent deletion. Grammar permits punctuation and + spacing changes, which can also naturally involve deletion. +- The shared [correction response schema](../../internal/framework/responseschema/assets/correction_set.v1.json) + requires `corrected_text` to have at least one character. Every built-in proposal + module uses that schema. An empty replacement invalidates the response before + Audita evaluates individual proposals. +- PromptKit performs bounded structural repair. If repair does not produce a + valid response, the proposal operation fails and yields no candidates, including + otherwise valid proposals in the same response. The current resilience path + records the error and continues other work. +- [CorrectionProposal](../../internal/framework/proposals/proposal.go) already + allows empty replacement text. The + [preview/application logic](../../internal/framework/proposals/preview.go) + supports literal span deletion when the resulting segment remains nonblank. +- The validator named `non_empty_corrected_text` actually checks the resulting + segment, not the replacement string. Both this check and application prevent + a segment from becoming empty or whitespace-only. Application changes segment + text; it does not remove segment records. + +A temporary controlled-provider review reproduced whole-response rejection for +an empty replacement alongside a valid correction, and immediate schema acceptance +for a space-only replacement. Existing proposal tests confirm internal support for +partial deletion and rejection of blank resulting segments. These observations +explain a plausible production failure mechanism; they do not establish the cause +of any particular historical incident without its payload and running version. + +## Target Correction Contract + +- `original_text` identifies an existing, nonblank literal span in the current + target segment. Preserve the existing matching and module replacement policies. +- `corrected_text` remains a required string. An empty string means delete the + identified span; it does not mean remove the segment record or skip validation. + Missing fields and non-string values remain structurally invalid. +- After applying a correction, the segment must contain non-whitespace text. + Enforce this using the source segment and replacement result in Audita, not a + minimum replacement length in the structural schema. +- Every accepted deletion passes the same required validators as other edits in + its module, including confidence, protected terms, editorial review, and meaning + preservation where applicable. Empty replacement is neither automatic approval + nor an automatic rejection. +- Application preserves segment identity, speaker, timing, membership, ordering, + and fixed chunk boundaries. Recheck applicability against text produced by + earlier accepted corrections, retaining existing conflict handling. +- Preserve literal replacement behavior. Prompts should select spans and + replacements that handle adjacent whitespace and punctuation appropriately; + this feature does not add automatic trimming or a second normalization pass. + Whitespace-only replacements are not categorically prohibited: their validity + depends on the resulting text and the ordinary module/domain checks. + +Illustrative cases: + +| Segment | Original span | Replacement | Intended handling | +| --- | --- | --- | --- | +| `uh hello` | `uh ` | empty string | Structurally valid; may produce `hello` after required validation. | +| `uh hello` | `uh hello` | `hello` | Remains a valid representation of the same cleanup, subject to validation. | +| `uh` | `uh` | empty string | Structurally valid proposal, then filtered because the resulting segment would be blank. | +| `hello world` | `hello world` | spaces only | Filtered because the resulting segment would be blank. | + +These examples describe eligibility and mechanical results, not unconditional +editorial approval. An unchanged segment or valid empty correction list remains +a successful no-change outcome. + +## Prompt and Module Alignment + +Proposal instructions should explicitly describe empty-string deletion and its +resulting-segment limit. Include a concise example where it helps the module's +task. Tell the model to omit a correction that would leave a blank segment rather +than fabricate replacement content or padding to satisfy a schema restriction. + +Retain each module's existing scope: + +- Spoken-word cleanup can propose appropriate filler or dysfluency deletion while + preserving substantive content and intentional emphasis. +- Grammar can propose permitted punctuation or spacing deletion; allowing an + empty replacement does not authorize word removal or spoken-word cleanup. +- Glossary and homophone modules retain their existing correction purposes. + Shared deletion syntax does not grant them new editorial responsibilities. + +Review the active LLM-validator prompts and their input construction as well as +proposal prompts. Validators must understand the before/after effect of deletion +and assess it under their existing criteria. A correct negative editorial decision +remains ordinary processing, not a defect that should be regenerated until approved. + +## Outcomes and Reporting + +A legitimate deletion is an ordinary correction. Empty replacement alone must +produce neither structural repair nor a recovery classification. If another +problem required successful repair or filtering, retain that actual recovery. + +A schema-valid proposal that would blank a segment is a defective proposal handled +through existing safe filtering. Preserve the `empty_resulting_segment` reason +and consolidated `proposals_filtered` recovery behavior; other usable proposals +in the response continue through required validation. Ordinary editorial rejection +remains debug evidence rather than a model error or filtering recovery. + +Debug evidence must accurately retain the empty replacement and the corresponding +validation/application outcome. Ordinary reports retain their existing counts and +purpose; no new report schema, incident category, or deletion-specific counter is +needed. Unrelated structural defects may still exhaust PromptKit repair and produce +an error for the operation, as today. + +## Work Required + +This is a description of the change surface, not an ordered implementation plan. + +- Relax the shared correction schema's replacement-length constraint while + retaining its required string field and the other structural checks. +- Align proposal prompts, relevant validator prompts, and validation payloads with + the selected semantics. Reuse the existing source/result preview behavior. +- Rename the misleading resulting-segment validator and associated package, + registry, chain, metadata, test, and documentation references consistently. + Choose a name that states the resulting-segment requirement. Treat any exposed + validator identity change deliberately; do not rename the existing failure + reason merely to match the new validator name. +- Review preview and application paths for consistency, including successive edits + to the same segment. Preserve existing safeguards without adding new deletion + abstractions, configuration controls, or independent retry loops. +- Replace tests that enforce nonempty replacement strings with meaningful contract + tests, and add representative integration coverage through the real schema, + PromptKit adapter, validation chain, application, and outcome mapping. +- Update canonical module, validator, prompt, and affected output documentation + when implementation lands. Keep current behavior references accurate until then. + +## Acceptance Criteria + +- A controlled valid response containing an empty replacement reaches Audita's + domain validation without structural repair solely due to that empty string. +- A permitted span deletion can pass the required validators and reach the final + transcript while preserving segment metadata and fixed chunk membership. +- A mixed response can apply an approved deletion and another approved correction; + an empty replacement does not invalidate the whole response. +- A proposal that would leave a blank segment is filtered independently, retains + the existing reason, and does not prevent unrelated valid candidates from + succeeding. Application still prevents a blank result if earlier edits change + what a later proposal would do. +- Protected-term and meaning-changing deletions remain subject to their normal + rejection rules. No candidate bypasses required validation. +- A deletion needing no recovery counts as an ordinary applied correction; actual + repair/filtering and editorial rejection retain their existing classifications. + Debug evidence represents the empty replacement faithfully. +- Schema and domain tests distinguish an empty string from missing/non-string + replacement values and distinguish empty replacement from blank resulting text. +- Prompt guidance and validator naming express the same contract as the code. + Tests protect outcomes rather than exact prompt prose or incidental formatting. + +Use deterministic, offline controlled responses for behavioral integration tests; +no live-provider or paid evaluation is a prerequisite. Representative production +examples can inform prompt quality without expanding this into an evaluation +framework or guaranteeing that models always follow the instructions. + +## Deferred Work and Planning Boundary + +[Whole-segment removal](future.md#whole-segment-removal) is only potential future +work and first requires deciding whether its benefit warrants implementation. +Domain-aware regeneration and independent whole-pipeline traversal also remain +in [future.md](future.md). This feature changes neither retry budgets nor the +Audita/PromptKit ownership boundary and requires no upstream library feature. + +The product choices above are settled. The next planning pass should select +concrete names and integration-test placement and produce a staged implementation +plan. No implementation stages or application changes are introduced by this +roadmap-writing pass.