audita/docs/roadmap/implementation.md

Implementation Plan: Consistent Span Deletion

Last updated: 2026-09-25

Status: Ready for implementation; no stages completed. Source baseline: a83a518 (feature roadmap); application baseline 0b6cd98. Audience: gpt-5.6-terra coding agent.

Scope and Execution Rules

Implement Consistent Span Deletion. That document owns the product intent, target semantics, and acceptance criteria. This document owns the concrete decisions, ordered work, and validation. Execute Stage 1 through Stage 4 in order. Record actual completion and checks beneath each stage when implemented. Do not interpret this ready status as implemented behavior.

Read development guidance and all agent, architecture, documentation, testing, and release policies. Use graph-first code discovery. Inspect the current workspace before editing; retain unrelated user work. Commit coherent completed units with plain English messages, following the agent policy. Do not add permission rounds between stages.

Aim directly at the final state. Do not add temporary compatibility aliases, feature flags, parallel implementations, or scaffolding for intermediate commits. Keep validators in their current order and preserve module scope, literal matching, replacement policies, fixed chunks, required validation, failure isolation, reporting/debug independence, and existing recovery budgets. Do not implement any of the deferred features, automatic whitespace cleanup, a new deletion operation type, segment tombstones, or a second normalization pass.

This replaces the completed resilience implementation plan, which remains in Git history at a83a518:docs/roadmap/implementation.md. Historical resilience decisions and their remediation are not instructions to rebuild or reverse that work. During final documentation updates, repair references that describe this filename as the resilience plan; use current canonical references or the historical revision. No release, version tag, push, upstream change, or new dependency is part of this implementation sprint.

Selected Implementation Decisions

Schema and library boundary

Remove only minLength: 1 from the corrected_text property of internal/framework/responseschema/assets/correction_set.v1.json. Keep the required field, string type, and all unrelated schema constraints. Empty and whitespace-only strings are structurally valid; missing, null, and non-string replacements are not. Nonblank source-span and resulting-segment rules stay in domain validation.

Keep CorrectionSetKey, the schema asset path, ID/name/version, and existing prompt IDs/v1 selections. This widens an internal model-response constraint without changing the correction object shape or creating a second contract to maintain. The schema hash and PromptKit's rendered/prepared hashes reflect changed content; no hard-coded hash is a compatibility promise. Ordinary report schema versions, transcript encoders, configuration, and CLI selections do not change.

PromptKit continues to validate the whole response and own structural repairs. Do not introduce partial JSON salvage or Audita-side regeneration. Once the schema accepts empty replacements, normal per-candidate domain filtering handles proposals that would blank a segment.

Validator identity

Rename the existing validator consistently:

Current	Target
non_empty_corrected_text key and package directory	non_empty_resulting_segment
KeyNonEmptyCorrectedText in registry and metadata	KeyNonEmptyResultingSegment
NonEmptyCorrectionValidator framework type	NonEmptyResultingSegmentValidator

Retain the empty_resulting_segment rejection/skip reason, deterministic execution class, chain position, and preview-based behavior. Do not retain an old-key alias. This is an intentional validator identity change in diagnostic/decision records; document it, including any affected ordinary filtered-proposal metadata. It is not a new operator-selectable setting. Historical release notes remain untouched.

Prompt and payload semantics

All four proposal prompts must explain that an empty replacement deletes the identified span, within the module's existing task, and that a correction leaving a blank segment should be omitted. Prompts must not ask the model to invent padding or replacement content to avoid an empty result.

Use a spoken-word example deleting "uh " from "uh hello" with "", producing "hello". A grammar example may delete the uniquely occurring extra comma from "Hello, world." when appropriate; the representation is illustrative, not a claim that the edit is always editorially justified. Explain that selection of surrounding whitespace/punctuation must produce the intended literal result. Keep the existing uniqueness requirement and module responsibilities.

Update the three active validator prompt definitions (editorial_review, meaning_reversal_review, spoken_form_plausibility) to interpret an empty replacement in the context of the full before/after segment. Empty syntax alone must not decide approval. Meaning-reversal review remains a narrow check for antonyms/reversals; do not turn it into a general semantic-equivalence validator. Spoken-form plausibility retains its existing phonetic/mistranscription remit. The registered grammar_review and spoken_word_review definitions should receive the same deletion interpretation as their editorial-review counterpart, without adding them to active chains or deleting them as unrelated cleanup.

BuildLLMValidationRequest already supplies corrected_text, original_segment_text, and corrected_segment_text; the JSON fields are not omitted when empty. Reuse this representation. Verify it behaviorally rather than adding a new payload shape, synthetic sentinel, or special deletion flag.

Debug representation and outcomes

CorrectionLedgerEntry currently uses strings with omitempty for both replacement fields, losing explicit empty values. Change ProposedCorrectedText and AppliedCorrectedText to *string with their existing JSON names and omitempty:

• Every candidate-backed entry has a non-nil proposed value, including "".
• Applied entries also have a non-nil applied value, including "".
• Rejected, skipped, and interrupted entries omit the applied value.
• Module-level failure entries without a candidate omit both values.

Construct the pointers directly when mapping the corresponding records; use the same simple convention across all dispositions. Do not build custom marshaling or a generic optional-value framework. These values remain replacement spans, not full resulting-segment text. Preserve redaction and the rest of the ledger.

Keep existing operation accounting. A valid approved deletion is an ordinary applied correction. A candidate that would blank the segment is filtered with the existing reason and contributes to the proposal operation's consolidated filtering recovery. Ordinary editorial/protected-term rejection does not become a recovery. Existing apply-time conflict handling and attribution remain unchanged. Structural repairs required for unrelated defects still count normally.

Stage 1 — Align Validator Identity and Confirm Domain Semantics

Work

• Rename the validator package/type/key described above. Update New() wiring, internal/validators/registry.go, chains.go, metadata, framework validators, consumers, and tests in one coherent change. Use graph discovery for symbols and string searches for serialized keys. Do not globally rewrite historical docs.
• Preserve existing CorrectionProposal.Validate, PreviewProposalForSegment, ApplyProposals, and deterministic validation behavior wherever already correct. Verify that there is no additional nonempty-replacement restriction downstream.
• Extend the existing proposal and validator tests only for missing meaningful coverage: empty replacement leaving a nonblank segment, whitespace-only resulting text rejection, literal spacing, and apply-time reevaluation after an earlier edit. For the last case, use segment "a b" and ordered deletions "a" -> "", "b" -> "": each is previewable initially, but after the first edit the second must be skipped as empty_resulting_segment. The surviving text is " b"; no automatic trim or segment deletion is expected.
• Ensure protected-term validation still rejects removal of a protected term with an empty replacement. Prefer one case in its existing test table. Retain existing metadata-preservation tests rather than duplicating all metadata assertions.
• Update docs/internal/validators.md for the renamed implemented validator and explain its resulting-segment responsibility; do not yet claim schema alignment.

Validation

Run go test ./internal/framework/proposals ./internal/framework/validators ./internal/validators/... ./internal/framework/runner ./internal/framework/processreport. Existing registry/chain tests must use the new identity and preserve order/class. Inspect remaining old-name references, retaining only deliberately historical ones.

End state: The accurately named domain validator and application preserve the agreed behavior. No schema or retry workaround has been introduced.

Stage 2 — Align Response Schema, Prompts, and LLM Validation Inputs

Work

• Apply the selected schema relaxation and prompt changes. Keep unrelated prompt instructions, hardening, variables, and response associations intact. Review the rendered definitions, not just YAML source, using existing PromptKit inspection and prompt tests.
• In internal/framework/responseschema/registry_test.go, retain the check that reported hashes equal hashes of the actual embedded bytes. Remove fixed digest expectations that merely freeze asset content; do not replace them with a new golden digest for this change.
• Extend the existing real-adapter tests in internal/framework/llm/adapter_test.go with a bounded table for replacement values: empty string, spaces, ordinary text, missing field, null, and a non-string value. Use the embedded schema and local HTTP fixture. Give accepted cases a positive repair allowance and assert one provider generation, zero repairs, and preservation of the actual string. For invalid cases, use zero repair attempts to prove rejection without redundant model calls. Existing tests already own successful repair/exhaustion behavior.
• Extend the validation-request tests in internal/framework/validators/llm_validators_test.go to marshal a real request for deleting "uh " from "uh hello". Assert that JSON contains the explicit empty replacement plus full before/after text, correct identity, and no immediate rejection. A field-presence assertion is required; decoding into a plain string alone cannot distinguish omitted from empty.
• Update canonical prompt/module guidance for the implemented semantics. Do not add exact-prose prompt snapshots or assertions for every new sentence; existing parsing/rendering/schema-association tests plus editorial review are sufficient.

Validation

Run go test ./internal/framework/responseschema ./internal/prompts ./internal/framework/llm ./internal/framework/proposal_generation ./internal/framework/validators ./internal/modules/.... Confirm missing/non-string replacements still fail structurally and domain-invalid but schema-valid proposals still reach ordinary filtering without an outer retry.

End state: A literal empty replacement can cross the real library/schema boundary, and the model instructions and validator inputs describe its meaning.

Stage 3 — Preserve Empty Replacements in Debug Evidence

Work

• Implement the pointer representation selected above in internal/framework/processreport/correction_ledger.go, updating every mapping branch and affected test consumer. Leave other debug structures unchanged where their string fields already serialize empty values explicitly.
• Extend correction_ledger_test.go with a small serialized-disposition table: applied deletion, rejected deletion, skipped deletion, interrupted deletion, and a module failure without a candidate. Decode to a map or raw JSON fields to check presence, "" values, and absence of the applied value on unapplied entries. Preserve existing decision classification/order tests using the renamed key.
• Update docs/integrations/output-files.md to explain the two debug replacement fields and the deliberate validator identity change. Keep ordinary schemas and report counts unchanged. Update the diagnostics/reporting internal reference for the implemented mapping, linking to the public artifact contract.

Validation

Run go test ./internal/framework/processreport ./internal/core/diagnostics ./internal/cli ./cmd/audita. Tests must inspect serialized output, not just non-nil in-memory values. Retain existing configured-secret redaction and debug-off persistence coverage.

End state: Debug evidence distinguishes a proposed/applied deletion from an absent proposal or an edit that was never applied.

Stage 4 — Verify the Integrated Contract and Finish Documentation

Work

Extend cmd/audita/main_integration_test.go with representative offline subprocess cases using its controlled HTTP provider, real configuration/profiles, embedded schemas, built-in module/validator chains, CLI, and output/report/debug writers. Keep fixture additions local and small; do not build a new integration framework. Use explicit fixture sizing so relevant validator candidates fit one batch, and use stable response indices. Do not rely on production chunk/token defaults or on concurrent request order to choose responses.

Cover these distinct outcomes:

• Clean deletion plus another correction: A spoken-word stage proposes "uh " -> "" in "uh hello" and an independent repetition collapse. Controlled validators approve both. Assert corrected transcript, two applied corrections, zero model errors/recoveries, zero structural repairs with a positive repair allowance, and explicit empty proposed/applied fields in the debug ledger.
• Partial success with filtering: Add a proposal to delete all text from a separate filler-only segment, preserving that separate segment through fixture normalization (use sufficient time separation or explicit normalization settings). Assert that this candidate is filtered with empty_resulting_segment, the other corrections still apply, the blanked segment is instead retained unchanged, and the proposal response contributes one consolidated filtering recovery and no work-unit error. The renamed deterministic validator must appear in evidence.
• Ordinary negative decision: Return a schema-valid deletion proposal, then a valid negative decision from meaning-reversal review after earlier checks approve it. Assert unchanged affected text, no applied deletion, no error/recovery solely from the rejection, and a rejected debug entry. The controlled decision proves the chain respects a negative decision, not that an arbitrary real model will make the same editorial judgment.

Reuse existing YAML/Markdown/CLI count assertions rather than snapshotting entire reports. Verify that the clean case's report totals agree with the transcript and ledger. Compare resulting metadata to the normalized input, since initial normalization may legitimately merge segments or reassign IDs. Confirm retained section membership/order in the existing runner boundary tests; add a targeted assertion only if that property is not already covered for the exercised path. Do not repeat all cases for all modules, encoders, debug/report combinations, or platforms: those unchanged mechanisms already have coverage.

Finish the documentation and status work:

• Review current module, validator, prompt, diagnostics, and output references for consistency. Add a concise note to docs/migrations/v2.md describing the renamed validator identity and explicit empty debug replacement fields; no alias or migration tool is required. No release note/version is selected in this sprint.
• Review links from other roadmaps to implementation.md. References to completed resilience stages should point to their historical plan at the recorded revision or current canonical behavior, not silently identify this plan as that work. Update those references only as needed; do not rewrite historical findings.
• Mark this plan and span-deletion.md implemented only after the criteria pass. Keep the baseline description historical and retain future.md's exploratory whole-segment-removal status. Record concise evidence under completed stages.

Final validation

Run go test ./... once after the integrated changes. Run git diff --check and validate affected documentation links/examples. Focused stage checks need not be repeated solely as ceremony. Additional race/platform/live-provider checks are not automatic gates for this feature; run additional checks only for a concrete risk introduced by actual changes. This is not a release invocation.

End state: The feature roadmap's acceptance criteria are evidenced by real schema/adapter tests, domain/application tests, serialized debug tests, and representative complete CLI workflows. The workspace is committed and the docs describe implemented behavior without claiming whole-segment removal.

Open Questions

None requiring a product decision remain. Validator names, schema/prompt identity, debug representation, module scope, test boundaries, and deferred behavior are selected above. Ordinary implementation details may be resolved idiomatically within these decisions; a newly discovered conflict with the agreed contract should be recorded rather than resolved by silently strengthening restrictions.