From 7199cb565fbe685801a2c0d8150254dda7a75d9e Mon Sep 17 00:00:00 2001 From: Eric Rakestraw Date: Fri, 25 Sep 2026 03:27:41 +0000 Subject: [PATCH] Plan partial validator decision handling --- docs/roadmap/decision-set-reconciliation.md | 55 +- docs/roadmap/implementation.md | 536 ++++++++------------ 2 files changed, 232 insertions(+), 359 deletions(-) diff --git a/docs/roadmap/decision-set-reconciliation.md b/docs/roadmap/decision-set-reconciliation.md index b4491d5..432f79e 100644 --- a/docs/roadmap/decision-set-reconciliation.md +++ b/docs/roadmap/decision-set-reconciliation.md @@ -3,8 +3,9 @@ **Last updated:** 2026-09-25 **Status:** Accepted feature direction; implementation not started. Intended for -consideration before v2.0.0. This document defines the target behavior and work -needed; a staged implementation plan will follow separately. +consideration before v2.0.0. This document owns the target behavior and user intent. +[implementation.md](implementation.md) owns concrete decisions, ordered work, +and verification. ## Purpose and User Intent @@ -109,43 +110,17 @@ entries as defective correction proposals. Concrete reason names and minimal dat model changes belong in the implementation plan. Keep YAML, Markdown, and CLI counts consistent, with the existing reporting-disable and independent debug options. -## Work Needed +## Ownership and Target Architecture -### Decision Mapping and Validator Integration +Audita's validator layer reconciles structurally valid model responses into +complete internal candidate outcomes. The runner preserves strict internal +cardinality and advances only approved candidates. Reporting consumes operation +facts without treating partial validation failure as loss of every decision. +Debug evidence distinguishes unavailable validation from editorial rejection. -Replace the fail-fast model-response mapper with reconciliation that preserves -unambiguous decisions and explicitly represents unavailable candidates and ignored -entries. Update the LLM-backed validator to return those outcomes and accurate -operation facts instead of rejecting a whole usable batch. Keep domain ownership -in Audita and reuse the existing unavailable-candidate path. - -Review the confidence check currently present in the mapper: structurally invalid -confidence is already handled by the schema boundary. Do not introduce a second -whole-batch failure path for an individual defective entry that reaches domain -mapping. Preserve valid negative decisions and remaining validator execution. - -### Reporting and Debug Evidence - -Adapt operation mapping and presentation where needed to express partial batch -failure and successful extra-entry filtering. Review the correction ledger so it -records unavailable outcomes distinctly from actual editorial decisions. Preserve -existing report grouping and avoid a new reporting subsystem or per-entry model -operation. Add only the attribution needed to explain these outcomes. - -### Regression Coverage and Documentation - -Replace tests that enforce whole-batch rejection with focused tests of partial -success. Use mixed responses to prove valid approvals and rejections survive -alongside missing, duplicate, and extraneous indices. Cover conflicting duplicates, -empty decision lists, and response-order independence without building an exhaustive -cross-product suite. - -Include representative assembled-path coverage proving that survivors complete -all remaining validators, unavailable candidates never apply, and reporting reflects -the correct errors/recoveries and applied corrections. Retain tests for strict -internal cardinality and genuine operation-level failures. Update canonical -validator and reporting documentation when implementation lands; do not describe -this roadmap's target as currently implemented elsewhere. +The schema/transport boundary remains with PromptKit; reconciliation needs no +additional model call. Concrete helper structures, reason names, mapping changes, +and verification steps belong in the [implementation plan](implementation.md). ## Scope Boundaries @@ -184,7 +159,5 @@ are introduced. This work does not itself authorize tagging or publishing v2.0.0 ## Open Questions -No unresolved product-policy questions block the staged implementation plan. -That plan should select the smallest reporting/model changes and concrete reason -codes needed to express the behavior above. Guided recovery and structural salvage -remain separate future decisions. +No unresolved product-policy questions block implementation. Guided recovery and +structural salvage remain separate future decisions. diff --git a/docs/roadmap/implementation.md b/docs/roadmap/implementation.md index 13b1e94..56a5470 100644 --- a/docs/roadmap/implementation.md +++ b/docs/roadmap/implementation.md @@ -1,368 +1,268 @@ -# Implementation Plan: Consistent Span Deletion +# Implementation Plan: Partial Validator Decision Sets **Last updated:** 2026-09-25 -**Status:** Implemented; all stages complete. -**Source baseline:** `a83a518` (feature roadmap); application baseline `0b6cd98`. +**Status:** Ready for implementation; no stages implemented by this planning pass. +**Source baseline:** `60e2fe7`. **Audience:** gpt-5.6-terra coding agent. -## Scope and Execution Rules +## Scope and Execution -Implement [Consistent Span Deletion](span-deletion.md). That document owns the -product intent, target semantics, and acceptance criteria. This document owns the -concrete decisions, ordered work, and validation. Execute Stage 1 through Stage 4 -in order. Record actual completion and checks beneath each stage when implemented. -The completion evidence below records the implemented behavior; the source -baseline remains historical. +Implement [partial success for validator decision sets](decision-set-reconciliation.md). +That roadmap owns product intent and target semantics; this document selects the +implementation and verification work. Follow Stage 1 through Stage 3 in order and +record completion evidence beneath each stage. Each stage is a focused agent task. -Read [development guidance](../development.md) and all [agent](../policy/agents.md), +Read [development guidance](../development.md) and all policies under +[docs/policy](../policy/): [agents](../policy/agents.md), [architecture](../policy/architecture.md), [documentation](../policy/documentation.md), -[testing](../policy/testing.md), and [release](../policy/release.md) policies. -Use graph-first code discovery. Inspect the current workspace before editing; -retain unrelated user work. Commit coherent completed units with plain English -messages, following the agent policy. Do not add permission rounds between stages. +[testing](../policy/testing.md), and [release](../policy/release.md). Use graph-first +code discovery, preserve unrelated work, and commit completed coherent units with +plain English messages. Do not introduce approval rounds between stages. -Aim directly at the final state. Do not add temporary compatibility aliases, -feature flags, parallel implementations, or scaffolding for intermediate commits. -Preserve module scope, literal matching, replacement policies, fixed chunks, -required validation, failure isolation, -reporting/debug independence, and existing recovery budgets. Do not implement any -of the [deferred features](future.md), automatic whitespace cleanup, a new deletion -operation type, segment tombstones, or a second normalization pass. +Aim directly at the final implementation. No temporary modes, compatibility shims, +new configuration, extra model calls, or upstream dependency work is required. +Keep PromptKit structural validation/repair and the runner's internal cardinality +checks. Implement the agreed outcomes, not extra restrictions suggested by existing +tests or incidental internal structure. Internal helper signatures and layout can +be adjusted idiomatically while preserving the decisions below. -Preserve intended safeguards and outcomes, not incidental implementation choices. -Validator ordering, internal structure, and payload construction may change where -needed to satisfy the feature contract. Current code and tests are evidence to -review, not authority to retain a conflicting restriction. Prefer reusing correct -mechanisms, but do not freeze them merely because they exist. - -This replaces the completed resilience implementation plan, which remains in Git -history at `a83a518:docs/roadmap/implementation.md`. Historical resilience decisions -and their remediation are not instructions to rebuild or reverse that work. -During final documentation updates, repair references that describe this filename -as the resilience plan; use current canonical references or the historical revision. -No release, version tag, push, upstream change, or new dependency is part of this -implementation sprint. +This replaces the completed span-deletion plan, available in Git at +`60e2fe7:docs/roadmap/implementation.md`. It does not reopen that feature. The final +stage includes correcting historical links that otherwise point to this new plan. +No release preparation, tag, push, CI wait, or paid-provider evaluation is included. ## Selected Implementation Decisions -### Schema and library boundary +### Reconciliation boundary -Remove `minLength: 1` from the `corrected_text` property of -`internal/framework/responseschema/assets/correction_set.v1.json`. Keep the required -field, string type, and all unrelated schema constraints. Empty and whitespace-only -strings are structurally valid; missing, null, and non-string replacements are not. -Keep `original_text` required, string-valued, and at least one character long. -Whitespace-only original spans are valid at the structural boundary and must not -be rejected solely on that basis by domain checks. Replace trimmed-emptiness checks -for original spans with literal empty-string checks in `CorrectionProposal.Validate` -and `ProposalShapeValidator.Validate`, and review other consumers for the same -restriction. Do not trim either span in transit. Resulting-segment checks still use -whitespace-aware emptiness. Preserve module matching policies: `require_unique` -for grammar, spoken-word, and homophones; `replace_all` for glossary. +Keep reconciliation in `internal/framework/validators`, after successful structured +completion and before converting the batch into runner-facing results. Replace +`mapLLMResponseToDecisions` with a small pure reconciliation helper; use an ordinary +private result struct containing: -Keep `CorrectionSetKey`, the schema asset path, ID/name/version, and existing prompt -IDs/`v1` selections. This widens an internal model-response constraint without -changing the correction object shape or creating a second contract to maintain. -The schema hash and PromptKit's rendered/prepared hashes reflect changed content; -no hard-coded hash is a compatibility promise. Ordinary report schema versions, -transcript encoders, configuration, and CLI selections do not change. +- `Decisions []Decision`: one outcome per requested item, in request order; +- `UnavailableIndices []int`: requested indices without a usable verdict; +- `IgnoredIndices []int`: unexpected indices encountered in response order, including + repeated unexpected entries, for debug attribution. -PromptKit continues to validate the whole response and own structural repairs. -Do not introduce partial JSON salvage or Audita-side regeneration. Once the schema -accepts empty replacements, normal per-candidate domain filtering handles proposals -that would blank their stage-input segment. +No generic reconciliation framework or new shared abstraction is needed. Requested +identities come from `batch.Items`, not transcript segment IDs, the whole section, +or any other batch. Expected input identities remain an Audita internal contract. -### Validator identity +First collect response entries by requested index and count occurrences, separating +unexpected indices. Then walk requested items once to construct outcomes. This +avoids accepting an early duplicate before seeing a conflicting later entry. -Rename the existing validator consistently: +Use these candidate-level reason codes in `validators/models.go`: -| Current | Target | +| Condition | Internal outcome/reason | | --- | --- | -| `non_empty_corrected_text` key and package directory | `non_empty_resulting_segment` | -| `KeyNonEmptyCorrectedText` in registry and metadata | `KeyNonEmptyResultingSegment` | -| `NonEmptyCorrectionValidator` framework type | `NonEmptyResultingSegmentValidator` | +| Missing requested entry | `DispositionUnavailable`, `missing_validation_decision` | +| More than one entry for a requested index | `DispositionUnavailable`, `duplicate_validation_decision` | +| Single requested entry failing the existing confidence range check | `DispositionUnavailable`, `invalid_validation_decision` | +| Single usable entry | Existing approved or `llm_rejected` decision with its model reason. | -Retain the `empty_resulting_segment` rejection/skip reason, deterministic execution -class, and the resulting-segment safeguard. Reuse preview-based validation where -correct; neither its exact implementation nor chain position is frozen. Do not -retain an old-key alias. -This is an intentional validator identity change in diagnostic/decision records; -document it, including any affected ordinary filtered-proposal metadata. It is not -a new operator-selectable setting. Historical release notes remain untouched. +Unavailable outcomes have `Approved: false` and fixed safe explanatory messages. +Duplicate classification takes precedence over inspecting individual duplicate +values; identical duplicates follow the same rule. Unexpected entries are ignored +before inspecting confidence. Retain the existing confidence check as a local +check for decoded callers/test doubles, scoped to that candidate; do not broaden +this into structural salvage, confidence thresholds, or new semantic validation. +The production JSON Schema still rejects out-of-range confidence before mapping. -### Prompt and payload semantics +An empty response list makes every requested item unavailable. Response ordering +must not change verdicts or which candidate is affected. Preserve the existing +handling of valid negative decisions, including reason text. No candidate may +advance merely because another candidate was approved. -Review all four proposal definitions and the active validator definitions -(`editorial_review`, `meaning_reversal_review`, `spoken_form_plausibility`) for -consistency. Edit only definitions needing clarification. Review the registered -but inactive `grammar_review` and `spoken_word_review` definitions too; unchanged, -consistent definitions need no edits. Do not activate additional validators as -part of this work. +### Operation accounting and ordinary reports -Where helpful, explain that an empty replacement deletes the identified span, -whitespace-only spans can express spacing corrections, and the resulting segment -must remain nonblank. Do not instruct the model to fabricate padding or content. -A spoken-word example could delete `"uh "` from `"uh hello"`; a grammar example -could reduce two spaces to one. Examples and wording are illustrative, not frozen -requirements or promises of automatic approval. Explain surrounding whitespace -and punctuation only to the extent needed to produce the intended literal result. -Qualify uniqueness by module: glossary uses `replace_all`, while the other three -modules use `require_unique`. +Each actual validation batch remains one `outcomes.Operation`. Use existing +`ReasonValidationDecisionsInvalid` (`validation_decisions_invalid`) for a batch +with unavailable requested candidates. Set `Succeeded: false`, set `ProposalIndices` +to only those unavailable indices, retain execution counts, and leave recovery +reasons empty. Still return usable decisions from that batch alongside explicit +unavailable outcomes, with no Go error for the reconciled model defect. -Keep module responsibilities intact. Meaning-reversal review remains a narrow -check for antonyms/reversals, not a new general semantic-equivalence validator. -Spoken-form plausibility retains its phonetic/mistranscription remit. Empty syntax -alone must not determine approval or rejection. +If no requested candidates are unavailable, set `Succeeded: true`. Add +`outcomes.RecoveryValidationDecisionsFiltered` with serialized value +`validation_decisions_filtered` when extras were ignored. Merge it with actual +`output_repaired` and `service_recovered` reasons in the same recovery operation. +Valid negative decisions do not prevent operation success. A clean response creates +no new recovery reason. -`BuildLLMValidationRequest` already supplies explicit `corrected_text` and full -before/after segment text. Start with that representation and verify that empty -and whitespace-only spans survive it faithfully. Adjust payload construction if -needed for consistency; do not treat its current structure as an independent -requirement. No new deletion flag or synthetic sentinel is needed merely to -represent an empty string. +When unavailable candidates and extras coexist, record the failed operation only; +keep ignored-entry evidence in debug. Do not create both an error and a recovery +for that operation. Successful survivors must continue even though the operation +itself records a failure. Retain actual execution counts on all paths. -### Debug representation and outcomes +Reuse existing ordinary YAML structures and version 1. The only new public report +value is the decision-specific recovery reason; affected error indices already fit +`Failure.ProposalIndices`. Do not add ignored entries to `Operation.Filtered` or +`filtered_proposals`, which describe correction proposals, and do not increment +`proposal_responses_with_filtering`. The recovery reason and existing operation +identity suffice for ordinary reporting; detailed extra indices belong in debug. -`CorrectionLedgerEntry` currently uses strings with `omitempty` for both replacement -fields, losing explicit empty values. Change `ProposedCorrectedText` and -`AppliedCorrectedText` to `*string` with their existing JSON names and `omitempty`: +Update `processreport.safeMessage` for `validation_decisions_invalid` to explain +that some required decisions were unavailable without claiming all batch decisions +were discarded. Generalize the recovery message to include response filtering, +rather than only proposal filtering. Exact prose is not a test contract. Preserve +work-unit error grouping, operation counts, and existing YAML/Markdown/CLI totals. -- Every candidate-backed entry has a non-nil proposed value, including `""`. -- Applied entries also have a non-nil applied value, including `""`. -- Rejected, skipped, and interrupted entries omit the applied value. -- Module-level failure entries without a candidate omit both values. +### Debug evidence and runner records -Construct the pointers directly when mapping the corresponding records; use the -same simple convention across all dispositions. Do not build custom marshaling -or a generic optional-value framework. These values remain replacement spans, -not full resulting-segment text. Preserve redaction and the rest of the ledger. +Existing interaction diagnostics already retain the full decoded response and final +content. Add the reconciliation result to the response diagnostic payload for a +successfully decoded validation response. Perform reconciliation before writing +that diagnostic; retain existing call-error diagnostics and diagnostic-write failure +handling. All reconciled decisions, including unavailable outcomes, receive the +returned response artifact path. This records unexpected indices and duplicates +without inventing candidate ledger entries or an additional debug file. -Keep existing operation accounting. A valid approved deletion is an ordinary -applied correction. A candidate that would blank its stage-input segment is -filtered with the existing reason and contributes to the proposal operation's -consolidated filtering recovery. Ordinary editorial/protected-term rejection does -not become a recovery. Multiple proposals can target the same segment; they are -validated against stage input and applied sequentially. A proposal applicable to -that input but made unsafe by an earlier accepted edit is skipped as a conflict, -with no added model error or filtering recovery. Keep the skipped evidence. The -runner's `attributeApplyFiltering` already makes this distinction: verify and reuse -it rather than invent new classification machinery. Structural repairs required -for unrelated defects still count normally. +Carry `validators.DecisionDisposition` through `runner.ValidatorDecisionRecord` +and `runner.ValidatorRejectedChange` in an additive `disposition` field. Populate +it from each actual `Decision`, including partial decision evidence returned on +fatal paths. Keep the existing collection names and application flow; no parallel +unavailable-candidate collection is necessary. -## Stage 1 — Align Validator Identity and Confirm Domain Semantics +In `processreport`, add `disposition` to `LedgerValidatorDecisionRecord` and map +unavailable candidates to a top-level ledger disposition `unavailable`, instead +of `rejected`. Ordinary editorial/deterministic rejections remain `rejected`. +Use the explicit disposition, not a reason-code allowlist, so existing timeout, +provider-failure, and oversized-input outcomes retain the same distinction too. +Update affected test fixtures to supply dispositions where they represent real +validator outputs. Do not invent a fatal check or compatibility subsystem for a +zero-value disposition in handcrafted fixtures. + +Retain candidate identity, literal proposed replacements (including explicit empty +strings), and omission of applied replacements for unapplied candidates. Existing +fatal/interrupted/module-failure handling and debug-off behavior remain. Supplied +model decisions remain visible in interaction evidence even when reconciliation +cannot accept them as authoritative verdicts. + +## Stage 1 — Reconcile Model Decisions and Preserve Valid Work **Work** -- Rename the validator package/type/key described above. Update `New()` wiring, - `internal/validators/registry.go`, `chains.go`, metadata, framework validators, - consumers, and tests in one coherent change. Use graph discovery for symbols and - string searches for serialized keys. Do not globally rewrite historical docs. -- Preserve existing `CorrectionProposal.Validate`, `PreviewProposalForSegment`, - `ApplyProposals`, and deterministic validation behavior wherever already correct. - Change original-span trimmed-emptiness checks as specified above, and find any - equivalent downstream restrictions. Keep empty-original rejection and - whitespace-aware resulting-segment rejection distinct. -- Extend the existing proposal and validator tests only for missing meaningful - coverage: empty replacement leaving a nonblank segment, whitespace-only resulting - text rejection, whitespace-only originals, literal spacing, and apply-time - reevaluation after an earlier edit. Cover `"hello world"` with `" " -> " "`, - an empty original, and two separate matching whitespace runs under `require_unique`. - Assert module-policy ambiguity handling without imposing uniqueness on glossary. - For apply-time reevaluation, use segment `"a b"` and ordered deletions `"a" -> ""`, - `"b" -> ""`: each is previewable initially, but after the first edit the second - must be skipped as `empty_resulting_segment`. The surviving text is `" b"`; - no automatic trim or segment deletion is expected. At the runner/outcome boundary, - verify this initially applicable second proposal produces skipped debug evidence - but no model error or filtering recovery. Reuse/extend existing conflict tests; - a second full CLI scenario is unnecessary for this attribution check. -- Ensure protected-term validation still rejects removal of a protected term with - an empty replacement. Prefer one case in its existing test table. Retain existing - metadata-preservation tests rather than duplicating all metadata assertions. -- Update `docs/internal/validators.md` for the renamed implemented validator and - explain its resulting-segment responsibility; do not yet claim schema alignment. +1. Read the validator models, `llm_validators.go`, existing partial-batch tests, + `outcomes/outcomes.go`, and `runner.validateSectionCandidates`. +2. Implement the reconciliation result/helper and candidate reason codes selected + above. Replace the whole-batch mapping-error branch with outcome construction + and correct batch accounting. Add the decision-specific recovery constant. +3. Move successful-response diagnostic assembly after reconciliation and include + its result. Preserve raw/decoded response evidence and diagnostic paths; keep + recoverable call errors, fatal errors, and cancellation on their existing paths. +4. Keep `EnforceDecisionCardinality` and `EnforcePartialDecisionCardinality` strict. + Ensure final validator decisions are complete and unique internally. The runner + should continue passing only approved candidates onward without treating an + operation's failure metadata as a reason to discard valid decisions. +5. Replace obsolete batch-rejection expectations in `llm_validators_test.go` and + relevant runner tests. Preserve cardinality tests that exercise broken internal + validators. Update the repaired-but-domain-invalid tests so actual unavailable + cases remain errors with counts retained and no recovery record. + +**Focused verification** + +Use a small table at the validator boundary for mixed missing/extra indices, +conflicting and identical requested duplicates, extra-only responses, empty lists, +reordered responses, and a single invalid-confidence decoded test-double entry. +Include noncontiguous expected indices and valid negative decisions. Assert semantic +outcomes and affected indices, not helper layout or exact messages. Include one +controlled case combining successful structural/service recovery facts with ignored +extras and one failed reconciliation retaining counts without recovery reasons. + +At the runner boundary, use the real LLM-backed validator with a controlled client +and a later validator: prove a survivor reaches later required validation, a later +rejection prevents application, and unavailable candidates never reach that later +validator as approved. Use successful proposal operation facts where accounting is +asserted; do not check an empty operation list. Reuse existing fixtures/tests rather +than creating a second framework or duplicating every table case. + +Run `go test ./internal/framework/validators ./internal/framework/runner`. + +**End state:** Usable decisions survive model indexing defects, strict internal +cardinality remains satisfied, and operation facts explain partial failure or +successful extra filtering. No extra model generation is introduced. + +## Stage 2 — Carry Dispositions Through Debug and Report Partial Outcomes + +**Work** + +1. Add and populate disposition fields through the runner and ledger as selected + above. Map unavailable candidates distinctly and preserve empty replacement + serialization. Check both normal and fatal partial-evidence paths. +2. Update `processreport` messages and review `Build`, `mapFailure`, and `mapRecovery`. + Reuse their existing aggregation; change only assumptions that would misreport + the new operation facts. Core YAML structures and Markdown table layout remain. +3. Extend existing report/ledger tests for an operation with surviving work plus + unavailable indices, extra-only recovery, and a mixed defect operation. Verify + error indices exclude survivors, extra filtering does not count as proposal + filtering, and failed operations do not also generate recovery records. +4. Verify debug ledger entries distinguish genuine rejection from unavailability, + and interaction diagnostics preserve ignored entries and reconciliation reasons. + Unknown response indices must not create fake ledger candidates. Retain existing + redaction and optional-debug-failure tests rather than reproducing them all. +5. Update `docs/integrations/output-files.md` for the new recovery reason and debug + disposition, and `docs/internal/diagnostics-reporting.md` for the mapping changes. + Explain that validation failure indices identify unavailable candidates, not + necessarily every candidate submitted to that batch. **Validation** -Run `go test ./internal/framework/proposals ./internal/framework/validators ./internal/validators/... ./internal/framework/runner ./internal/framework/processreport`. -Existing registry/chain tests must use the new identity and verify the required -validation and execution class. Update incidental order assertions if a justified -ordering change preserves those safeguards. -Inspect remaining old-name references, retaining only deliberately historical ones. +Run `go test ./internal/framework/runner ./internal/framework/processreport ./internal/core/reporting`. +Use semantic YAML/summary assertions; no whole-report prose golden, exact message +snapshot, new schema version, or new rendering mode is needed. -**End state:** The accurately named domain validator and application preserve the -agreed behavior. No schema or retry workaround has been introduced. +**End state:** Reports and debug evidence accurately describe partial validation +success and handled extras through existing reporting mechanisms. -**Completed 2026-09-25.** Renamed the validator, admitted nonempty whitespace -original spans, and verified literal spacing, blank-result protection, protected -terms, module-policy ambiguity, and apply-time conflict attribution. The specified -focused package tests passed. - -## Stage 2 — Align Response Schema, Prompts, and LLM Validation Inputs +## Stage 3 — Verify the Assembled Workflow and Finalize Documentation **Work** -- Apply the selected schema relaxation and selectively clarify prompts. Preserve - intended task scope, hardening, and response contracts while correcting conflicting - instructions wherever encountered within this feature. Review the - rendered definitions, not just YAML source, using existing PromptKit inspection - and prompt tests. -- In `internal/framework/responseschema/registry_test.go`, retain the check that - reported hashes equal hashes of the actual embedded bytes. Remove fixed digest - expectations that merely freeze asset content; do not replace them with a new - golden digest for this change. -- Extend the existing real-adapter tests in `internal/framework/llm/adapter_test.go` - with a bounded table for replacement values: empty string, spaces, ordinary text, - missing field, null, and a non-string value. Use the embedded schema and local - HTTP fixture. Give accepted cases a positive repair allowance and assert one - provider generation, zero repairs, and preservation of the actual string. For - invalid cases, use zero repair attempts to prove rejection without redundant - model calls. Include a whitespace-only original in an accepted case and confirm - an empty original is structurally invalid. Existing tests already own successful - repair/exhaustion behavior. -- Extend the validation-request tests in - `internal/framework/validators/llm_validators_test.go` to marshal a real request - for deleting `"uh "` from `"uh hello"`. Assert that JSON contains the explicit - empty replacement plus full before/after text, correct identity, and no immediate - rejection. Include a whitespace-only-original spacing correction in the request - coverage and verify literal preservation. A field-presence assertion is required; - decoding into a plain string alone cannot distinguish omitted from empty. -- Update canonical prompt/module guidance for the implemented semantics. Do not - add exact-prose prompt snapshots or assertions for every new sentence; existing - parsing/rendering/schema-association tests plus editorial review are sufficient. +1. Extend the existing local-provider subprocess fixtures in + `cmd/audita/main_integration_test.go`. Add a representative mixed-decision run + with an applied survivor, a legitimate rejection, a missing requested decision, + a duplicate requested decision, and an extra index. Return responses by the + requested correction identities; do not freeze incidental validator call order. + Ensure the survivor passes all remaining required validation. Assert successful + CLI completion, correct transcript, one work-unit error with only unavailable + indices, no reconciliation recovery for the failed operation, and debug + evidence distinguishing rejection/unavailability without unknown candidates. +2. Add or adapt a compact extra-only case proving usable work survives and one + `validation_decisions_filtered` recovery appears. Verify YAML, Markdown, and CLI + agree on totals using semantic assertions. Keep proposal-filtering counts separate. + Existing reporting-disable/debug selection tests remain the owners of those options. +3. In these fixtures give PromptKit its normal configured structural repair allowance, + but supply schema-valid responses. Verify reconciliation adds no generations or + structural repairs using fixture requests/execution facts, without encoding today's + total validator count as a permanent contract. Keep structural-malformation tests + and recoverable batch-failure tests; do not relax the schema to make fixtures pass. +4. Update `docs/internal/validators.md` and any affected canonical descriptions found + by searching for whole-batch rejection claims. Explain tolerant external response + reconciliation versus strict internal cardinality. The structural-validation + boundary remains explicit, and guided recovery remains deferred in `future.md`. +5. Mark the feature roadmap implemented only after the assembled checks pass, and + record actual completion evidence in this plan. Repair the two span-deletion + roadmap links to the old plan with the historical URL + `https://gitea.maximumdirect.net/eric/audita/src/commit/60e2fe7/docs/roadmap/implementation.md`. + Preserve historical release notes and unrelated roadmaps. **Validation** -Run `go test ./internal/framework/responseschema ./internal/prompts ./internal/framework/llm ./internal/framework/proposal_generation ./internal/framework/validators ./internal/modules/...`. -Confirm missing/non-string replacements still fail structurally and domain-invalid -but schema-valid proposals still reach ordinary filtering without an outer retry. +Run `go test ./cmd/audita` while developing the fixture, then `go test ./...` once +for the completed implementation. Run `git diff --check` and verify affected +Markdown links. Do not add live-provider checks, a platform matrix, repeated full +suite runs, or release gates solely for this feature. No code is implemented by +this planning pass; these commands are instructions for the implementation sprint. -**End state:** A literal empty replacement can cross the real library/schema -boundary, and the model instructions and validator inputs describe its meaning. - -**Completed 2026-09-25.** Relaxed only the replacement length constraint, -clarified the relevant proposal and active validation prompts, and verified -rendering, real-adapter acceptance/rejection, repair counts, and serialized -before/after validator payloads. The specified focused package tests passed. - -## Stage 3 — Preserve Empty Replacements in Debug Evidence - -**Work** - -- Implement the pointer representation selected above in - `internal/framework/processreport/correction_ledger.go`, updating every mapping - branch and affected test consumer. Leave other debug structures unchanged where - their string fields already serialize empty values explicitly. -- Extend `correction_ledger_test.go` with a small serialized-disposition table: - applied deletion, rejected deletion, skipped deletion, interrupted deletion, - and a module failure without a candidate. Decode to a map or raw JSON fields to - check presence, `""` values, and absence of the applied value on unapplied entries. - Preserve existing decision classification/order tests using the renamed key. -- Update `docs/integrations/output-files.md` to explain the two debug replacement - fields and the deliberate validator identity change. Keep ordinary schemas and - report counts unchanged. Update the diagnostics/reporting internal reference - for the implemented mapping, linking to the public artifact contract. - -**Validation** - -Run `go test ./internal/framework/processreport ./internal/core/diagnostics ./internal/cli ./cmd/audita`. -Tests must inspect serialized output, not just non-nil in-memory values. Retain -existing configured-secret redaction and debug-off persistence coverage. - -**End state:** Debug evidence distinguishes a proposed/applied deletion from an -absent proposal or an edit that was never applied. - -**Completed 2026-09-25.** Candidate-backed ledger entries now preserve an -explicit empty proposed replacement, and applied entries preserve an explicit -empty applied replacement. Serialized disposition coverage and the specified -focused package tests passed; the artifact contract documents the fields and -validator identity change. - -## Stage 4 — Verify the Integrated Contract and Finish Documentation - -**Work** - -Extend `cmd/audita/main_integration_test.go` with representative offline subprocess -cases using its controlled HTTP provider, real configuration/profiles, embedded -schemas, built-in module/validator chains, CLI, and output/report/debug writers. -Keep fixture additions local and small; do not build a new integration framework. -Use explicit fixture sizing so relevant validator candidates fit one batch, and -use stable response indices. Do not rely on production chunk/token defaults or -on concurrent request order to choose responses. - -Cover these distinct outcomes: - -- **Clean deletion plus another correction:** A spoken-word stage proposes - `"uh " -> ""` in `"uh hello"` and an independent repetition collapse. Controlled - validators approve both. Assert corrected transcript, two applied corrections, - zero model errors/recoveries, zero structural repairs with a positive repair - allowance, and explicit empty proposed/applied fields in the debug ledger. -- **Whitespace-only span:** A grammar stage reduces one uniquely occurring run - of two spaces to one. Controlled validators approve it; assert the expected - text and no error/recovery solely due to whitespace. This proves schema, domain - validation, and application agree through the actual CLI. Use a compact fixture - without adding a matrix of whitespace characters or lengths. -- **Partial success with filtering:** Add a proposal to delete all text from a - separate filler-only segment, preserving that separate segment through fixture - normalization (use sufficient time separation or explicit normalization settings). - Assert that this candidate is filtered with `empty_resulting_segment`, the other - corrections still apply, the blanked segment is instead retained unchanged, and - the proposal response contributes one consolidated filtering recovery and no - work-unit error. The renamed deterministic validator must appear in evidence. -- **Ordinary negative decision:** Return a schema-valid deletion proposal, then a - valid negative decision from meaning-reversal review after earlier checks approve - it. Assert unchanged affected text, no applied deletion, no error/recovery solely - from the rejection, and a rejected debug entry. The controlled decision proves - the chain respects a negative decision, not that an arbitrary real model will - make the same editorial judgment. - -Reuse existing YAML/Markdown/CLI count assertions rather than snapshotting entire -reports. Verify that the clean case's report totals agree with the transcript and -ledger. Compare resulting metadata to the normalized input, since initial -normalization may legitimately merge segments or reassign IDs. Confirm retained -section membership/order in the existing runner boundary tests; add a targeted -assertion only if that property is not already covered for the exercised path. -Do not repeat all cases for all modules, encoders, debug/report combinations, or -platforms: those unchanged mechanisms already have coverage. - -Finish the documentation and status work: - -- Review current module, validator, prompt, diagnostics, and output references for - consistency. Add a concise note to `docs/migrations/v2.md` describing the renamed - validator identity and explicit empty debug replacement fields; no alias or - migration tool is required. No release note/version is selected in this sprint. -- Review links from other roadmaps to `implementation.md`. References to completed - resilience stages should point to their historical plan at the recorded revision - or current canonical behavior, not silently identify this plan as that work. - Update those references only as needed; do not rewrite historical findings. -- Mark this plan and `span-deletion.md` implemented only after the criteria pass. - Keep the baseline description historical and retain future.md's exploratory - whole-segment-removal status. Record concise evidence under completed stages. - -**Final validation** - -Run `go test ./...` once after the integrated changes. Run `git diff --check` and -validate affected documentation links/examples. Focused stage checks need not be -repeated solely as ceremony. Additional race/platform/live-provider checks are -not automatic gates for this feature; run additional checks only for a concrete -risk introduced by actual changes. This is not a release invocation. - -**End state:** The feature roadmap's acceptance criteria are evidenced by real -schema/adapter tests, domain/application tests, serialized debug tests, and -representative complete CLI workflows. The workspace is committed and the docs -describe implemented behavior without claiming whole-segment removal. - -**Completed 2026-09-25.** Four offline CLI workflows verified clean deletion, -literal spacing, partial success with blank-result filtering, and an ordinary -negative meaning-review decision. Metadata was checked against normalized -input; the existing runner test covers fixed section membership and order. -`go test ./...` passed. Documentation references, affected links, and whitespace -were checked. Whole-segment removal remains deferred. +**End state:** The accepted partial-success behavior is exercised through the real +CLI/library boundary, canonical documentation matches it, and coherent changes +are committed for subsequent release consideration. ## Open Questions -None requiring a product decision remain. Validator names, schema/prompt identity, -debug representation, module scope, test boundaries, and deferred behavior are -selected above. Ordinary implementation details may be resolved idiomatically -within these decisions; a newly discovered conflict with the agreed contract -should be recorded rather than resolved by silently strengthening restrictions. +None required before implementation. The implementation decisions above preserve +the agreed product scope. General structural salvage and guided regeneration remain +separate future decisions, not unresolved prerequisites for this work.