Plan partial validator decision handling
This commit is contained in:
@@ -3,8 +3,9 @@
|
||||
**Last updated:** 2026-09-25
|
||||
|
||||
**Status:** Accepted feature direction; implementation not started. Intended for
|
||||
consideration before v2.0.0. This document defines the target behavior and work
|
||||
needed; a staged implementation plan will follow separately.
|
||||
consideration before v2.0.0. This document owns the target behavior and user intent.
|
||||
[implementation.md](implementation.md) owns concrete decisions, ordered work,
|
||||
and verification.
|
||||
|
||||
## Purpose and User Intent
|
||||
|
||||
@@ -109,43 +110,17 @@ entries as defective correction proposals. Concrete reason names and minimal dat
|
||||
model changes belong in the implementation plan. Keep YAML, Markdown, and CLI
|
||||
counts consistent, with the existing reporting-disable and independent debug options.
|
||||
|
||||
## Work Needed
|
||||
## Ownership and Target Architecture
|
||||
|
||||
### Decision Mapping and Validator Integration
|
||||
Audita's validator layer reconciles structurally valid model responses into
|
||||
complete internal candidate outcomes. The runner preserves strict internal
|
||||
cardinality and advances only approved candidates. Reporting consumes operation
|
||||
facts without treating partial validation failure as loss of every decision.
|
||||
Debug evidence distinguishes unavailable validation from editorial rejection.
|
||||
|
||||
Replace the fail-fast model-response mapper with reconciliation that preserves
|
||||
unambiguous decisions and explicitly represents unavailable candidates and ignored
|
||||
entries. Update the LLM-backed validator to return those outcomes and accurate
|
||||
operation facts instead of rejecting a whole usable batch. Keep domain ownership
|
||||
in Audita and reuse the existing unavailable-candidate path.
|
||||
|
||||
Review the confidence check currently present in the mapper: structurally invalid
|
||||
confidence is already handled by the schema boundary. Do not introduce a second
|
||||
whole-batch failure path for an individual defective entry that reaches domain
|
||||
mapping. Preserve valid negative decisions and remaining validator execution.
|
||||
|
||||
### Reporting and Debug Evidence
|
||||
|
||||
Adapt operation mapping and presentation where needed to express partial batch
|
||||
failure and successful extra-entry filtering. Review the correction ledger so it
|
||||
records unavailable outcomes distinctly from actual editorial decisions. Preserve
|
||||
existing report grouping and avoid a new reporting subsystem or per-entry model
|
||||
operation. Add only the attribution needed to explain these outcomes.
|
||||
|
||||
### Regression Coverage and Documentation
|
||||
|
||||
Replace tests that enforce whole-batch rejection with focused tests of partial
|
||||
success. Use mixed responses to prove valid approvals and rejections survive
|
||||
alongside missing, duplicate, and extraneous indices. Cover conflicting duplicates,
|
||||
empty decision lists, and response-order independence without building an exhaustive
|
||||
cross-product suite.
|
||||
|
||||
Include representative assembled-path coverage proving that survivors complete
|
||||
all remaining validators, unavailable candidates never apply, and reporting reflects
|
||||
the correct errors/recoveries and applied corrections. Retain tests for strict
|
||||
internal cardinality and genuine operation-level failures. Update canonical
|
||||
validator and reporting documentation when implementation lands; do not describe
|
||||
this roadmap's target as currently implemented elsewhere.
|
||||
The schema/transport boundary remains with PromptKit; reconciliation needs no
|
||||
additional model call. Concrete helper structures, reason names, mapping changes,
|
||||
and verification steps belong in the [implementation plan](implementation.md).
|
||||
|
||||
## Scope Boundaries
|
||||
|
||||
@@ -184,7 +159,5 @@ are introduced. This work does not itself authorize tagging or publishing v2.0.0
|
||||
|
||||
## Open Questions
|
||||
|
||||
No unresolved product-policy questions block the staged implementation plan.
|
||||
That plan should select the smallest reporting/model changes and concrete reason
|
||||
codes needed to express the behavior above. Guided recovery and structural salvage
|
||||
remain separate future decisions.
|
||||
No unresolved product-policy questions block implementation. Guided recovery and
|
||||
structural salvage remain separate future decisions.
|
||||
|
||||
@@ -1,368 +1,268 @@
|
||||
# Implementation Plan: Consistent Span Deletion
|
||||
# Implementation Plan: Partial Validator Decision Sets
|
||||
|
||||
**Last updated:** 2026-09-25
|
||||
|
||||
**Status:** Implemented; all stages complete.
|
||||
**Source baseline:** `a83a518` (feature roadmap); application baseline `0b6cd98`.
|
||||
**Status:** Ready for implementation; no stages implemented by this planning pass.
|
||||
**Source baseline:** `60e2fe7`.
|
||||
**Audience:** gpt-5.6-terra coding agent.
|
||||
|
||||
## Scope and Execution Rules
|
||||
## Scope and Execution
|
||||
|
||||
Implement [Consistent Span Deletion](span-deletion.md). That document owns the
|
||||
product intent, target semantics, and acceptance criteria. This document owns the
|
||||
concrete decisions, ordered work, and validation. Execute Stage 1 through Stage 4
|
||||
in order. Record actual completion and checks beneath each stage when implemented.
|
||||
The completion evidence below records the implemented behavior; the source
|
||||
baseline remains historical.
|
||||
Implement [partial success for validator decision sets](decision-set-reconciliation.md).
|
||||
That roadmap owns product intent and target semantics; this document selects the
|
||||
implementation and verification work. Follow Stage 1 through Stage 3 in order and
|
||||
record completion evidence beneath each stage. Each stage is a focused agent task.
|
||||
|
||||
Read [development guidance](../development.md) and all [agent](../policy/agents.md),
|
||||
Read [development guidance](../development.md) and all policies under
|
||||
[docs/policy](../policy/): [agents](../policy/agents.md),
|
||||
[architecture](../policy/architecture.md), [documentation](../policy/documentation.md),
|
||||
[testing](../policy/testing.md), and [release](../policy/release.md) policies.
|
||||
Use graph-first code discovery. Inspect the current workspace before editing;
|
||||
retain unrelated user work. Commit coherent completed units with plain English
|
||||
messages, following the agent policy. Do not add permission rounds between stages.
|
||||
[testing](../policy/testing.md), and [release](../policy/release.md). Use graph-first
|
||||
code discovery, preserve unrelated work, and commit completed coherent units with
|
||||
plain English messages. Do not introduce approval rounds between stages.
|
||||
|
||||
Aim directly at the final state. Do not add temporary compatibility aliases,
|
||||
feature flags, parallel implementations, or scaffolding for intermediate commits.
|
||||
Preserve module scope, literal matching, replacement policies, fixed chunks,
|
||||
required validation, failure isolation,
|
||||
reporting/debug independence, and existing recovery budgets. Do not implement any
|
||||
of the [deferred features](future.md), automatic whitespace cleanup, a new deletion
|
||||
operation type, segment tombstones, or a second normalization pass.
|
||||
Aim directly at the final implementation. No temporary modes, compatibility shims,
|
||||
new configuration, extra model calls, or upstream dependency work is required.
|
||||
Keep PromptKit structural validation/repair and the runner's internal cardinality
|
||||
checks. Implement the agreed outcomes, not extra restrictions suggested by existing
|
||||
tests or incidental internal structure. Internal helper signatures and layout can
|
||||
be adjusted idiomatically while preserving the decisions below.
|
||||
|
||||
Preserve intended safeguards and outcomes, not incidental implementation choices.
|
||||
Validator ordering, internal structure, and payload construction may change where
|
||||
needed to satisfy the feature contract. Current code and tests are evidence to
|
||||
review, not authority to retain a conflicting restriction. Prefer reusing correct
|
||||
mechanisms, but do not freeze them merely because they exist.
|
||||
|
||||
This replaces the completed resilience implementation plan, which remains in Git
|
||||
history at `a83a518:docs/roadmap/implementation.md`. Historical resilience decisions
|
||||
and their remediation are not instructions to rebuild or reverse that work.
|
||||
During final documentation updates, repair references that describe this filename
|
||||
as the resilience plan; use current canonical references or the historical revision.
|
||||
No release, version tag, push, upstream change, or new dependency is part of this
|
||||
implementation sprint.
|
||||
This replaces the completed span-deletion plan, available in Git at
|
||||
`60e2fe7:docs/roadmap/implementation.md`. It does not reopen that feature. The final
|
||||
stage includes correcting historical links that otherwise point to this new plan.
|
||||
No release preparation, tag, push, CI wait, or paid-provider evaluation is included.
|
||||
|
||||
## Selected Implementation Decisions
|
||||
|
||||
### Schema and library boundary
|
||||
### Reconciliation boundary
|
||||
|
||||
Remove `minLength: 1` from the `corrected_text` property of
|
||||
`internal/framework/responseschema/assets/correction_set.v1.json`. Keep the required
|
||||
field, string type, and all unrelated schema constraints. Empty and whitespace-only
|
||||
strings are structurally valid; missing, null, and non-string replacements are not.
|
||||
Keep `original_text` required, string-valued, and at least one character long.
|
||||
Whitespace-only original spans are valid at the structural boundary and must not
|
||||
be rejected solely on that basis by domain checks. Replace trimmed-emptiness checks
|
||||
for original spans with literal empty-string checks in `CorrectionProposal.Validate`
|
||||
and `ProposalShapeValidator.Validate`, and review other consumers for the same
|
||||
restriction. Do not trim either span in transit. Resulting-segment checks still use
|
||||
whitespace-aware emptiness. Preserve module matching policies: `require_unique`
|
||||
for grammar, spoken-word, and homophones; `replace_all` for glossary.
|
||||
Keep reconciliation in `internal/framework/validators`, after successful structured
|
||||
completion and before converting the batch into runner-facing results. Replace
|
||||
`mapLLMResponseToDecisions` with a small pure reconciliation helper; use an ordinary
|
||||
private result struct containing:
|
||||
|
||||
Keep `CorrectionSetKey`, the schema asset path, ID/name/version, and existing prompt
|
||||
IDs/`v1` selections. This widens an internal model-response constraint without
|
||||
changing the correction object shape or creating a second contract to maintain.
|
||||
The schema hash and PromptKit's rendered/prepared hashes reflect changed content;
|
||||
no hard-coded hash is a compatibility promise. Ordinary report schema versions,
|
||||
transcript encoders, configuration, and CLI selections do not change.
|
||||
- `Decisions []Decision`: one outcome per requested item, in request order;
|
||||
- `UnavailableIndices []int`: requested indices without a usable verdict;
|
||||
- `IgnoredIndices []int`: unexpected indices encountered in response order, including
|
||||
repeated unexpected entries, for debug attribution.
|
||||
|
||||
PromptKit continues to validate the whole response and own structural repairs.
|
||||
Do not introduce partial JSON salvage or Audita-side regeneration. Once the schema
|
||||
accepts empty replacements, normal per-candidate domain filtering handles proposals
|
||||
that would blank their stage-input segment.
|
||||
No generic reconciliation framework or new shared abstraction is needed. Requested
|
||||
identities come from `batch.Items`, not transcript segment IDs, the whole section,
|
||||
or any other batch. Expected input identities remain an Audita internal contract.
|
||||
|
||||
### Validator identity
|
||||
First collect response entries by requested index and count occurrences, separating
|
||||
unexpected indices. Then walk requested items once to construct outcomes. This
|
||||
avoids accepting an early duplicate before seeing a conflicting later entry.
|
||||
|
||||
Rename the existing validator consistently:
|
||||
Use these candidate-level reason codes in `validators/models.go`:
|
||||
|
||||
| Current | Target |
|
||||
| Condition | Internal outcome/reason |
|
||||
| --- | --- |
|
||||
| `non_empty_corrected_text` key and package directory | `non_empty_resulting_segment` |
|
||||
| `KeyNonEmptyCorrectedText` in registry and metadata | `KeyNonEmptyResultingSegment` |
|
||||
| `NonEmptyCorrectionValidator` framework type | `NonEmptyResultingSegmentValidator` |
|
||||
| Missing requested entry | `DispositionUnavailable`, `missing_validation_decision` |
|
||||
| More than one entry for a requested index | `DispositionUnavailable`, `duplicate_validation_decision` |
|
||||
| Single requested entry failing the existing confidence range check | `DispositionUnavailable`, `invalid_validation_decision` |
|
||||
| Single usable entry | Existing approved or `llm_rejected` decision with its model reason. |
|
||||
|
||||
Retain the `empty_resulting_segment` rejection/skip reason, deterministic execution
|
||||
class, and the resulting-segment safeguard. Reuse preview-based validation where
|
||||
correct; neither its exact implementation nor chain position is frozen. Do not
|
||||
retain an old-key alias.
|
||||
This is an intentional validator identity change in diagnostic/decision records;
|
||||
document it, including any affected ordinary filtered-proposal metadata. It is not
|
||||
a new operator-selectable setting. Historical release notes remain untouched.
|
||||
Unavailable outcomes have `Approved: false` and fixed safe explanatory messages.
|
||||
Duplicate classification takes precedence over inspecting individual duplicate
|
||||
values; identical duplicates follow the same rule. Unexpected entries are ignored
|
||||
before inspecting confidence. Retain the existing confidence check as a local
|
||||
check for decoded callers/test doubles, scoped to that candidate; do not broaden
|
||||
this into structural salvage, confidence thresholds, or new semantic validation.
|
||||
The production JSON Schema still rejects out-of-range confidence before mapping.
|
||||
|
||||
### Prompt and payload semantics
|
||||
An empty response list makes every requested item unavailable. Response ordering
|
||||
must not change verdicts or which candidate is affected. Preserve the existing
|
||||
handling of valid negative decisions, including reason text. No candidate may
|
||||
advance merely because another candidate was approved.
|
||||
|
||||
Review all four proposal definitions and the active validator definitions
|
||||
(`editorial_review`, `meaning_reversal_review`, `spoken_form_plausibility`) for
|
||||
consistency. Edit only definitions needing clarification. Review the registered
|
||||
but inactive `grammar_review` and `spoken_word_review` definitions too; unchanged,
|
||||
consistent definitions need no edits. Do not activate additional validators as
|
||||
part of this work.
|
||||
### Operation accounting and ordinary reports
|
||||
|
||||
Where helpful, explain that an empty replacement deletes the identified span,
|
||||
whitespace-only spans can express spacing corrections, and the resulting segment
|
||||
must remain nonblank. Do not instruct the model to fabricate padding or content.
|
||||
A spoken-word example could delete `"uh "` from `"uh hello"`; a grammar example
|
||||
could reduce two spaces to one. Examples and wording are illustrative, not frozen
|
||||
requirements or promises of automatic approval. Explain surrounding whitespace
|
||||
and punctuation only to the extent needed to produce the intended literal result.
|
||||
Qualify uniqueness by module: glossary uses `replace_all`, while the other three
|
||||
modules use `require_unique`.
|
||||
Each actual validation batch remains one `outcomes.Operation`. Use existing
|
||||
`ReasonValidationDecisionsInvalid` (`validation_decisions_invalid`) for a batch
|
||||
with unavailable requested candidates. Set `Succeeded: false`, set `ProposalIndices`
|
||||
to only those unavailable indices, retain execution counts, and leave recovery
|
||||
reasons empty. Still return usable decisions from that batch alongside explicit
|
||||
unavailable outcomes, with no Go error for the reconciled model defect.
|
||||
|
||||
Keep module responsibilities intact. Meaning-reversal review remains a narrow
|
||||
check for antonyms/reversals, not a new general semantic-equivalence validator.
|
||||
Spoken-form plausibility retains its phonetic/mistranscription remit. Empty syntax
|
||||
alone must not determine approval or rejection.
|
||||
If no requested candidates are unavailable, set `Succeeded: true`. Add
|
||||
`outcomes.RecoveryValidationDecisionsFiltered` with serialized value
|
||||
`validation_decisions_filtered` when extras were ignored. Merge it with actual
|
||||
`output_repaired` and `service_recovered` reasons in the same recovery operation.
|
||||
Valid negative decisions do not prevent operation success. A clean response creates
|
||||
no new recovery reason.
|
||||
|
||||
`BuildLLMValidationRequest` already supplies explicit `corrected_text` and full
|
||||
before/after segment text. Start with that representation and verify that empty
|
||||
and whitespace-only spans survive it faithfully. Adjust payload construction if
|
||||
needed for consistency; do not treat its current structure as an independent
|
||||
requirement. No new deletion flag or synthetic sentinel is needed merely to
|
||||
represent an empty string.
|
||||
When unavailable candidates and extras coexist, record the failed operation only;
|
||||
keep ignored-entry evidence in debug. Do not create both an error and a recovery
|
||||
for that operation. Successful survivors must continue even though the operation
|
||||
itself records a failure. Retain actual execution counts on all paths.
|
||||
|
||||
### Debug representation and outcomes
|
||||
Reuse existing ordinary YAML structures and version 1. The only new public report
|
||||
value is the decision-specific recovery reason; affected error indices already fit
|
||||
`Failure.ProposalIndices`. Do not add ignored entries to `Operation.Filtered` or
|
||||
`filtered_proposals`, which describe correction proposals, and do not increment
|
||||
`proposal_responses_with_filtering`. The recovery reason and existing operation
|
||||
identity suffice for ordinary reporting; detailed extra indices belong in debug.
|
||||
|
||||
`CorrectionLedgerEntry` currently uses strings with `omitempty` for both replacement
|
||||
fields, losing explicit empty values. Change `ProposedCorrectedText` and
|
||||
`AppliedCorrectedText` to `*string` with their existing JSON names and `omitempty`:
|
||||
Update `processreport.safeMessage` for `validation_decisions_invalid` to explain
|
||||
that some required decisions were unavailable without claiming all batch decisions
|
||||
were discarded. Generalize the recovery message to include response filtering,
|
||||
rather than only proposal filtering. Exact prose is not a test contract. Preserve
|
||||
work-unit error grouping, operation counts, and existing YAML/Markdown/CLI totals.
|
||||
|
||||
- Every candidate-backed entry has a non-nil proposed value, including `""`.
|
||||
- Applied entries also have a non-nil applied value, including `""`.
|
||||
- Rejected, skipped, and interrupted entries omit the applied value.
|
||||
- Module-level failure entries without a candidate omit both values.
|
||||
### Debug evidence and runner records
|
||||
|
||||
Construct the pointers directly when mapping the corresponding records; use the
|
||||
same simple convention across all dispositions. Do not build custom marshaling
|
||||
or a generic optional-value framework. These values remain replacement spans,
|
||||
not full resulting-segment text. Preserve redaction and the rest of the ledger.
|
||||
Existing interaction diagnostics already retain the full decoded response and final
|
||||
content. Add the reconciliation result to the response diagnostic payload for a
|
||||
successfully decoded validation response. Perform reconciliation before writing
|
||||
that diagnostic; retain existing call-error diagnostics and diagnostic-write failure
|
||||
handling. All reconciled decisions, including unavailable outcomes, receive the
|
||||
returned response artifact path. This records unexpected indices and duplicates
|
||||
without inventing candidate ledger entries or an additional debug file.
|
||||
|
||||
Keep existing operation accounting. A valid approved deletion is an ordinary
|
||||
applied correction. A candidate that would blank its stage-input segment is
|
||||
filtered with the existing reason and contributes to the proposal operation's
|
||||
consolidated filtering recovery. Ordinary editorial/protected-term rejection does
|
||||
not become a recovery. Multiple proposals can target the same segment; they are
|
||||
validated against stage input and applied sequentially. A proposal applicable to
|
||||
that input but made unsafe by an earlier accepted edit is skipped as a conflict,
|
||||
with no added model error or filtering recovery. Keep the skipped evidence. The
|
||||
runner's `attributeApplyFiltering` already makes this distinction: verify and reuse
|
||||
it rather than invent new classification machinery. Structural repairs required
|
||||
for unrelated defects still count normally.
|
||||
Carry `validators.DecisionDisposition` through `runner.ValidatorDecisionRecord`
|
||||
and `runner.ValidatorRejectedChange` in an additive `disposition` field. Populate
|
||||
it from each actual `Decision`, including partial decision evidence returned on
|
||||
fatal paths. Keep the existing collection names and application flow; no parallel
|
||||
unavailable-candidate collection is necessary.
|
||||
|
||||
## Stage 1 — Align Validator Identity and Confirm Domain Semantics
|
||||
In `processreport`, add `disposition` to `LedgerValidatorDecisionRecord` and map
|
||||
unavailable candidates to a top-level ledger disposition `unavailable`, instead
|
||||
of `rejected`. Ordinary editorial/deterministic rejections remain `rejected`.
|
||||
Use the explicit disposition, not a reason-code allowlist, so existing timeout,
|
||||
provider-failure, and oversized-input outcomes retain the same distinction too.
|
||||
Update affected test fixtures to supply dispositions where they represent real
|
||||
validator outputs. Do not invent a fatal check or compatibility subsystem for a
|
||||
zero-value disposition in handcrafted fixtures.
|
||||
|
||||
Retain candidate identity, literal proposed replacements (including explicit empty
|
||||
strings), and omission of applied replacements for unapplied candidates. Existing
|
||||
fatal/interrupted/module-failure handling and debug-off behavior remain. Supplied
|
||||
model decisions remain visible in interaction evidence even when reconciliation
|
||||
cannot accept them as authoritative verdicts.
|
||||
|
||||
## Stage 1 — Reconcile Model Decisions and Preserve Valid Work
|
||||
|
||||
**Work**
|
||||
|
||||
- Rename the validator package/type/key described above. Update `New()` wiring,
|
||||
`internal/validators/registry.go`, `chains.go`, metadata, framework validators,
|
||||
consumers, and tests in one coherent change. Use graph discovery for symbols and
|
||||
string searches for serialized keys. Do not globally rewrite historical docs.
|
||||
- Preserve existing `CorrectionProposal.Validate`, `PreviewProposalForSegment`,
|
||||
`ApplyProposals`, and deterministic validation behavior wherever already correct.
|
||||
Change original-span trimmed-emptiness checks as specified above, and find any
|
||||
equivalent downstream restrictions. Keep empty-original rejection and
|
||||
whitespace-aware resulting-segment rejection distinct.
|
||||
- Extend the existing proposal and validator tests only for missing meaningful
|
||||
coverage: empty replacement leaving a nonblank segment, whitespace-only resulting
|
||||
text rejection, whitespace-only originals, literal spacing, and apply-time
|
||||
reevaluation after an earlier edit. Cover `"hello world"` with `" " -> " "`,
|
||||
an empty original, and two separate matching whitespace runs under `require_unique`.
|
||||
Assert module-policy ambiguity handling without imposing uniqueness on glossary.
|
||||
For apply-time reevaluation, use segment `"a b"` and ordered deletions `"a" -> ""`,
|
||||
`"b" -> ""`: each is previewable initially, but after the first edit the second
|
||||
must be skipped as `empty_resulting_segment`. The surviving text is `" b"`;
|
||||
no automatic trim or segment deletion is expected. At the runner/outcome boundary,
|
||||
verify this initially applicable second proposal produces skipped debug evidence
|
||||
but no model error or filtering recovery. Reuse/extend existing conflict tests;
|
||||
a second full CLI scenario is unnecessary for this attribution check.
|
||||
- Ensure protected-term validation still rejects removal of a protected term with
|
||||
an empty replacement. Prefer one case in its existing test table. Retain existing
|
||||
metadata-preservation tests rather than duplicating all metadata assertions.
|
||||
- Update `docs/internal/validators.md` for the renamed implemented validator and
|
||||
explain its resulting-segment responsibility; do not yet claim schema alignment.
|
||||
1. Read the validator models, `llm_validators.go`, existing partial-batch tests,
|
||||
`outcomes/outcomes.go`, and `runner.validateSectionCandidates`.
|
||||
2. Implement the reconciliation result/helper and candidate reason codes selected
|
||||
above. Replace the whole-batch mapping-error branch with outcome construction
|
||||
and correct batch accounting. Add the decision-specific recovery constant.
|
||||
3. Move successful-response diagnostic assembly after reconciliation and include
|
||||
its result. Preserve raw/decoded response evidence and diagnostic paths; keep
|
||||
recoverable call errors, fatal errors, and cancellation on their existing paths.
|
||||
4. Keep `EnforceDecisionCardinality` and `EnforcePartialDecisionCardinality` strict.
|
||||
Ensure final validator decisions are complete and unique internally. The runner
|
||||
should continue passing only approved candidates onward without treating an
|
||||
operation's failure metadata as a reason to discard valid decisions.
|
||||
5. Replace obsolete batch-rejection expectations in `llm_validators_test.go` and
|
||||
relevant runner tests. Preserve cardinality tests that exercise broken internal
|
||||
validators. Update the repaired-but-domain-invalid tests so actual unavailable
|
||||
cases remain errors with counts retained and no recovery record.
|
||||
|
||||
**Focused verification**
|
||||
|
||||
Use a small table at the validator boundary for mixed missing/extra indices,
|
||||
conflicting and identical requested duplicates, extra-only responses, empty lists,
|
||||
reordered responses, and a single invalid-confidence decoded test-double entry.
|
||||
Include noncontiguous expected indices and valid negative decisions. Assert semantic
|
||||
outcomes and affected indices, not helper layout or exact messages. Include one
|
||||
controlled case combining successful structural/service recovery facts with ignored
|
||||
extras and one failed reconciliation retaining counts without recovery reasons.
|
||||
|
||||
At the runner boundary, use the real LLM-backed validator with a controlled client
|
||||
and a later validator: prove a survivor reaches later required validation, a later
|
||||
rejection prevents application, and unavailable candidates never reach that later
|
||||
validator as approved. Use successful proposal operation facts where accounting is
|
||||
asserted; do not check an empty operation list. Reuse existing fixtures/tests rather
|
||||
than creating a second framework or duplicating every table case.
|
||||
|
||||
Run `go test ./internal/framework/validators ./internal/framework/runner`.
|
||||
|
||||
**End state:** Usable decisions survive model indexing defects, strict internal
|
||||
cardinality remains satisfied, and operation facts explain partial failure or
|
||||
successful extra filtering. No extra model generation is introduced.
|
||||
|
||||
## Stage 2 — Carry Dispositions Through Debug and Report Partial Outcomes
|
||||
|
||||
**Work**
|
||||
|
||||
1. Add and populate disposition fields through the runner and ledger as selected
|
||||
above. Map unavailable candidates distinctly and preserve empty replacement
|
||||
serialization. Check both normal and fatal partial-evidence paths.
|
||||
2. Update `processreport` messages and review `Build`, `mapFailure`, and `mapRecovery`.
|
||||
Reuse their existing aggregation; change only assumptions that would misreport
|
||||
the new operation facts. Core YAML structures and Markdown table layout remain.
|
||||
3. Extend existing report/ledger tests for an operation with surviving work plus
|
||||
unavailable indices, extra-only recovery, and a mixed defect operation. Verify
|
||||
error indices exclude survivors, extra filtering does not count as proposal
|
||||
filtering, and failed operations do not also generate recovery records.
|
||||
4. Verify debug ledger entries distinguish genuine rejection from unavailability,
|
||||
and interaction diagnostics preserve ignored entries and reconciliation reasons.
|
||||
Unknown response indices must not create fake ledger candidates. Retain existing
|
||||
redaction and optional-debug-failure tests rather than reproducing them all.
|
||||
5. Update `docs/integrations/output-files.md` for the new recovery reason and debug
|
||||
disposition, and `docs/internal/diagnostics-reporting.md` for the mapping changes.
|
||||
Explain that validation failure indices identify unavailable candidates, not
|
||||
necessarily every candidate submitted to that batch.
|
||||
|
||||
**Validation**
|
||||
|
||||
Run `go test ./internal/framework/proposals ./internal/framework/validators ./internal/validators/... ./internal/framework/runner ./internal/framework/processreport`.
|
||||
Existing registry/chain tests must use the new identity and verify the required
|
||||
validation and execution class. Update incidental order assertions if a justified
|
||||
ordering change preserves those safeguards.
|
||||
Inspect remaining old-name references, retaining only deliberately historical ones.
|
||||
Run `go test ./internal/framework/runner ./internal/framework/processreport ./internal/core/reporting`.
|
||||
Use semantic YAML/summary assertions; no whole-report prose golden, exact message
|
||||
snapshot, new schema version, or new rendering mode is needed.
|
||||
|
||||
**End state:** The accurately named domain validator and application preserve the
|
||||
agreed behavior. No schema or retry workaround has been introduced.
|
||||
**End state:** Reports and debug evidence accurately describe partial validation
|
||||
success and handled extras through existing reporting mechanisms.
|
||||
|
||||
**Completed 2026-09-25.** Renamed the validator, admitted nonempty whitespace
|
||||
original spans, and verified literal spacing, blank-result protection, protected
|
||||
terms, module-policy ambiguity, and apply-time conflict attribution. The specified
|
||||
focused package tests passed.
|
||||
|
||||
## Stage 2 — Align Response Schema, Prompts, and LLM Validation Inputs
|
||||
## Stage 3 — Verify the Assembled Workflow and Finalize Documentation
|
||||
|
||||
**Work**
|
||||
|
||||
- Apply the selected schema relaxation and selectively clarify prompts. Preserve
|
||||
intended task scope, hardening, and response contracts while correcting conflicting
|
||||
instructions wherever encountered within this feature. Review the
|
||||
rendered definitions, not just YAML source, using existing PromptKit inspection
|
||||
and prompt tests.
|
||||
- In `internal/framework/responseschema/registry_test.go`, retain the check that
|
||||
reported hashes equal hashes of the actual embedded bytes. Remove fixed digest
|
||||
expectations that merely freeze asset content; do not replace them with a new
|
||||
golden digest for this change.
|
||||
- Extend the existing real-adapter tests in `internal/framework/llm/adapter_test.go`
|
||||
with a bounded table for replacement values: empty string, spaces, ordinary text,
|
||||
missing field, null, and a non-string value. Use the embedded schema and local
|
||||
HTTP fixture. Give accepted cases a positive repair allowance and assert one
|
||||
provider generation, zero repairs, and preservation of the actual string. For
|
||||
invalid cases, use zero repair attempts to prove rejection without redundant
|
||||
model calls. Include a whitespace-only original in an accepted case and confirm
|
||||
an empty original is structurally invalid. Existing tests already own successful
|
||||
repair/exhaustion behavior.
|
||||
- Extend the validation-request tests in
|
||||
`internal/framework/validators/llm_validators_test.go` to marshal a real request
|
||||
for deleting `"uh "` from `"uh hello"`. Assert that JSON contains the explicit
|
||||
empty replacement plus full before/after text, correct identity, and no immediate
|
||||
rejection. Include a whitespace-only-original spacing correction in the request
|
||||
coverage and verify literal preservation. A field-presence assertion is required;
|
||||
decoding into a plain string alone cannot distinguish omitted from empty.
|
||||
- Update canonical prompt/module guidance for the implemented semantics. Do not
|
||||
add exact-prose prompt snapshots or assertions for every new sentence; existing
|
||||
parsing/rendering/schema-association tests plus editorial review are sufficient.
|
||||
1. Extend the existing local-provider subprocess fixtures in
|
||||
`cmd/audita/main_integration_test.go`. Add a representative mixed-decision run
|
||||
with an applied survivor, a legitimate rejection, a missing requested decision,
|
||||
a duplicate requested decision, and an extra index. Return responses by the
|
||||
requested correction identities; do not freeze incidental validator call order.
|
||||
Ensure the survivor passes all remaining required validation. Assert successful
|
||||
CLI completion, correct transcript, one work-unit error with only unavailable
|
||||
indices, no reconciliation recovery for the failed operation, and debug
|
||||
evidence distinguishing rejection/unavailability without unknown candidates.
|
||||
2. Add or adapt a compact extra-only case proving usable work survives and one
|
||||
`validation_decisions_filtered` recovery appears. Verify YAML, Markdown, and CLI
|
||||
agree on totals using semantic assertions. Keep proposal-filtering counts separate.
|
||||
Existing reporting-disable/debug selection tests remain the owners of those options.
|
||||
3. In these fixtures give PromptKit its normal configured structural repair allowance,
|
||||
but supply schema-valid responses. Verify reconciliation adds no generations or
|
||||
structural repairs using fixture requests/execution facts, without encoding today's
|
||||
total validator count as a permanent contract. Keep structural-malformation tests
|
||||
and recoverable batch-failure tests; do not relax the schema to make fixtures pass.
|
||||
4. Update `docs/internal/validators.md` and any affected canonical descriptions found
|
||||
by searching for whole-batch rejection claims. Explain tolerant external response
|
||||
reconciliation versus strict internal cardinality. The structural-validation
|
||||
boundary remains explicit, and guided recovery remains deferred in `future.md`.
|
||||
5. Mark the feature roadmap implemented only after the assembled checks pass, and
|
||||
record actual completion evidence in this plan. Repair the two span-deletion
|
||||
roadmap links to the old plan with the historical URL
|
||||
`https://gitea.maximumdirect.net/eric/audita/src/commit/60e2fe7/docs/roadmap/implementation.md`.
|
||||
Preserve historical release notes and unrelated roadmaps.
|
||||
|
||||
**Validation**
|
||||
|
||||
Run `go test ./internal/framework/responseschema ./internal/prompts ./internal/framework/llm ./internal/framework/proposal_generation ./internal/framework/validators ./internal/modules/...`.
|
||||
Confirm missing/non-string replacements still fail structurally and domain-invalid
|
||||
but schema-valid proposals still reach ordinary filtering without an outer retry.
|
||||
Run `go test ./cmd/audita` while developing the fixture, then `go test ./...` once
|
||||
for the completed implementation. Run `git diff --check` and verify affected
|
||||
Markdown links. Do not add live-provider checks, a platform matrix, repeated full
|
||||
suite runs, or release gates solely for this feature. No code is implemented by
|
||||
this planning pass; these commands are instructions for the implementation sprint.
|
||||
|
||||
**End state:** A literal empty replacement can cross the real library/schema
|
||||
boundary, and the model instructions and validator inputs describe its meaning.
|
||||
|
||||
**Completed 2026-09-25.** Relaxed only the replacement length constraint,
|
||||
clarified the relevant proposal and active validation prompts, and verified
|
||||
rendering, real-adapter acceptance/rejection, repair counts, and serialized
|
||||
before/after validator payloads. The specified focused package tests passed.
|
||||
|
||||
## Stage 3 — Preserve Empty Replacements in Debug Evidence
|
||||
|
||||
**Work**
|
||||
|
||||
- Implement the pointer representation selected above in
|
||||
`internal/framework/processreport/correction_ledger.go`, updating every mapping
|
||||
branch and affected test consumer. Leave other debug structures unchanged where
|
||||
their string fields already serialize empty values explicitly.
|
||||
- Extend `correction_ledger_test.go` with a small serialized-disposition table:
|
||||
applied deletion, rejected deletion, skipped deletion, interrupted deletion,
|
||||
and a module failure without a candidate. Decode to a map or raw JSON fields to
|
||||
check presence, `""` values, and absence of the applied value on unapplied entries.
|
||||
Preserve existing decision classification/order tests using the renamed key.
|
||||
- Update `docs/integrations/output-files.md` to explain the two debug replacement
|
||||
fields and the deliberate validator identity change. Keep ordinary schemas and
|
||||
report counts unchanged. Update the diagnostics/reporting internal reference
|
||||
for the implemented mapping, linking to the public artifact contract.
|
||||
|
||||
**Validation**
|
||||
|
||||
Run `go test ./internal/framework/processreport ./internal/core/diagnostics ./internal/cli ./cmd/audita`.
|
||||
Tests must inspect serialized output, not just non-nil in-memory values. Retain
|
||||
existing configured-secret redaction and debug-off persistence coverage.
|
||||
|
||||
**End state:** Debug evidence distinguishes a proposed/applied deletion from an
|
||||
absent proposal or an edit that was never applied.
|
||||
|
||||
**Completed 2026-09-25.** Candidate-backed ledger entries now preserve an
|
||||
explicit empty proposed replacement, and applied entries preserve an explicit
|
||||
empty applied replacement. Serialized disposition coverage and the specified
|
||||
focused package tests passed; the artifact contract documents the fields and
|
||||
validator identity change.
|
||||
|
||||
## Stage 4 — Verify the Integrated Contract and Finish Documentation
|
||||
|
||||
**Work**
|
||||
|
||||
Extend `cmd/audita/main_integration_test.go` with representative offline subprocess
|
||||
cases using its controlled HTTP provider, real configuration/profiles, embedded
|
||||
schemas, built-in module/validator chains, CLI, and output/report/debug writers.
|
||||
Keep fixture additions local and small; do not build a new integration framework.
|
||||
Use explicit fixture sizing so relevant validator candidates fit one batch, and
|
||||
use stable response indices. Do not rely on production chunk/token defaults or
|
||||
on concurrent request order to choose responses.
|
||||
|
||||
Cover these distinct outcomes:
|
||||
|
||||
- **Clean deletion plus another correction:** A spoken-word stage proposes
|
||||
`"uh " -> ""` in `"uh hello"` and an independent repetition collapse. Controlled
|
||||
validators approve both. Assert corrected transcript, two applied corrections,
|
||||
zero model errors/recoveries, zero structural repairs with a positive repair
|
||||
allowance, and explicit empty proposed/applied fields in the debug ledger.
|
||||
- **Whitespace-only span:** A grammar stage reduces one uniquely occurring run
|
||||
of two spaces to one. Controlled validators approve it; assert the expected
|
||||
text and no error/recovery solely due to whitespace. This proves schema, domain
|
||||
validation, and application agree through the actual CLI. Use a compact fixture
|
||||
without adding a matrix of whitespace characters or lengths.
|
||||
- **Partial success with filtering:** Add a proposal to delete all text from a
|
||||
separate filler-only segment, preserving that separate segment through fixture
|
||||
normalization (use sufficient time separation or explicit normalization settings).
|
||||
Assert that this candidate is filtered with `empty_resulting_segment`, the other
|
||||
corrections still apply, the blanked segment is instead retained unchanged, and
|
||||
the proposal response contributes one consolidated filtering recovery and no
|
||||
work-unit error. The renamed deterministic validator must appear in evidence.
|
||||
- **Ordinary negative decision:** Return a schema-valid deletion proposal, then a
|
||||
valid negative decision from meaning-reversal review after earlier checks approve
|
||||
it. Assert unchanged affected text, no applied deletion, no error/recovery solely
|
||||
from the rejection, and a rejected debug entry. The controlled decision proves
|
||||
the chain respects a negative decision, not that an arbitrary real model will
|
||||
make the same editorial judgment.
|
||||
|
||||
Reuse existing YAML/Markdown/CLI count assertions rather than snapshotting entire
|
||||
reports. Verify that the clean case's report totals agree with the transcript and
|
||||
ledger. Compare resulting metadata to the normalized input, since initial
|
||||
normalization may legitimately merge segments or reassign IDs. Confirm retained
|
||||
section membership/order in the existing runner boundary tests; add a targeted
|
||||
assertion only if that property is not already covered for the exercised path.
|
||||
Do not repeat all cases for all modules, encoders, debug/report combinations, or
|
||||
platforms: those unchanged mechanisms already have coverage.
|
||||
|
||||
Finish the documentation and status work:
|
||||
|
||||
- Review current module, validator, prompt, diagnostics, and output references for
|
||||
consistency. Add a concise note to `docs/migrations/v2.md` describing the renamed
|
||||
validator identity and explicit empty debug replacement fields; no alias or
|
||||
migration tool is required. No release note/version is selected in this sprint.
|
||||
- Review links from other roadmaps to `implementation.md`. References to completed
|
||||
resilience stages should point to their historical plan at the recorded revision
|
||||
or current canonical behavior, not silently identify this plan as that work.
|
||||
Update those references only as needed; do not rewrite historical findings.
|
||||
- Mark this plan and `span-deletion.md` implemented only after the criteria pass.
|
||||
Keep the baseline description historical and retain future.md's exploratory
|
||||
whole-segment-removal status. Record concise evidence under completed stages.
|
||||
|
||||
**Final validation**
|
||||
|
||||
Run `go test ./...` once after the integrated changes. Run `git diff --check` and
|
||||
validate affected documentation links/examples. Focused stage checks need not be
|
||||
repeated solely as ceremony. Additional race/platform/live-provider checks are
|
||||
not automatic gates for this feature; run additional checks only for a concrete
|
||||
risk introduced by actual changes. This is not a release invocation.
|
||||
|
||||
**End state:** The feature roadmap's acceptance criteria are evidenced by real
|
||||
schema/adapter tests, domain/application tests, serialized debug tests, and
|
||||
representative complete CLI workflows. The workspace is committed and the docs
|
||||
describe implemented behavior without claiming whole-segment removal.
|
||||
|
||||
**Completed 2026-09-25.** Four offline CLI workflows verified clean deletion,
|
||||
literal spacing, partial success with blank-result filtering, and an ordinary
|
||||
negative meaning-review decision. Metadata was checked against normalized
|
||||
input; the existing runner test covers fixed section membership and order.
|
||||
`go test ./...` passed. Documentation references, affected links, and whitespace
|
||||
were checked. Whole-segment removal remains deferred.
|
||||
**End state:** The accepted partial-success behavior is exercised through the real
|
||||
CLI/library boundary, canonical documentation matches it, and coherent changes
|
||||
are committed for subsequent release consideration.
|
||||
|
||||
## Open Questions
|
||||
|
||||
None requiring a product decision remain. Validator names, schema/prompt identity,
|
||||
debug representation, module scope, test boundaries, and deferred behavior are
|
||||
selected above. Ordinary implementation details may be resolved idiomatically
|
||||
within these decisions; a newly discovered conflict with the agreed contract
|
||||
should be recorded rather than resolved by silently strengthening restrictions.
|
||||
None required before implementation. The implementation decisions above preserve
|
||||
the agreed product scope. General structural salvage and guided regeneration remain
|
||||
separate future decisions, not unresolved prerequisites for this work.
|
||||
|
||||
Reference in New Issue
Block a user