Plan partial validator decision handling
This commit is contained in:
@@ -3,8 +3,9 @@
|
|||||||
**Last updated:** 2026-09-25
|
**Last updated:** 2026-09-25
|
||||||
|
|
||||||
**Status:** Accepted feature direction; implementation not started. Intended for
|
**Status:** Accepted feature direction; implementation not started. Intended for
|
||||||
consideration before v2.0.0. This document defines the target behavior and work
|
consideration before v2.0.0. This document owns the target behavior and user intent.
|
||||||
needed; a staged implementation plan will follow separately.
|
[implementation.md](implementation.md) owns concrete decisions, ordered work,
|
||||||
|
and verification.
|
||||||
|
|
||||||
## Purpose and User Intent
|
## Purpose and User Intent
|
||||||
|
|
||||||
@@ -109,43 +110,17 @@ entries as defective correction proposals. Concrete reason names and minimal dat
|
|||||||
model changes belong in the implementation plan. Keep YAML, Markdown, and CLI
|
model changes belong in the implementation plan. Keep YAML, Markdown, and CLI
|
||||||
counts consistent, with the existing reporting-disable and independent debug options.
|
counts consistent, with the existing reporting-disable and independent debug options.
|
||||||
|
|
||||||
## Work Needed
|
## Ownership and Target Architecture
|
||||||
|
|
||||||
### Decision Mapping and Validator Integration
|
Audita's validator layer reconciles structurally valid model responses into
|
||||||
|
complete internal candidate outcomes. The runner preserves strict internal
|
||||||
|
cardinality and advances only approved candidates. Reporting consumes operation
|
||||||
|
facts without treating partial validation failure as loss of every decision.
|
||||||
|
Debug evidence distinguishes unavailable validation from editorial rejection.
|
||||||
|
|
||||||
Replace the fail-fast model-response mapper with reconciliation that preserves
|
The schema/transport boundary remains with PromptKit; reconciliation needs no
|
||||||
unambiguous decisions and explicitly represents unavailable candidates and ignored
|
additional model call. Concrete helper structures, reason names, mapping changes,
|
||||||
entries. Update the LLM-backed validator to return those outcomes and accurate
|
and verification steps belong in the [implementation plan](implementation.md).
|
||||||
operation facts instead of rejecting a whole usable batch. Keep domain ownership
|
|
||||||
in Audita and reuse the existing unavailable-candidate path.
|
|
||||||
|
|
||||||
Review the confidence check currently present in the mapper: structurally invalid
|
|
||||||
confidence is already handled by the schema boundary. Do not introduce a second
|
|
||||||
whole-batch failure path for an individual defective entry that reaches domain
|
|
||||||
mapping. Preserve valid negative decisions and remaining validator execution.
|
|
||||||
|
|
||||||
### Reporting and Debug Evidence
|
|
||||||
|
|
||||||
Adapt operation mapping and presentation where needed to express partial batch
|
|
||||||
failure and successful extra-entry filtering. Review the correction ledger so it
|
|
||||||
records unavailable outcomes distinctly from actual editorial decisions. Preserve
|
|
||||||
existing report grouping and avoid a new reporting subsystem or per-entry model
|
|
||||||
operation. Add only the attribution needed to explain these outcomes.
|
|
||||||
|
|
||||||
### Regression Coverage and Documentation
|
|
||||||
|
|
||||||
Replace tests that enforce whole-batch rejection with focused tests of partial
|
|
||||||
success. Use mixed responses to prove valid approvals and rejections survive
|
|
||||||
alongside missing, duplicate, and extraneous indices. Cover conflicting duplicates,
|
|
||||||
empty decision lists, and response-order independence without building an exhaustive
|
|
||||||
cross-product suite.
|
|
||||||
|
|
||||||
Include representative assembled-path coverage proving that survivors complete
|
|
||||||
all remaining validators, unavailable candidates never apply, and reporting reflects
|
|
||||||
the correct errors/recoveries and applied corrections. Retain tests for strict
|
|
||||||
internal cardinality and genuine operation-level failures. Update canonical
|
|
||||||
validator and reporting documentation when implementation lands; do not describe
|
|
||||||
this roadmap's target as currently implemented elsewhere.
|
|
||||||
|
|
||||||
## Scope Boundaries
|
## Scope Boundaries
|
||||||
|
|
||||||
@@ -184,7 +159,5 @@ are introduced. This work does not itself authorize tagging or publishing v2.0.0
|
|||||||
|
|
||||||
## Open Questions
|
## Open Questions
|
||||||
|
|
||||||
No unresolved product-policy questions block the staged implementation plan.
|
No unresolved product-policy questions block implementation. Guided recovery and
|
||||||
That plan should select the smallest reporting/model changes and concrete reason
|
structural salvage remain separate future decisions.
|
||||||
codes needed to express the behavior above. Guided recovery and structural salvage
|
|
||||||
remain separate future decisions.
|
|
||||||
|
|||||||
@@ -1,368 +1,268 @@
|
|||||||
# Implementation Plan: Consistent Span Deletion
|
# Implementation Plan: Partial Validator Decision Sets
|
||||||
|
|
||||||
**Last updated:** 2026-09-25
|
**Last updated:** 2026-09-25
|
||||||
|
|
||||||
**Status:** Implemented; all stages complete.
|
**Status:** Ready for implementation; no stages implemented by this planning pass.
|
||||||
**Source baseline:** `a83a518` (feature roadmap); application baseline `0b6cd98`.
|
**Source baseline:** `60e2fe7`.
|
||||||
**Audience:** gpt-5.6-terra coding agent.
|
**Audience:** gpt-5.6-terra coding agent.
|
||||||
|
|
||||||
## Scope and Execution Rules
|
## Scope and Execution
|
||||||
|
|
||||||
Implement [Consistent Span Deletion](span-deletion.md). That document owns the
|
Implement [partial success for validator decision sets](decision-set-reconciliation.md).
|
||||||
product intent, target semantics, and acceptance criteria. This document owns the
|
That roadmap owns product intent and target semantics; this document selects the
|
||||||
concrete decisions, ordered work, and validation. Execute Stage 1 through Stage 4
|
implementation and verification work. Follow Stage 1 through Stage 3 in order and
|
||||||
in order. Record actual completion and checks beneath each stage when implemented.
|
record completion evidence beneath each stage. Each stage is a focused agent task.
|
||||||
The completion evidence below records the implemented behavior; the source
|
|
||||||
baseline remains historical.
|
|
||||||
|
|
||||||
Read [development guidance](../development.md) and all [agent](../policy/agents.md),
|
Read [development guidance](../development.md) and all policies under
|
||||||
|
[docs/policy](../policy/): [agents](../policy/agents.md),
|
||||||
[architecture](../policy/architecture.md), [documentation](../policy/documentation.md),
|
[architecture](../policy/architecture.md), [documentation](../policy/documentation.md),
|
||||||
[testing](../policy/testing.md), and [release](../policy/release.md) policies.
|
[testing](../policy/testing.md), and [release](../policy/release.md). Use graph-first
|
||||||
Use graph-first code discovery. Inspect the current workspace before editing;
|
code discovery, preserve unrelated work, and commit completed coherent units with
|
||||||
retain unrelated user work. Commit coherent completed units with plain English
|
plain English messages. Do not introduce approval rounds between stages.
|
||||||
messages, following the agent policy. Do not add permission rounds between stages.
|
|
||||||
|
|
||||||
Aim directly at the final state. Do not add temporary compatibility aliases,
|
Aim directly at the final implementation. No temporary modes, compatibility shims,
|
||||||
feature flags, parallel implementations, or scaffolding for intermediate commits.
|
new configuration, extra model calls, or upstream dependency work is required.
|
||||||
Preserve module scope, literal matching, replacement policies, fixed chunks,
|
Keep PromptKit structural validation/repair and the runner's internal cardinality
|
||||||
required validation, failure isolation,
|
checks. Implement the agreed outcomes, not extra restrictions suggested by existing
|
||||||
reporting/debug independence, and existing recovery budgets. Do not implement any
|
tests or incidental internal structure. Internal helper signatures and layout can
|
||||||
of the [deferred features](future.md), automatic whitespace cleanup, a new deletion
|
be adjusted idiomatically while preserving the decisions below.
|
||||||
operation type, segment tombstones, or a second normalization pass.
|
|
||||||
|
|
||||||
Preserve intended safeguards and outcomes, not incidental implementation choices.
|
This replaces the completed span-deletion plan, available in Git at
|
||||||
Validator ordering, internal structure, and payload construction may change where
|
`60e2fe7:docs/roadmap/implementation.md`. It does not reopen that feature. The final
|
||||||
needed to satisfy the feature contract. Current code and tests are evidence to
|
stage includes correcting historical links that otherwise point to this new plan.
|
||||||
review, not authority to retain a conflicting restriction. Prefer reusing correct
|
No release preparation, tag, push, CI wait, or paid-provider evaluation is included.
|
||||||
mechanisms, but do not freeze them merely because they exist.
|
|
||||||
|
|
||||||
This replaces the completed resilience implementation plan, which remains in Git
|
|
||||||
history at `a83a518:docs/roadmap/implementation.md`. Historical resilience decisions
|
|
||||||
and their remediation are not instructions to rebuild or reverse that work.
|
|
||||||
During final documentation updates, repair references that describe this filename
|
|
||||||
as the resilience plan; use current canonical references or the historical revision.
|
|
||||||
No release, version tag, push, upstream change, or new dependency is part of this
|
|
||||||
implementation sprint.
|
|
||||||
|
|
||||||
## Selected Implementation Decisions
|
## Selected Implementation Decisions
|
||||||
|
|
||||||
### Schema and library boundary
|
### Reconciliation boundary
|
||||||
|
|
||||||
Remove `minLength: 1` from the `corrected_text` property of
|
Keep reconciliation in `internal/framework/validators`, after successful structured
|
||||||
`internal/framework/responseschema/assets/correction_set.v1.json`. Keep the required
|
completion and before converting the batch into runner-facing results. Replace
|
||||||
field, string type, and all unrelated schema constraints. Empty and whitespace-only
|
`mapLLMResponseToDecisions` with a small pure reconciliation helper; use an ordinary
|
||||||
strings are structurally valid; missing, null, and non-string replacements are not.
|
private result struct containing:
|
||||||
Keep `original_text` required, string-valued, and at least one character long.
|
|
||||||
Whitespace-only original spans are valid at the structural boundary and must not
|
|
||||||
be rejected solely on that basis by domain checks. Replace trimmed-emptiness checks
|
|
||||||
for original spans with literal empty-string checks in `CorrectionProposal.Validate`
|
|
||||||
and `ProposalShapeValidator.Validate`, and review other consumers for the same
|
|
||||||
restriction. Do not trim either span in transit. Resulting-segment checks still use
|
|
||||||
whitespace-aware emptiness. Preserve module matching policies: `require_unique`
|
|
||||||
for grammar, spoken-word, and homophones; `replace_all` for glossary.
|
|
||||||
|
|
||||||
Keep `CorrectionSetKey`, the schema asset path, ID/name/version, and existing prompt
|
- `Decisions []Decision`: one outcome per requested item, in request order;
|
||||||
IDs/`v1` selections. This widens an internal model-response constraint without
|
- `UnavailableIndices []int`: requested indices without a usable verdict;
|
||||||
changing the correction object shape or creating a second contract to maintain.
|
- `IgnoredIndices []int`: unexpected indices encountered in response order, including
|
||||||
The schema hash and PromptKit's rendered/prepared hashes reflect changed content;
|
repeated unexpected entries, for debug attribution.
|
||||||
no hard-coded hash is a compatibility promise. Ordinary report schema versions,
|
|
||||||
transcript encoders, configuration, and CLI selections do not change.
|
|
||||||
|
|
||||||
PromptKit continues to validate the whole response and own structural repairs.
|
No generic reconciliation framework or new shared abstraction is needed. Requested
|
||||||
Do not introduce partial JSON salvage or Audita-side regeneration. Once the schema
|
identities come from `batch.Items`, not transcript segment IDs, the whole section,
|
||||||
accepts empty replacements, normal per-candidate domain filtering handles proposals
|
or any other batch. Expected input identities remain an Audita internal contract.
|
||||||
that would blank their stage-input segment.
|
|
||||||
|
|
||||||
### Validator identity
|
First collect response entries by requested index and count occurrences, separating
|
||||||
|
unexpected indices. Then walk requested items once to construct outcomes. This
|
||||||
|
avoids accepting an early duplicate before seeing a conflicting later entry.
|
||||||
|
|
||||||
Rename the existing validator consistently:
|
Use these candidate-level reason codes in `validators/models.go`:
|
||||||
|
|
||||||
| Current | Target |
|
| Condition | Internal outcome/reason |
|
||||||
| --- | --- |
|
| --- | --- |
|
||||||
| `non_empty_corrected_text` key and package directory | `non_empty_resulting_segment` |
|
| Missing requested entry | `DispositionUnavailable`, `missing_validation_decision` |
|
||||||
| `KeyNonEmptyCorrectedText` in registry and metadata | `KeyNonEmptyResultingSegment` |
|
| More than one entry for a requested index | `DispositionUnavailable`, `duplicate_validation_decision` |
|
||||||
| `NonEmptyCorrectionValidator` framework type | `NonEmptyResultingSegmentValidator` |
|
| Single requested entry failing the existing confidence range check | `DispositionUnavailable`, `invalid_validation_decision` |
|
||||||
|
| Single usable entry | Existing approved or `llm_rejected` decision with its model reason. |
|
||||||
|
|
||||||
Retain the `empty_resulting_segment` rejection/skip reason, deterministic execution
|
Unavailable outcomes have `Approved: false` and fixed safe explanatory messages.
|
||||||
class, and the resulting-segment safeguard. Reuse preview-based validation where
|
Duplicate classification takes precedence over inspecting individual duplicate
|
||||||
correct; neither its exact implementation nor chain position is frozen. Do not
|
values; identical duplicates follow the same rule. Unexpected entries are ignored
|
||||||
retain an old-key alias.
|
before inspecting confidence. Retain the existing confidence check as a local
|
||||||
This is an intentional validator identity change in diagnostic/decision records;
|
check for decoded callers/test doubles, scoped to that candidate; do not broaden
|
||||||
document it, including any affected ordinary filtered-proposal metadata. It is not
|
this into structural salvage, confidence thresholds, or new semantic validation.
|
||||||
a new operator-selectable setting. Historical release notes remain untouched.
|
The production JSON Schema still rejects out-of-range confidence before mapping.
|
||||||
|
|
||||||
### Prompt and payload semantics
|
An empty response list makes every requested item unavailable. Response ordering
|
||||||
|
must not change verdicts or which candidate is affected. Preserve the existing
|
||||||
|
handling of valid negative decisions, including reason text. No candidate may
|
||||||
|
advance merely because another candidate was approved.
|
||||||
|
|
||||||
Review all four proposal definitions and the active validator definitions
|
### Operation accounting and ordinary reports
|
||||||
(`editorial_review`, `meaning_reversal_review`, `spoken_form_plausibility`) for
|
|
||||||
consistency. Edit only definitions needing clarification. Review the registered
|
|
||||||
but inactive `grammar_review` and `spoken_word_review` definitions too; unchanged,
|
|
||||||
consistent definitions need no edits. Do not activate additional validators as
|
|
||||||
part of this work.
|
|
||||||
|
|
||||||
Where helpful, explain that an empty replacement deletes the identified span,
|
Each actual validation batch remains one `outcomes.Operation`. Use existing
|
||||||
whitespace-only spans can express spacing corrections, and the resulting segment
|
`ReasonValidationDecisionsInvalid` (`validation_decisions_invalid`) for a batch
|
||||||
must remain nonblank. Do not instruct the model to fabricate padding or content.
|
with unavailable requested candidates. Set `Succeeded: false`, set `ProposalIndices`
|
||||||
A spoken-word example could delete `"uh "` from `"uh hello"`; a grammar example
|
to only those unavailable indices, retain execution counts, and leave recovery
|
||||||
could reduce two spaces to one. Examples and wording are illustrative, not frozen
|
reasons empty. Still return usable decisions from that batch alongside explicit
|
||||||
requirements or promises of automatic approval. Explain surrounding whitespace
|
unavailable outcomes, with no Go error for the reconciled model defect.
|
||||||
and punctuation only to the extent needed to produce the intended literal result.
|
|
||||||
Qualify uniqueness by module: glossary uses `replace_all`, while the other three
|
|
||||||
modules use `require_unique`.
|
|
||||||
|
|
||||||
Keep module responsibilities intact. Meaning-reversal review remains a narrow
|
If no requested candidates are unavailable, set `Succeeded: true`. Add
|
||||||
check for antonyms/reversals, not a new general semantic-equivalence validator.
|
`outcomes.RecoveryValidationDecisionsFiltered` with serialized value
|
||||||
Spoken-form plausibility retains its phonetic/mistranscription remit. Empty syntax
|
`validation_decisions_filtered` when extras were ignored. Merge it with actual
|
||||||
alone must not determine approval or rejection.
|
`output_repaired` and `service_recovered` reasons in the same recovery operation.
|
||||||
|
Valid negative decisions do not prevent operation success. A clean response creates
|
||||||
|
no new recovery reason.
|
||||||
|
|
||||||
`BuildLLMValidationRequest` already supplies explicit `corrected_text` and full
|
When unavailable candidates and extras coexist, record the failed operation only;
|
||||||
before/after segment text. Start with that representation and verify that empty
|
keep ignored-entry evidence in debug. Do not create both an error and a recovery
|
||||||
and whitespace-only spans survive it faithfully. Adjust payload construction if
|
for that operation. Successful survivors must continue even though the operation
|
||||||
needed for consistency; do not treat its current structure as an independent
|
itself records a failure. Retain actual execution counts on all paths.
|
||||||
requirement. No new deletion flag or synthetic sentinel is needed merely to
|
|
||||||
represent an empty string.
|
|
||||||
|
|
||||||
### Debug representation and outcomes
|
Reuse existing ordinary YAML structures and version 1. The only new public report
|
||||||
|
value is the decision-specific recovery reason; affected error indices already fit
|
||||||
|
`Failure.ProposalIndices`. Do not add ignored entries to `Operation.Filtered` or
|
||||||
|
`filtered_proposals`, which describe correction proposals, and do not increment
|
||||||
|
`proposal_responses_with_filtering`. The recovery reason and existing operation
|
||||||
|
identity suffice for ordinary reporting; detailed extra indices belong in debug.
|
||||||
|
|
||||||
`CorrectionLedgerEntry` currently uses strings with `omitempty` for both replacement
|
Update `processreport.safeMessage` for `validation_decisions_invalid` to explain
|
||||||
fields, losing explicit empty values. Change `ProposedCorrectedText` and
|
that some required decisions were unavailable without claiming all batch decisions
|
||||||
`AppliedCorrectedText` to `*string` with their existing JSON names and `omitempty`:
|
were discarded. Generalize the recovery message to include response filtering,
|
||||||
|
rather than only proposal filtering. Exact prose is not a test contract. Preserve
|
||||||
|
work-unit error grouping, operation counts, and existing YAML/Markdown/CLI totals.
|
||||||
|
|
||||||
- Every candidate-backed entry has a non-nil proposed value, including `""`.
|
### Debug evidence and runner records
|
||||||
- Applied entries also have a non-nil applied value, including `""`.
|
|
||||||
- Rejected, skipped, and interrupted entries omit the applied value.
|
|
||||||
- Module-level failure entries without a candidate omit both values.
|
|
||||||
|
|
||||||
Construct the pointers directly when mapping the corresponding records; use the
|
Existing interaction diagnostics already retain the full decoded response and final
|
||||||
same simple convention across all dispositions. Do not build custom marshaling
|
content. Add the reconciliation result to the response diagnostic payload for a
|
||||||
or a generic optional-value framework. These values remain replacement spans,
|
successfully decoded validation response. Perform reconciliation before writing
|
||||||
not full resulting-segment text. Preserve redaction and the rest of the ledger.
|
that diagnostic; retain existing call-error diagnostics and diagnostic-write failure
|
||||||
|
handling. All reconciled decisions, including unavailable outcomes, receive the
|
||||||
|
returned response artifact path. This records unexpected indices and duplicates
|
||||||
|
without inventing candidate ledger entries or an additional debug file.
|
||||||
|
|
||||||
Keep existing operation accounting. A valid approved deletion is an ordinary
|
Carry `validators.DecisionDisposition` through `runner.ValidatorDecisionRecord`
|
||||||
applied correction. A candidate that would blank its stage-input segment is
|
and `runner.ValidatorRejectedChange` in an additive `disposition` field. Populate
|
||||||
filtered with the existing reason and contributes to the proposal operation's
|
it from each actual `Decision`, including partial decision evidence returned on
|
||||||
consolidated filtering recovery. Ordinary editorial/protected-term rejection does
|
fatal paths. Keep the existing collection names and application flow; no parallel
|
||||||
not become a recovery. Multiple proposals can target the same segment; they are
|
unavailable-candidate collection is necessary.
|
||||||
validated against stage input and applied sequentially. A proposal applicable to
|
|
||||||
that input but made unsafe by an earlier accepted edit is skipped as a conflict,
|
|
||||||
with no added model error or filtering recovery. Keep the skipped evidence. The
|
|
||||||
runner's `attributeApplyFiltering` already makes this distinction: verify and reuse
|
|
||||||
it rather than invent new classification machinery. Structural repairs required
|
|
||||||
for unrelated defects still count normally.
|
|
||||||
|
|
||||||
## Stage 1 — Align Validator Identity and Confirm Domain Semantics
|
In `processreport`, add `disposition` to `LedgerValidatorDecisionRecord` and map
|
||||||
|
unavailable candidates to a top-level ledger disposition `unavailable`, instead
|
||||||
|
of `rejected`. Ordinary editorial/deterministic rejections remain `rejected`.
|
||||||
|
Use the explicit disposition, not a reason-code allowlist, so existing timeout,
|
||||||
|
provider-failure, and oversized-input outcomes retain the same distinction too.
|
||||||
|
Update affected test fixtures to supply dispositions where they represent real
|
||||||
|
validator outputs. Do not invent a fatal check or compatibility subsystem for a
|
||||||
|
zero-value disposition in handcrafted fixtures.
|
||||||
|
|
||||||
|
Retain candidate identity, literal proposed replacements (including explicit empty
|
||||||
|
strings), and omission of applied replacements for unapplied candidates. Existing
|
||||||
|
fatal/interrupted/module-failure handling and debug-off behavior remain. Supplied
|
||||||
|
model decisions remain visible in interaction evidence even when reconciliation
|
||||||
|
cannot accept them as authoritative verdicts.
|
||||||
|
|
||||||
|
## Stage 1 — Reconcile Model Decisions and Preserve Valid Work
|
||||||
|
|
||||||
**Work**
|
**Work**
|
||||||
|
|
||||||
- Rename the validator package/type/key described above. Update `New()` wiring,
|
1. Read the validator models, `llm_validators.go`, existing partial-batch tests,
|
||||||
`internal/validators/registry.go`, `chains.go`, metadata, framework validators,
|
`outcomes/outcomes.go`, and `runner.validateSectionCandidates`.
|
||||||
consumers, and tests in one coherent change. Use graph discovery for symbols and
|
2. Implement the reconciliation result/helper and candidate reason codes selected
|
||||||
string searches for serialized keys. Do not globally rewrite historical docs.
|
above. Replace the whole-batch mapping-error branch with outcome construction
|
||||||
- Preserve existing `CorrectionProposal.Validate`, `PreviewProposalForSegment`,
|
and correct batch accounting. Add the decision-specific recovery constant.
|
||||||
`ApplyProposals`, and deterministic validation behavior wherever already correct.
|
3. Move successful-response diagnostic assembly after reconciliation and include
|
||||||
Change original-span trimmed-emptiness checks as specified above, and find any
|
its result. Preserve raw/decoded response evidence and diagnostic paths; keep
|
||||||
equivalent downstream restrictions. Keep empty-original rejection and
|
recoverable call errors, fatal errors, and cancellation on their existing paths.
|
||||||
whitespace-aware resulting-segment rejection distinct.
|
4. Keep `EnforceDecisionCardinality` and `EnforcePartialDecisionCardinality` strict.
|
||||||
- Extend the existing proposal and validator tests only for missing meaningful
|
Ensure final validator decisions are complete and unique internally. The runner
|
||||||
coverage: empty replacement leaving a nonblank segment, whitespace-only resulting
|
should continue passing only approved candidates onward without treating an
|
||||||
text rejection, whitespace-only originals, literal spacing, and apply-time
|
operation's failure metadata as a reason to discard valid decisions.
|
||||||
reevaluation after an earlier edit. Cover `"hello world"` with `" " -> " "`,
|
5. Replace obsolete batch-rejection expectations in `llm_validators_test.go` and
|
||||||
an empty original, and two separate matching whitespace runs under `require_unique`.
|
relevant runner tests. Preserve cardinality tests that exercise broken internal
|
||||||
Assert module-policy ambiguity handling without imposing uniqueness on glossary.
|
validators. Update the repaired-but-domain-invalid tests so actual unavailable
|
||||||
For apply-time reevaluation, use segment `"a b"` and ordered deletions `"a" -> ""`,
|
cases remain errors with counts retained and no recovery record.
|
||||||
`"b" -> ""`: each is previewable initially, but after the first edit the second
|
|
||||||
must be skipped as `empty_resulting_segment`. The surviving text is `" b"`;
|
**Focused verification**
|
||||||
no automatic trim or segment deletion is expected. At the runner/outcome boundary,
|
|
||||||
verify this initially applicable second proposal produces skipped debug evidence
|
Use a small table at the validator boundary for mixed missing/extra indices,
|
||||||
but no model error or filtering recovery. Reuse/extend existing conflict tests;
|
conflicting and identical requested duplicates, extra-only responses, empty lists,
|
||||||
a second full CLI scenario is unnecessary for this attribution check.
|
reordered responses, and a single invalid-confidence decoded test-double entry.
|
||||||
- Ensure protected-term validation still rejects removal of a protected term with
|
Include noncontiguous expected indices and valid negative decisions. Assert semantic
|
||||||
an empty replacement. Prefer one case in its existing test table. Retain existing
|
outcomes and affected indices, not helper layout or exact messages. Include one
|
||||||
metadata-preservation tests rather than duplicating all metadata assertions.
|
controlled case combining successful structural/service recovery facts with ignored
|
||||||
- Update `docs/internal/validators.md` for the renamed implemented validator and
|
extras and one failed reconciliation retaining counts without recovery reasons.
|
||||||
explain its resulting-segment responsibility; do not yet claim schema alignment.
|
|
||||||
|
At the runner boundary, use the real LLM-backed validator with a controlled client
|
||||||
|
and a later validator: prove a survivor reaches later required validation, a later
|
||||||
|
rejection prevents application, and unavailable candidates never reach that later
|
||||||
|
validator as approved. Use successful proposal operation facts where accounting is
|
||||||
|
asserted; do not check an empty operation list. Reuse existing fixtures/tests rather
|
||||||
|
than creating a second framework or duplicating every table case.
|
||||||
|
|
||||||
|
Run `go test ./internal/framework/validators ./internal/framework/runner`.
|
||||||
|
|
||||||
|
**End state:** Usable decisions survive model indexing defects, strict internal
|
||||||
|
cardinality remains satisfied, and operation facts explain partial failure or
|
||||||
|
successful extra filtering. No extra model generation is introduced.
|
||||||
|
|
||||||
|
## Stage 2 — Carry Dispositions Through Debug and Report Partial Outcomes
|
||||||
|
|
||||||
|
**Work**
|
||||||
|
|
||||||
|
1. Add and populate disposition fields through the runner and ledger as selected
|
||||||
|
above. Map unavailable candidates distinctly and preserve empty replacement
|
||||||
|
serialization. Check both normal and fatal partial-evidence paths.
|
||||||
|
2. Update `processreport` messages and review `Build`, `mapFailure`, and `mapRecovery`.
|
||||||
|
Reuse their existing aggregation; change only assumptions that would misreport
|
||||||
|
the new operation facts. Core YAML structures and Markdown table layout remain.
|
||||||
|
3. Extend existing report/ledger tests for an operation with surviving work plus
|
||||||
|
unavailable indices, extra-only recovery, and a mixed defect operation. Verify
|
||||||
|
error indices exclude survivors, extra filtering does not count as proposal
|
||||||
|
filtering, and failed operations do not also generate recovery records.
|
||||||
|
4. Verify debug ledger entries distinguish genuine rejection from unavailability,
|
||||||
|
and interaction diagnostics preserve ignored entries and reconciliation reasons.
|
||||||
|
Unknown response indices must not create fake ledger candidates. Retain existing
|
||||||
|
redaction and optional-debug-failure tests rather than reproducing them all.
|
||||||
|
5. Update `docs/integrations/output-files.md` for the new recovery reason and debug
|
||||||
|
disposition, and `docs/internal/diagnostics-reporting.md` for the mapping changes.
|
||||||
|
Explain that validation failure indices identify unavailable candidates, not
|
||||||
|
necessarily every candidate submitted to that batch.
|
||||||
|
|
||||||
**Validation**
|
**Validation**
|
||||||
|
|
||||||
Run `go test ./internal/framework/proposals ./internal/framework/validators ./internal/validators/... ./internal/framework/runner ./internal/framework/processreport`.
|
Run `go test ./internal/framework/runner ./internal/framework/processreport ./internal/core/reporting`.
|
||||||
Existing registry/chain tests must use the new identity and verify the required
|
Use semantic YAML/summary assertions; no whole-report prose golden, exact message
|
||||||
validation and execution class. Update incidental order assertions if a justified
|
snapshot, new schema version, or new rendering mode is needed.
|
||||||
ordering change preserves those safeguards.
|
|
||||||
Inspect remaining old-name references, retaining only deliberately historical ones.
|
|
||||||
|
|
||||||
**End state:** The accurately named domain validator and application preserve the
|
**End state:** Reports and debug evidence accurately describe partial validation
|
||||||
agreed behavior. No schema or retry workaround has been introduced.
|
success and handled extras through existing reporting mechanisms.
|
||||||
|
|
||||||
**Completed 2026-09-25.** Renamed the validator, admitted nonempty whitespace
|
## Stage 3 — Verify the Assembled Workflow and Finalize Documentation
|
||||||
original spans, and verified literal spacing, blank-result protection, protected
|
|
||||||
terms, module-policy ambiguity, and apply-time conflict attribution. The specified
|
|
||||||
focused package tests passed.
|
|
||||||
|
|
||||||
## Stage 2 — Align Response Schema, Prompts, and LLM Validation Inputs
|
|
||||||
|
|
||||||
**Work**
|
**Work**
|
||||||
|
|
||||||
- Apply the selected schema relaxation and selectively clarify prompts. Preserve
|
1. Extend the existing local-provider subprocess fixtures in
|
||||||
intended task scope, hardening, and response contracts while correcting conflicting
|
`cmd/audita/main_integration_test.go`. Add a representative mixed-decision run
|
||||||
instructions wherever encountered within this feature. Review the
|
with an applied survivor, a legitimate rejection, a missing requested decision,
|
||||||
rendered definitions, not just YAML source, using existing PromptKit inspection
|
a duplicate requested decision, and an extra index. Return responses by the
|
||||||
and prompt tests.
|
requested correction identities; do not freeze incidental validator call order.
|
||||||
- In `internal/framework/responseschema/registry_test.go`, retain the check that
|
Ensure the survivor passes all remaining required validation. Assert successful
|
||||||
reported hashes equal hashes of the actual embedded bytes. Remove fixed digest
|
CLI completion, correct transcript, one work-unit error with only unavailable
|
||||||
expectations that merely freeze asset content; do not replace them with a new
|
indices, no reconciliation recovery for the failed operation, and debug
|
||||||
golden digest for this change.
|
evidence distinguishing rejection/unavailability without unknown candidates.
|
||||||
- Extend the existing real-adapter tests in `internal/framework/llm/adapter_test.go`
|
2. Add or adapt a compact extra-only case proving usable work survives and one
|
||||||
with a bounded table for replacement values: empty string, spaces, ordinary text,
|
`validation_decisions_filtered` recovery appears. Verify YAML, Markdown, and CLI
|
||||||
missing field, null, and a non-string value. Use the embedded schema and local
|
agree on totals using semantic assertions. Keep proposal-filtering counts separate.
|
||||||
HTTP fixture. Give accepted cases a positive repair allowance and assert one
|
Existing reporting-disable/debug selection tests remain the owners of those options.
|
||||||
provider generation, zero repairs, and preservation of the actual string. For
|
3. In these fixtures give PromptKit its normal configured structural repair allowance,
|
||||||
invalid cases, use zero repair attempts to prove rejection without redundant
|
but supply schema-valid responses. Verify reconciliation adds no generations or
|
||||||
model calls. Include a whitespace-only original in an accepted case and confirm
|
structural repairs using fixture requests/execution facts, without encoding today's
|
||||||
an empty original is structurally invalid. Existing tests already own successful
|
total validator count as a permanent contract. Keep structural-malformation tests
|
||||||
repair/exhaustion behavior.
|
and recoverable batch-failure tests; do not relax the schema to make fixtures pass.
|
||||||
- Extend the validation-request tests in
|
4. Update `docs/internal/validators.md` and any affected canonical descriptions found
|
||||||
`internal/framework/validators/llm_validators_test.go` to marshal a real request
|
by searching for whole-batch rejection claims. Explain tolerant external response
|
||||||
for deleting `"uh "` from `"uh hello"`. Assert that JSON contains the explicit
|
reconciliation versus strict internal cardinality. The structural-validation
|
||||||
empty replacement plus full before/after text, correct identity, and no immediate
|
boundary remains explicit, and guided recovery remains deferred in `future.md`.
|
||||||
rejection. Include a whitespace-only-original spacing correction in the request
|
5. Mark the feature roadmap implemented only after the assembled checks pass, and
|
||||||
coverage and verify literal preservation. A field-presence assertion is required;
|
record actual completion evidence in this plan. Repair the two span-deletion
|
||||||
decoding into a plain string alone cannot distinguish omitted from empty.
|
roadmap links to the old plan with the historical URL
|
||||||
- Update canonical prompt/module guidance for the implemented semantics. Do not
|
`https://gitea.maximumdirect.net/eric/audita/src/commit/60e2fe7/docs/roadmap/implementation.md`.
|
||||||
add exact-prose prompt snapshots or assertions for every new sentence; existing
|
Preserve historical release notes and unrelated roadmaps.
|
||||||
parsing/rendering/schema-association tests plus editorial review are sufficient.
|
|
||||||
|
|
||||||
**Validation**
|
**Validation**
|
||||||
|
|
||||||
Run `go test ./internal/framework/responseschema ./internal/prompts ./internal/framework/llm ./internal/framework/proposal_generation ./internal/framework/validators ./internal/modules/...`.
|
Run `go test ./cmd/audita` while developing the fixture, then `go test ./...` once
|
||||||
Confirm missing/non-string replacements still fail structurally and domain-invalid
|
for the completed implementation. Run `git diff --check` and verify affected
|
||||||
but schema-valid proposals still reach ordinary filtering without an outer retry.
|
Markdown links. Do not add live-provider checks, a platform matrix, repeated full
|
||||||
|
suite runs, or release gates solely for this feature. No code is implemented by
|
||||||
|
this planning pass; these commands are instructions for the implementation sprint.
|
||||||
|
|
||||||
**End state:** A literal empty replacement can cross the real library/schema
|
**End state:** The accepted partial-success behavior is exercised through the real
|
||||||
boundary, and the model instructions and validator inputs describe its meaning.
|
CLI/library boundary, canonical documentation matches it, and coherent changes
|
||||||
|
are committed for subsequent release consideration.
|
||||||
**Completed 2026-09-25.** Relaxed only the replacement length constraint,
|
|
||||||
clarified the relevant proposal and active validation prompts, and verified
|
|
||||||
rendering, real-adapter acceptance/rejection, repair counts, and serialized
|
|
||||||
before/after validator payloads. The specified focused package tests passed.
|
|
||||||
|
|
||||||
## Stage 3 — Preserve Empty Replacements in Debug Evidence
|
|
||||||
|
|
||||||
**Work**
|
|
||||||
|
|
||||||
- Implement the pointer representation selected above in
|
|
||||||
`internal/framework/processreport/correction_ledger.go`, updating every mapping
|
|
||||||
branch and affected test consumer. Leave other debug structures unchanged where
|
|
||||||
their string fields already serialize empty values explicitly.
|
|
||||||
- Extend `correction_ledger_test.go` with a small serialized-disposition table:
|
|
||||||
applied deletion, rejected deletion, skipped deletion, interrupted deletion,
|
|
||||||
and a module failure without a candidate. Decode to a map or raw JSON fields to
|
|
||||||
check presence, `""` values, and absence of the applied value on unapplied entries.
|
|
||||||
Preserve existing decision classification/order tests using the renamed key.
|
|
||||||
- Update `docs/integrations/output-files.md` to explain the two debug replacement
|
|
||||||
fields and the deliberate validator identity change. Keep ordinary schemas and
|
|
||||||
report counts unchanged. Update the diagnostics/reporting internal reference
|
|
||||||
for the implemented mapping, linking to the public artifact contract.
|
|
||||||
|
|
||||||
**Validation**
|
|
||||||
|
|
||||||
Run `go test ./internal/framework/processreport ./internal/core/diagnostics ./internal/cli ./cmd/audita`.
|
|
||||||
Tests must inspect serialized output, not just non-nil in-memory values. Retain
|
|
||||||
existing configured-secret redaction and debug-off persistence coverage.
|
|
||||||
|
|
||||||
**End state:** Debug evidence distinguishes a proposed/applied deletion from an
|
|
||||||
absent proposal or an edit that was never applied.
|
|
||||||
|
|
||||||
**Completed 2026-09-25.** Candidate-backed ledger entries now preserve an
|
|
||||||
explicit empty proposed replacement, and applied entries preserve an explicit
|
|
||||||
empty applied replacement. Serialized disposition coverage and the specified
|
|
||||||
focused package tests passed; the artifact contract documents the fields and
|
|
||||||
validator identity change.
|
|
||||||
|
|
||||||
## Stage 4 — Verify the Integrated Contract and Finish Documentation
|
|
||||||
|
|
||||||
**Work**
|
|
||||||
|
|
||||||
Extend `cmd/audita/main_integration_test.go` with representative offline subprocess
|
|
||||||
cases using its controlled HTTP provider, real configuration/profiles, embedded
|
|
||||||
schemas, built-in module/validator chains, CLI, and output/report/debug writers.
|
|
||||||
Keep fixture additions local and small; do not build a new integration framework.
|
|
||||||
Use explicit fixture sizing so relevant validator candidates fit one batch, and
|
|
||||||
use stable response indices. Do not rely on production chunk/token defaults or
|
|
||||||
on concurrent request order to choose responses.
|
|
||||||
|
|
||||||
Cover these distinct outcomes:
|
|
||||||
|
|
||||||
- **Clean deletion plus another correction:** A spoken-word stage proposes
|
|
||||||
`"uh " -> ""` in `"uh hello"` and an independent repetition collapse. Controlled
|
|
||||||
validators approve both. Assert corrected transcript, two applied corrections,
|
|
||||||
zero model errors/recoveries, zero structural repairs with a positive repair
|
|
||||||
allowance, and explicit empty proposed/applied fields in the debug ledger.
|
|
||||||
- **Whitespace-only span:** A grammar stage reduces one uniquely occurring run
|
|
||||||
of two spaces to one. Controlled validators approve it; assert the expected
|
|
||||||
text and no error/recovery solely due to whitespace. This proves schema, domain
|
|
||||||
validation, and application agree through the actual CLI. Use a compact fixture
|
|
||||||
without adding a matrix of whitespace characters or lengths.
|
|
||||||
- **Partial success with filtering:** Add a proposal to delete all text from a
|
|
||||||
separate filler-only segment, preserving that separate segment through fixture
|
|
||||||
normalization (use sufficient time separation or explicit normalization settings).
|
|
||||||
Assert that this candidate is filtered with `empty_resulting_segment`, the other
|
|
||||||
corrections still apply, the blanked segment is instead retained unchanged, and
|
|
||||||
the proposal response contributes one consolidated filtering recovery and no
|
|
||||||
work-unit error. The renamed deterministic validator must appear in evidence.
|
|
||||||
- **Ordinary negative decision:** Return a schema-valid deletion proposal, then a
|
|
||||||
valid negative decision from meaning-reversal review after earlier checks approve
|
|
||||||
it. Assert unchanged affected text, no applied deletion, no error/recovery solely
|
|
||||||
from the rejection, and a rejected debug entry. The controlled decision proves
|
|
||||||
the chain respects a negative decision, not that an arbitrary real model will
|
|
||||||
make the same editorial judgment.
|
|
||||||
|
|
||||||
Reuse existing YAML/Markdown/CLI count assertions rather than snapshotting entire
|
|
||||||
reports. Verify that the clean case's report totals agree with the transcript and
|
|
||||||
ledger. Compare resulting metadata to the normalized input, since initial
|
|
||||||
normalization may legitimately merge segments or reassign IDs. Confirm retained
|
|
||||||
section membership/order in the existing runner boundary tests; add a targeted
|
|
||||||
assertion only if that property is not already covered for the exercised path.
|
|
||||||
Do not repeat all cases for all modules, encoders, debug/report combinations, or
|
|
||||||
platforms: those unchanged mechanisms already have coverage.
|
|
||||||
|
|
||||||
Finish the documentation and status work:
|
|
||||||
|
|
||||||
- Review current module, validator, prompt, diagnostics, and output references for
|
|
||||||
consistency. Add a concise note to `docs/migrations/v2.md` describing the renamed
|
|
||||||
validator identity and explicit empty debug replacement fields; no alias or
|
|
||||||
migration tool is required. No release note/version is selected in this sprint.
|
|
||||||
- Review links from other roadmaps to `implementation.md`. References to completed
|
|
||||||
resilience stages should point to their historical plan at the recorded revision
|
|
||||||
or current canonical behavior, not silently identify this plan as that work.
|
|
||||||
Update those references only as needed; do not rewrite historical findings.
|
|
||||||
- Mark this plan and `span-deletion.md` implemented only after the criteria pass.
|
|
||||||
Keep the baseline description historical and retain future.md's exploratory
|
|
||||||
whole-segment-removal status. Record concise evidence under completed stages.
|
|
||||||
|
|
||||||
**Final validation**
|
|
||||||
|
|
||||||
Run `go test ./...` once after the integrated changes. Run `git diff --check` and
|
|
||||||
validate affected documentation links/examples. Focused stage checks need not be
|
|
||||||
repeated solely as ceremony. Additional race/platform/live-provider checks are
|
|
||||||
not automatic gates for this feature; run additional checks only for a concrete
|
|
||||||
risk introduced by actual changes. This is not a release invocation.
|
|
||||||
|
|
||||||
**End state:** The feature roadmap's acceptance criteria are evidenced by real
|
|
||||||
schema/adapter tests, domain/application tests, serialized debug tests, and
|
|
||||||
representative complete CLI workflows. The workspace is committed and the docs
|
|
||||||
describe implemented behavior without claiming whole-segment removal.
|
|
||||||
|
|
||||||
**Completed 2026-09-25.** Four offline CLI workflows verified clean deletion,
|
|
||||||
literal spacing, partial success with blank-result filtering, and an ordinary
|
|
||||||
negative meaning-review decision. Metadata was checked against normalized
|
|
||||||
input; the existing runner test covers fixed section membership and order.
|
|
||||||
`go test ./...` passed. Documentation references, affected links, and whitespace
|
|
||||||
were checked. Whole-segment removal remains deferred.
|
|
||||||
|
|
||||||
## Open Questions
|
## Open Questions
|
||||||
|
|
||||||
None requiring a product decision remain. Validator names, schema/prompt identity,
|
None required before implementation. The implementation decisions above preserve
|
||||||
debug representation, module scope, test boundaries, and deferred behavior are
|
the agreed product scope. General structural salvage and guided regeneration remain
|
||||||
selected above. Ordinary implementation details may be resolved idiomatically
|
separate future decisions, not unresolved prerequisites for this work.
|
||||||
within these decisions; a newly discovered conflict with the agreed contract
|
|
||||||
should be recorded rather than resolved by silently strengthening restrictions.
|
|
||||||
|
|||||||
Reference in New Issue
Block a user