118 lines
6.7 KiB
Markdown
118 lines
6.7 KiB
Markdown
# Feedback-Aware Semantic-Reconciliation Retries
|
|
|
|
## Purpose
|
|
|
|
Improve semantic registry reconciliation for smaller models by turning the
|
|
existing module-requested retry into a bounded corrective exchange. Preserve
|
|
the current deterministic safety checks, shared retry budget, and safe fallback
|
|
when correction is unsuccessful.
|
|
|
|
## Problem
|
|
|
|
NPC, item, and location registry normalizers can reject unsafe duplicate-group
|
|
proposals and retain a safe partial result. They currently ask the pipeline for
|
|
another attempt, but the next attempt receives the original prompt without the
|
|
defective response or an explanation of the defect. The normalizers also build
|
|
their retry messages from internal issue-category strings, which are suitable
|
|
for diagnostics but not useful model instructions. If all attempts are
|
|
exhausted, the warning reports the omitted-group count without explaining the
|
|
final defect or confirming that affected candidates were preserved separately.
|
|
|
|
## Target State
|
|
|
|
- A normalizer retry directive may carry model-facing correction guidance
|
|
separately from its stable reason code, operator message, and fallback
|
|
diagnostics.
|
|
- When the directive includes correction guidance and a valid
|
|
`single_response_v1` model candidate, the shared producer-attempt state
|
|
machine reconstructs the complete original prompt and appends the exact
|
|
defective response as an assistant message followed by one user correction
|
|
message. Earlier failed attempts do not accumulate.
|
|
- This correction consumes the existing normalize-stage retry budget. It does
|
|
not add a module-local loop or alter PromptKit's independent structured-output
|
|
repair budget.
|
|
- A retry caused by structurally invalid output remains a fresh stage attempt
|
|
without semantic feedback when no valid model candidate is available.
|
|
- Internal issue categories remain available for operator and debug
|
|
diagnostics, but are never sent to a model. A shared renderer translates
|
|
every known category into concise prose that explains the defect and the
|
|
required valid shape.
|
|
- Item reconciliation appends its domain rule when a proposal attempts to
|
|
combine different currency denominations or currency with a non-currency
|
|
item. NPC and location reconciliation use only the shared protocol guidance.
|
|
- On final exhaustion, safe proposal groups remain applied, unsafe groups
|
|
remain separate, and one process warning gives a bounded summary of the final
|
|
defect and explicitly states that the affected candidates were preserved
|
|
separately.
|
|
|
|
## Required Work
|
|
|
|
1. Extend `contracts.NormalizeRetry` and its clone, validation, and debug paths
|
|
with an optional, bounded model-facing correction-guidance field. Keep
|
|
reason codes and operator messages out of model input. Require guidance to
|
|
be paired with a valid `single_response_v1` model candidate; preserve the
|
|
existing feedback-free directive for structural retry cases.
|
|
2. Extend the generic producer-attempt state machine so a feedback-capable
|
|
module retry constructs `SemanticCorrection` from the latest exact response
|
|
and the supplied guidance. Preserve attempt provenance, cancellation,
|
|
terminal policy, diagnostic handling, byte-faithful response provenance,
|
|
and the rule that only the latest response is included.
|
|
3. Add a model-facing renderer to `internal/framework/semanticreconcile`.
|
|
Implement it with a typed lookup table keyed by `IssueCategory` and an
|
|
authoritative list of all categories, rather than a positional array.
|
|
Render response-local duplicate-group numbers in human-friendly form,
|
|
de-duplicate repeated guidance, enforce a size bound, and return an error for
|
|
any unmapped category instead of exposing its raw value. Retain
|
|
`IssueDetails` for operator/debug diagnostics.
|
|
4. Update the NPC, item, and location registry normalizers to populate the new
|
|
correction guidance for semantically invalid proposals. Add the item-only
|
|
currency guidance after the shared prose when typed application rejects a
|
|
group. Do not add guidance to the invalid-structured-output path when no
|
|
exact valid candidate exists.
|
|
5. Improve exhausted-fallback diagnostics across all three registries. Describe
|
|
the final proposal failure accurately, state that affected candidates were
|
|
preserved separately, and keep warning samples bounded and free of raw model
|
|
responses.
|
|
6. Update the shared semantic-reconciliation and affected normalizer policy
|
|
fingerprints so checkpoints produced under blind retry behavior are not
|
|
reused. Record the durable module-retry correction decision in a new ADR,
|
|
and update the architecture, pipeline, LLM-runtime, and D&D internal docs in
|
|
their canonical scopes when the behavior is implemented.
|
|
7. Add lean offline behavioral coverage for corrective module retries, exact
|
|
latest-response use, feedback-free structural retries, contract validation,
|
|
complete category-to-prose coverage, domain-specific item guidance, bounded
|
|
messages, safe final fallback, and warning content. Avoid exact prose
|
|
snapshots, prompt-length assertions, live providers, and redundant tests of
|
|
PromptKit internals.
|
|
|
|
## Non-Goals
|
|
|
|
- Changing normalize retry counts, PromptKit repair settings, or pipeline
|
|
terminal policies.
|
|
- Changing the reconciliation prompt protocol, response schema, registry
|
|
artifact schemas, or request-local candidate IDs.
|
|
- Reintroducing provider-sensitive schema constraints such as `uniqueItems`.
|
|
- Automatically applying an unsafe duplicate proposal or treating a fallback
|
|
warning as an extraction-quality warning.
|
|
- Persisting raw model responses or correction text in ordinary warnings,
|
|
receipts, manifests, caches, or checkpoints.
|
|
|
|
## Acceptance Criteria
|
|
|
|
- A semantic proposal rejected by an NPC, item, or location registry normalizer
|
|
receives the exact defective response and actionable prose on the next
|
|
normalize attempt, using only the configured stage retry budget.
|
|
- Every `IssueCategory` has non-empty model-facing guidance, no raw category or
|
|
reason-code string reaches the model, and an unmapped future category fails
|
|
closed.
|
|
- An invalid item consolidation additionally tells the model to keep currency
|
|
denominations distinct and not mix currency with non-currency items.
|
|
- Structural-output failures without a valid model candidate retain the current
|
|
fresh-retry behavior.
|
|
- After retry exhaustion, safe consolidations remain applied, affected
|
|
candidates remain separate, and the single fallback warning explains both
|
|
the final cause and preservation behavior without including raw model output.
|
|
- Checkpoint identities change where policy changed, canonical documentation
|
|
and a new ADR describe the implemented behavior, and focused tests plus
|
|
`go test ./...`, `go vet ./...`, and `go build ./cmd/notarius` pass.
|