Improve semantic reconciliation retries
This commit is contained in:
76
docs/adr/0016-feedback-aware-module-requested-retries.md
Normal file
76
docs/adr/0016-feedback-aware-module-requested-retries.md
Normal file
@@ -0,0 +1,76 @@
|
|||||||
|
# ADR-0016: Use feedback-aware module-requested retries
|
||||||
|
|
||||||
|
**Status:** Accepted
|
||||||
|
**Date:** 2026-08-28
|
||||||
|
|
||||||
|
## Context
|
||||||
|
|
||||||
|
An LLM-backed module can discover that a structurally valid model response is
|
||||||
|
unsafe while translating it into a typed candidate, before the ordinary
|
||||||
|
validator chain runs. Semantic registry reconciliation is the first such case:
|
||||||
|
the shared core can identify invalid duplicate-group proposals, and typed
|
||||||
|
application can reject a domain-incompatible group while preserving a safe
|
||||||
|
partial result. Repeating the original request without the rejected response or
|
||||||
|
corrective explanation gives the model no information with which to improve.
|
||||||
|
|
||||||
|
The existing feedback-aware validation mechanism already establishes the safe
|
||||||
|
correction protocol, but module-owned retry directives also carry internal
|
||||||
|
reason codes, operator messages, and fallback diagnostics. Those values are not
|
||||||
|
model instructions. Some module retry conditions, including exhausted
|
||||||
|
structured-output repair, also have no valid exact response to append.
|
||||||
|
|
||||||
|
## Decision
|
||||||
|
|
||||||
|
An LLM-backed normalizer may attach optional, bounded model-facing correction
|
||||||
|
guidance to a module-requested retry. Guidance is a separate contract field; the
|
||||||
|
framework never derives it from a reason code, operator message, diagnostic, or
|
||||||
|
error. A feedback-capable directive must include the exact model candidate that
|
||||||
|
controlled the safe fallback and must use `single_response_v1`.
|
||||||
|
|
||||||
|
The artifact-neutral producer-attempt state machine constructs the correction
|
||||||
|
from that exact latest response and the supplied guidance. The producer rebuilds
|
||||||
|
its complete ordinary request and appends the response as one assistant message
|
||||||
|
and the guidance as one user message. Earlier attempts do not accumulate, and
|
||||||
|
the attempt consumes the same configured stage retry budget as operational,
|
||||||
|
structural, validation, and feedback-free module retries.
|
||||||
|
|
||||||
|
A module retry without guidance remains valid and starts a fresh attempt. This
|
||||||
|
is the required behavior when no valid model candidate exists, including after
|
||||||
|
structured-output repair is exhausted. If feedback is supplied without a valid
|
||||||
|
supported candidate, the framework fails the module contract rather than
|
||||||
|
retrying blindly or inventing guidance.
|
||||||
|
|
||||||
|
After retry exhaustion, the normalizer's safe fallback continues through its
|
||||||
|
validator chain. Safe semantic groups may remain applied, unsafe groups remain
|
||||||
|
separate, and bounded fallback diagnostics may explain the process degradation.
|
||||||
|
Exact responses and correction text remain attempt-local and are excluded from
|
||||||
|
ordinary errors, warnings, manifests, receipts, caches, and checkpoints.
|
||||||
|
|
||||||
|
This decision extends, rather than supersedes,
|
||||||
|
[ADR-0014](0014-feedback-aware-validation-retries.md): both correction sources
|
||||||
|
use the same transport-neutral payload, replacement-request semantics, outer
|
||||||
|
retry budget, and sensitive-data boundary.
|
||||||
|
|
||||||
|
## Alternatives considered
|
||||||
|
|
||||||
|
- Continue blind module retries. This preserves a smaller contract but wastes
|
||||||
|
the module's deterministic diagnosis and commonly repeats the same defect.
|
||||||
|
- Convert module safety checks into validators. Typed reconciliation must apply
|
||||||
|
only safe proposal groups and retain a fallback before validation; moving
|
||||||
|
artifact-owned translation and application policy into validators would blur
|
||||||
|
stage ownership.
|
||||||
|
- Copy the retry reason or operator message into the model request. Those values
|
||||||
|
are written for provenance and humans, can contain opaque internal labels,
|
||||||
|
and do not reliably describe a correct replacement.
|
||||||
|
- Require feedback for every module retry. Structural failures may have no
|
||||||
|
valid exact candidate, so this would either prevent useful fresh retries or
|
||||||
|
fabricate prior-response material.
|
||||||
|
|
||||||
|
## Consequences
|
||||||
|
|
||||||
|
The normalize retry contract and generic producer-attempt directive gain an
|
||||||
|
optional correction-guidance field and candidate-pairing validation. Modules
|
||||||
|
that use it must provide semantically meaningful bounded prose and an exact
|
||||||
|
candidate. Registry reconciliation maintains separate operator and model
|
||||||
|
renderers, and policy fingerprints change so checkpoints created under blind
|
||||||
|
retry behavior are not reused.
|
||||||
@@ -29,13 +29,17 @@ The D&D registrar registers the family’s artifact codecs, extractors, typed
|
|||||||
append-order mergers, normalizers, validators, prompt assets, fallback LLM
|
append-order mergers, normalizers, validators, prompt assets, fallback LLM
|
||||||
profile asset, and default validator chains. Each extractor and normalizer has
|
profile asset, and default validator chains. Each extractor and normalizer has
|
||||||
a stable module spec, explicit execution class, strict option decoding, and a
|
a stable module spec, explicit execution class, strict option decoding, and a
|
||||||
typed builder. Scene chunking, every extractor, and NPC, location, and item-registry
|
typed builder. Scene chunking, every extractor, and NPC, location, and
|
||||||
normalization are registered as `llm_backed`; the remaining current D&D mergers
|
item-registry normalization are registered as `llm_backed`; the remaining
|
||||||
and normalizers are `deterministic`. The metadata is available to catalog inspection and
|
current D&D mergers and normalizers are `deterministic`. The metadata is
|
||||||
resolved-pipeline debug data and determines which selected bindings inherit the
|
available to catalog inspection and resolved-pipeline debug data and determines
|
||||||
pipeline profile. The registry normalizers use `single_response_v1`, forwarding
|
which selected bindings inherit the pipeline profile. The registry normalizers
|
||||||
corrections to their reconciliation completion and retaining the accepted raw
|
use `single_response_v1`, forwarding corrections to their reconciliation
|
||||||
proposal only as an owned model candidate. Configuration remains the canonical owner of the exact keys,
|
completion and retaining the accepted raw proposal only as an owned model
|
||||||
|
candidate. When deterministic proposal safety or typed application rejects a
|
||||||
|
group, they provide separate model-facing prose for a corrective module retry;
|
||||||
|
internal issue categories, reason codes, and operator messages remain
|
||||||
|
diagnostic-only. Configuration remains the canonical owner of the exact keys,
|
||||||
profile precedence, and validator order.
|
profile precedence, and validator order.
|
||||||
|
|
||||||
Private structured-LLM response schemas are deliberately minimal. They reject
|
Private structured-LLM response schemas are deliberately minimal. They reject
|
||||||
@@ -117,6 +121,17 @@ presentation, and final ephemeral generic transcript windows. These orders and
|
|||||||
cache controls are prompt behavior; change them only through the owning
|
cache controls are prompt behavior; change them only through the owning
|
||||||
manifest and prompt declaration.
|
manifest and prompt declaration.
|
||||||
|
|
||||||
|
NPC, item, and location registry reconciliation translate shared proposal
|
||||||
|
safety categories into bounded prose that references only the response-local
|
||||||
|
duplicate-group ordinals and candidate handles. Item reconciliation appends its
|
||||||
|
typed rule that currency may be consolidated only with aliases of the same
|
||||||
|
denomination and never with non-currency items. The next normalize attempt
|
||||||
|
receives that prose with the exact defective proposal under the shared
|
||||||
|
replacement-request protocol. Structurally invalid output has no valid proposal
|
||||||
|
candidate and receives a fresh feedback-free attempt instead. If the stage
|
||||||
|
budget is exhausted, safe groups stay applied, unsafe groups stay separate, and
|
||||||
|
one fallback warning summarizes the final defect without raw model content.
|
||||||
|
|
||||||
## Evidence, Candidates, And Normalization
|
## Evidence, Candidates, And Normalization
|
||||||
|
|
||||||
The current transcript is the only durable evidence source. Extractors assign
|
The current transcript is the only durable evidence source. Extractors assign
|
||||||
|
|||||||
@@ -247,8 +247,12 @@ redaction boundary.
|
|||||||
Structural repair does not replace pipeline retry behavior: a binding's
|
Structural repair does not replace pipeline retry behavior: a binding's
|
||||||
configured retry count reruns its complete stage attempt after an operational
|
configured retry count reruns its complete stage attempt after an operational
|
||||||
or structural error, module-requested retry, or actionable semantic rejection.
|
or structural error, module-requested retry, or actionable semantic rejection.
|
||||||
The pipeline owns attempt lifecycle, validation chains, and retry diagnostics;
|
An actionable module-requested retry and a validator rejection both use the
|
||||||
see [Pipeline Internals](pipeline.md#validation-retries-and-output) and the
|
same correction payload when the producer exposes an exact latest response;
|
||||||
|
feedback-free module retries reconstruct the ordinary request without appended
|
||||||
|
messages. The pipeline owns attempt lifecycle, validation chains, and retry
|
||||||
|
diagnostics; see
|
||||||
|
[Pipeline Internals](pipeline.md#validation-retries-and-output) and the
|
||||||
[binding reference](../config.md#module-bindings-and-validators).
|
[binding reference](../config.md#module-bindings-and-validators).
|
||||||
|
|
||||||
## Timeout Ownership
|
## Timeout Ownership
|
||||||
|
|||||||
@@ -115,6 +115,16 @@ eligible producers. Deterministic skip, limit, and fallback outcomes carry no
|
|||||||
model candidate, so a later rejection applies terminal policy without spending
|
model candidate, so a later rejection applies terminal policy without spending
|
||||||
an ineffective semantic retry.
|
an ineffective semantic retry.
|
||||||
|
|
||||||
|
When proposal assessment or typed group application rejects a structurally
|
||||||
|
valid group, the normalizer may return its safe partial value with a
|
||||||
|
module-requested retry. A feedback-capable directive supplies bounded
|
||||||
|
model-facing correction guidance separately from operator diagnostics and
|
||||||
|
retains the exact proposal response as its candidate. The shared stage retry
|
||||||
|
mechanism appends that response and guidance to a fresh complete request. A
|
||||||
|
structurally invalid completion has no valid candidate and therefore requests a
|
||||||
|
feedback-free fresh attempt. On exhaustion, only the final safe fallback and
|
||||||
|
its bounded process diagnostic advance to validation.
|
||||||
|
|
||||||
The core supplies a conservative generic prompt and the single private
|
The core supplies a conservative generic prompt and the single private
|
||||||
response schema. A domain prompt may substitute its semantic instructions but
|
response schema. A domain prompt may substitute its semantic instructions but
|
||||||
mounts the core-owned protocol and candidate/transcript presentation assets.
|
mounts the core-owned protocol and candidate/transcript presentation assets.
|
||||||
|
|||||||
@@ -154,18 +154,25 @@ complete validation chain. It preserves terminal diagnostics only from the final
|
|||||||
or rejected attempt, plus one fixed validation-incomplete warning per validator whose execution
|
or rejected attempt, plus one fixed validation-incomplete warning per validator whose execution
|
||||||
budget was exhausted under `warn_continue`. Cancellation stops retries.
|
budget was exhausted under `warn_continue`. Cancellation stops retries.
|
||||||
Normalizer-specific retry directives consume this same budget and validate any
|
Normalizer-specific retry directives consume this same budget and validate any
|
||||||
final safe fallback through the normalizer chain.
|
final safe fallback through the normalizer chain. A directive may carry bounded
|
||||||
|
correction guidance only when it also exposes the exact latest
|
||||||
|
`single_response_v1` candidate. The state machine then uses the same replacement
|
||||||
|
request shape as validator correction. A directive without guidance clears any
|
||||||
|
prior correction and starts a fresh attempt, which preserves structural retry
|
||||||
|
behavior when no valid response exists.
|
||||||
|
|
||||||
The artifact-neutral producer-attempt state machine owns that shared budget,
|
The artifact-neutral producer-attempt state machine owns that shared budget,
|
||||||
attempt provenance, semantic-correction material, and terminal-policy
|
attempt provenance, semantic-correction material, and terminal-policy
|
||||||
selection. It accepts producer and complete-validation closures, so artifact
|
selection. It accepts producer and complete-validation closures, so artifact
|
||||||
materialization, cache handling, checkpoints, and debug output stay at the
|
materialization, cache handling, checkpoints, and debug output stay at the
|
||||||
operation boundary. It distinguishes operational, structural, module-requested,
|
operation boundary. It distinguishes operational, structural, module-requested,
|
||||||
and semantic retries. A semantic retry is available only for a valid latest
|
and validator-semantic retries. Model feedback from either semantic source is
|
||||||
`single_response_v1` candidate; a deterministic or no-model rejection instead
|
available only for a valid latest `single_response_v1` candidate. A
|
||||||
settles the semantic policy immediately. Structural-output errors alone use the
|
deterministic or no-model validator rejection instead settles the semantic
|
||||||
structural policy, and validation failure without rejection settles the
|
policy immediately, while a feedback-free module directive remains an ordinary
|
||||||
validator-failure policy without regenerating the producer.
|
fresh retry. Structural-output errors alone use the structural policy, and
|
||||||
|
validation failure without rejection settles the validator-failure policy
|
||||||
|
without regenerating the producer.
|
||||||
|
|
||||||
Chunk planning uses this state machine for generated plans. A rejected or
|
Chunk planning uses this state machine for generated plans. A rejected or
|
||||||
validation-incomplete automatic cache hit is not model material and therefore
|
validation-incomplete automatic cache hit is not model material and therefore
|
||||||
|
|||||||
@@ -201,6 +201,13 @@ they are not model instructions. The framework constructs model-facing retry
|
|||||||
text only from the semantic guidance and fails the contract rather than
|
text only from the semantic guidance and fails the contract rather than
|
||||||
inventing or truncating missing guidance.
|
inventing or truncating missing guidance.
|
||||||
|
|
||||||
|
An LLM-backed module may also request a feedback-aware retry when its own
|
||||||
|
deterministic translation or typed safety policy rejects a structurally valid
|
||||||
|
model response. It must supply model-facing guidance separately from its
|
||||||
|
reason code, operator message, and fallback diagnostics, together with the
|
||||||
|
exact `single_response_v1` candidate. A feedback-free module retry remains
|
||||||
|
valid when no exact candidate exists.
|
||||||
|
|
||||||
Default validator chains are production composition policy and are registered
|
Default validator chains are production composition policy and are registered
|
||||||
centrally by stage and module. Configuration may replace a stage-local default,
|
centrally by stage and module. Configuration may replace a stage-local default,
|
||||||
including with an explicitly empty chain. Configured validator order is
|
including with an explicitly empty chain. Configured validator order is
|
||||||
@@ -222,12 +229,14 @@ the two budgets must remain separate.
|
|||||||
|
|
||||||
An LLM-backed producer can participate in semantic correction only when it
|
An LLM-backed producer can participate in semantic correction only when it
|
||||||
declares `single_response_v1` and returns the exact one response that directly
|
declares `single_response_v1` and returns the exact one response that directly
|
||||||
controlled its candidate. On an actionable rejection, the framework rebuilds
|
controlled its candidate. On an actionable validator rejection or
|
||||||
the ordinary request and appends only the latest defective response as an
|
feedback-capable module retry, the framework rebuilds the ordinary request and
|
||||||
`assistant` message plus one aggregated `user` correction message. This is a
|
appends only the latest defective response as an `assistant` message plus one
|
||||||
fresh replacement request, not a growing conversation. The retry budgets,
|
aggregated `user` correction message. This is a fresh replacement request, not
|
||||||
terminal policy, and sensitive-data rationale are recorded in
|
a growing conversation. The retry budgets, terminal policy, and sensitive-data
|
||||||
[ADR-0014](../adr/0014-feedback-aware-validation-retries.md).
|
rationale are recorded in
|
||||||
|
[ADR-0014](../adr/0014-feedback-aware-validation-retries.md) and
|
||||||
|
[ADR-0016](../adr/0016-feedback-aware-module-requested-retries.md).
|
||||||
|
|
||||||
When a model selects an application entity, callers must supply a contextual
|
When a model selects an application entity, callers must supply a contextual
|
||||||
selection and deterministically attach the opaque application identity whenever
|
selection and deterministically attach the opaque application identity whenever
|
||||||
|
|||||||
@@ -15,7 +15,7 @@ diagnostics. The remaining near-term work applies those completed foundations
|
|||||||
to domain review and empirical evaluation.
|
to domain review and empirical evaluation.
|
||||||
|
|
||||||
The active D&D reliability work is defined by
|
The active D&D reliability work is defined by
|
||||||
[D&D Item-Occurrence Holder Reliability](item-occurrence-holder-reliability.md).
|
[Feedback-Aware Semantic-Reconciliation Retries](semantic-reconciliation-retry-feedback.md).
|
||||||
|
|
||||||
## Near-Term D&D Pipeline
|
## Near-Term D&D Pipeline
|
||||||
|
|
||||||
|
|||||||
117
docs/roadmap/semantic-reconciliation-retry-feedback.md
Normal file
117
docs/roadmap/semantic-reconciliation-retry-feedback.md
Normal file
@@ -0,0 +1,117 @@
|
|||||||
|
# Feedback-Aware Semantic-Reconciliation Retries
|
||||||
|
|
||||||
|
## Purpose
|
||||||
|
|
||||||
|
Improve semantic registry reconciliation for smaller models by turning the
|
||||||
|
existing module-requested retry into a bounded corrective exchange. Preserve
|
||||||
|
the current deterministic safety checks, shared retry budget, and safe fallback
|
||||||
|
when correction is unsuccessful.
|
||||||
|
|
||||||
|
## Problem
|
||||||
|
|
||||||
|
NPC, item, and location registry normalizers can reject unsafe duplicate-group
|
||||||
|
proposals and retain a safe partial result. They currently ask the pipeline for
|
||||||
|
another attempt, but the next attempt receives the original prompt without the
|
||||||
|
defective response or an explanation of the defect. The normalizers also build
|
||||||
|
their retry messages from internal issue-category strings, which are suitable
|
||||||
|
for diagnostics but not useful model instructions. If all attempts are
|
||||||
|
exhausted, the warning reports the omitted-group count without explaining the
|
||||||
|
final defect or confirming that affected candidates were preserved separately.
|
||||||
|
|
||||||
|
## Target State
|
||||||
|
|
||||||
|
- A normalizer retry directive may carry model-facing correction guidance
|
||||||
|
separately from its stable reason code, operator message, and fallback
|
||||||
|
diagnostics.
|
||||||
|
- When the directive includes correction guidance and a valid
|
||||||
|
`single_response_v1` model candidate, the shared producer-attempt state
|
||||||
|
machine reconstructs the complete original prompt and appends the exact
|
||||||
|
defective response as an assistant message followed by one user correction
|
||||||
|
message. Earlier failed attempts do not accumulate.
|
||||||
|
- This correction consumes the existing normalize-stage retry budget. It does
|
||||||
|
not add a module-local loop or alter PromptKit's independent structured-output
|
||||||
|
repair budget.
|
||||||
|
- A retry caused by structurally invalid output remains a fresh stage attempt
|
||||||
|
without semantic feedback when no valid model candidate is available.
|
||||||
|
- Internal issue categories remain available for operator and debug
|
||||||
|
diagnostics, but are never sent to a model. A shared renderer translates
|
||||||
|
every known category into concise prose that explains the defect and the
|
||||||
|
required valid shape.
|
||||||
|
- Item reconciliation appends its domain rule when a proposal attempts to
|
||||||
|
combine different currency denominations or currency with a non-currency
|
||||||
|
item. NPC and location reconciliation use only the shared protocol guidance.
|
||||||
|
- On final exhaustion, safe proposal groups remain applied, unsafe groups
|
||||||
|
remain separate, and one process warning gives a bounded summary of the final
|
||||||
|
defect and explicitly states that the affected candidates were preserved
|
||||||
|
separately.
|
||||||
|
|
||||||
|
## Required Work
|
||||||
|
|
||||||
|
1. Extend `contracts.NormalizeRetry` and its clone, validation, and debug paths
|
||||||
|
with an optional, bounded model-facing correction-guidance field. Keep
|
||||||
|
reason codes and operator messages out of model input. Require guidance to
|
||||||
|
be paired with a valid `single_response_v1` model candidate; preserve the
|
||||||
|
existing feedback-free directive for structural retry cases.
|
||||||
|
2. Extend the generic producer-attempt state machine so a feedback-capable
|
||||||
|
module retry constructs `SemanticCorrection` from the latest exact response
|
||||||
|
and the supplied guidance. Preserve attempt provenance, cancellation,
|
||||||
|
terminal policy, diagnostic handling, byte-faithful response provenance,
|
||||||
|
and the rule that only the latest response is included.
|
||||||
|
3. Add a model-facing renderer to `internal/framework/semanticreconcile`.
|
||||||
|
Implement it with a typed lookup table keyed by `IssueCategory` and an
|
||||||
|
authoritative list of all categories, rather than a positional array.
|
||||||
|
Render response-local duplicate-group numbers in human-friendly form,
|
||||||
|
de-duplicate repeated guidance, enforce a size bound, and return an error for
|
||||||
|
any unmapped category instead of exposing its raw value. Retain
|
||||||
|
`IssueDetails` for operator/debug diagnostics.
|
||||||
|
4. Update the NPC, item, and location registry normalizers to populate the new
|
||||||
|
correction guidance for semantically invalid proposals. Add the item-only
|
||||||
|
currency guidance after the shared prose when typed application rejects a
|
||||||
|
group. Do not add guidance to the invalid-structured-output path when no
|
||||||
|
exact valid candidate exists.
|
||||||
|
5. Improve exhausted-fallback diagnostics across all three registries. Describe
|
||||||
|
the final proposal failure accurately, state that affected candidates were
|
||||||
|
preserved separately, and keep warning samples bounded and free of raw model
|
||||||
|
responses.
|
||||||
|
6. Update the shared semantic-reconciliation and affected normalizer policy
|
||||||
|
fingerprints so checkpoints produced under blind retry behavior are not
|
||||||
|
reused. Record the durable module-retry correction decision in a new ADR,
|
||||||
|
and update the architecture, pipeline, LLM-runtime, and D&D internal docs in
|
||||||
|
their canonical scopes when the behavior is implemented.
|
||||||
|
7. Add lean offline behavioral coverage for corrective module retries, exact
|
||||||
|
latest-response use, feedback-free structural retries, contract validation,
|
||||||
|
complete category-to-prose coverage, domain-specific item guidance, bounded
|
||||||
|
messages, safe final fallback, and warning content. Avoid exact prose
|
||||||
|
snapshots, prompt-length assertions, live providers, and redundant tests of
|
||||||
|
PromptKit internals.
|
||||||
|
|
||||||
|
## Non-Goals
|
||||||
|
|
||||||
|
- Changing normalize retry counts, PromptKit repair settings, or pipeline
|
||||||
|
terminal policies.
|
||||||
|
- Changing the reconciliation prompt protocol, response schema, registry
|
||||||
|
artifact schemas, or request-local candidate IDs.
|
||||||
|
- Reintroducing provider-sensitive schema constraints such as `uniqueItems`.
|
||||||
|
- Automatically applying an unsafe duplicate proposal or treating a fallback
|
||||||
|
warning as an extraction-quality warning.
|
||||||
|
- Persisting raw model responses or correction text in ordinary warnings,
|
||||||
|
receipts, manifests, caches, or checkpoints.
|
||||||
|
|
||||||
|
## Acceptance Criteria
|
||||||
|
|
||||||
|
- A semantic proposal rejected by an NPC, item, or location registry normalizer
|
||||||
|
receives the exact defective response and actionable prose on the next
|
||||||
|
normalize attempt, using only the configured stage retry budget.
|
||||||
|
- Every `IssueCategory` has non-empty model-facing guidance, no raw category or
|
||||||
|
reason-code string reaches the model, and an unmapped future category fails
|
||||||
|
closed.
|
||||||
|
- An invalid item consolidation additionally tells the model to keep currency
|
||||||
|
denominations distinct and not mix currency with non-currency items.
|
||||||
|
- Structural-output failures without a valid model candidate retain the current
|
||||||
|
fresh-retry behavior.
|
||||||
|
- After retry exhaustion, safe consolidations remain applied, affected
|
||||||
|
candidates remain separate, and the single fallback warning explains both
|
||||||
|
the final cause and preservation behavior without including raw model output.
|
||||||
|
- Checkpoint identities change where policy changed, canonical documentation
|
||||||
|
and a new ADR describe the implemented behavior, and focused tests plus
|
||||||
|
`go test ./...`, `go vet ./...`, and `go build ./cmd/notarius` pass.
|
||||||
@@ -102,11 +102,12 @@ type TypedNormalizeResult[T any] struct {
|
|||||||
ModelCandidate *ModelCandidate
|
ModelCandidate *ModelCandidate
|
||||||
}
|
}
|
||||||
|
|
||||||
// Normalize retry diagnostic limits bound module-provided values before the
|
// Normalize retry limits bound module-provided control and diagnostic text
|
||||||
// framework persists them in debug artifacts.
|
// before the framework consumes or records it.
|
||||||
const (
|
const (
|
||||||
MaxNormalizeRetryReasonCodeBytes = 128
|
MaxNormalizeRetryReasonCodeBytes = 128
|
||||||
MaxNormalizeRetryMessageBytes = 4096
|
MaxNormalizeRetryMessageBytes = 4096
|
||||||
|
MaxNormalizeRetryCorrectionGuidanceBytes = 4096
|
||||||
)
|
)
|
||||||
|
|
||||||
// NormalizeRetry asks the framework to retry normalization while retaining a
|
// NormalizeRetry asks the framework to retry normalization while retaining a
|
||||||
@@ -114,6 +115,7 @@ const (
|
|||||||
type NormalizeRetry struct {
|
type NormalizeRetry struct {
|
||||||
ReasonCode string
|
ReasonCode string
|
||||||
Message string
|
Message string
|
||||||
|
CorrectionGuidance string
|
||||||
FallbackDiagnostics []ProducerDiagnostic
|
FallbackDiagnostics []ProducerDiagnostic
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|||||||
@@ -30,5 +30,14 @@ func validateNormalizeRetry(retry *contracts.NormalizeRetry) error {
|
|||||||
if len(retry.Message) > contracts.MaxNormalizeRetryMessageBytes {
|
if len(retry.Message) > contracts.MaxNormalizeRetryMessageBytes {
|
||||||
return errors.New("normalize retry directive message exceeds maximum length")
|
return errors.New("normalize retry directive message exceeds maximum length")
|
||||||
}
|
}
|
||||||
|
if !utf8.ValidString(retry.CorrectionGuidance) {
|
||||||
|
return errors.New("normalize retry directive correction guidance has invalid UTF-8")
|
||||||
|
}
|
||||||
|
if retry.CorrectionGuidance != "" && strings.TrimSpace(retry.CorrectionGuidance) == "" {
|
||||||
|
return errors.New("normalize retry directive correction guidance is blank")
|
||||||
|
}
|
||||||
|
if len(retry.CorrectionGuidance) > contracts.MaxNormalizeRetryCorrectionGuidanceBytes {
|
||||||
|
return errors.New("normalize retry directive correction guidance exceeds maximum length")
|
||||||
|
}
|
||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -101,6 +101,7 @@ func cloneNormalizeRetry(retry *contracts.NormalizeRetry) *contracts.NormalizeRe
|
|||||||
return &contracts.NormalizeRetry{
|
return &contracts.NormalizeRetry{
|
||||||
ReasonCode: retry.ReasonCode,
|
ReasonCode: retry.ReasonCode,
|
||||||
Message: retry.Message,
|
Message: retry.Message,
|
||||||
|
CorrectionGuidance: retry.CorrectionGuidance,
|
||||||
FallbackDiagnostics: contracts.CloneProducerDiagnostics(retry.FallbackDiagnostics),
|
FallbackDiagnostics: contracts.CloneProducerDiagnostics(retry.FallbackDiagnostics),
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -19,8 +19,9 @@ func (n retryingNotesNormalizer) Normalize(_ context.Context, req contracts.Type
|
|||||||
|
|
||||||
func TestNormalizerRegistryErasureClonesRetryDirective(t *testing.T) {
|
func TestNormalizerRegistryErasureClonesRetryDirective(t *testing.T) {
|
||||||
retry := &contracts.NormalizeRetry{
|
retry := &contracts.NormalizeRetry{
|
||||||
ReasonCode: "retryable",
|
ReasonCode: "retryable",
|
||||||
Message: "safe fallback available",
|
Message: "safe fallback available",
|
||||||
|
CorrectionGuidance: "return a complete corrected proposal",
|
||||||
}
|
}
|
||||||
registry := NewNormalizerRegistry()
|
registry := NewNormalizerRegistry()
|
||||||
if err := RegisterNormalizer(registry, ModuleSpec{Key: "test/retry-normalize", Stage: StageNormalize, ExecutionClass: contracts.ExecutionClassDeterministic, ArtifactKind: "test/notes"}, func() (contracts.Normalizer[codecNotes], error) {
|
if err := RegisterNormalizer(registry, ModuleSpec{Key: "test/retry-normalize", Stage: StageNormalize, ExecutionClass: contracts.ExecutionClassDeterministic, ArtifactKind: "test/notes"}, func() (contracts.Normalizer[codecNotes], error) {
|
||||||
@@ -41,7 +42,8 @@ func TestNormalizerRegistryErasureClonesRetryDirective(t *testing.T) {
|
|||||||
t.Fatalf("normalize() error = %v", err)
|
t.Fatalf("normalize() error = %v", err)
|
||||||
}
|
}
|
||||||
retry.Message = "mutated"
|
retry.Message = "mutated"
|
||||||
if result.Retry == nil || result.Retry.Message != "safe fallback available" {
|
retry.CorrectionGuidance = "mutated guidance"
|
||||||
|
if result.Retry == nil || result.Retry.Message != "safe fallback available" || result.Retry.CorrectionGuidance != "return a complete corrected proposal" {
|
||||||
t.Fatalf("erased retry result = %#v, want independent retry data", result)
|
t.Fatalf("erased retry result = %#v, want independent retry data", result)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -52,6 +52,7 @@ type producerAttemptRequest struct {
|
|||||||
// the current value as a safe fallback if its shared budget is exhausted.
|
// the current value as a safe fallback if its shared budget is exhausted.
|
||||||
// Artifact-specific adapters are responsible for validating and populating it.
|
// Artifact-specific adapters are responsible for validating and populating it.
|
||||||
type producerRetryDirective struct {
|
type producerRetryDirective struct {
|
||||||
|
CorrectionGuidance string
|
||||||
FallbackDiagnostics []contracts.ProducerDiagnostic
|
FallbackDiagnostics []contracts.ProducerDiagnostic
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -59,7 +60,10 @@ func (directive *producerRetryDirective) clone() *producerRetryDirective {
|
|||||||
if directive == nil {
|
if directive == nil {
|
||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
return &producerRetryDirective{FallbackDiagnostics: contracts.CloneProducerDiagnostics(directive.FallbackDiagnostics)}
|
return &producerRetryDirective{
|
||||||
|
CorrectionGuidance: directive.CorrectionGuidance,
|
||||||
|
FallbackDiagnostics: contracts.CloneProducerDiagnostics(directive.FallbackDiagnostics),
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
// producerAttemptOutput is intentionally artifact-neutral. Value remains
|
// producerAttemptOutput is intentionally artifact-neutral. Value remains
|
||||||
@@ -204,10 +208,21 @@ func runProducerAttempts(ctx context.Context, config producerAttemptConfig, prod
|
|||||||
return failedProducerAttempt(provenance), err
|
return failedProducerAttempt(provenance), err
|
||||||
}
|
}
|
||||||
if output.Retry != nil && number < attemptLimit {
|
if output.Retry != nil && number < attemptLimit {
|
||||||
|
correction, err = moduleRetryCorrection(output)
|
||||||
|
if err != nil {
|
||||||
|
provenance = append(provenance, producerAttemptProvenance{Number: number, Kind: kind, Outcome: producerAttemptFailed})
|
||||||
|
return failedProducerAttempt(provenance), err
|
||||||
|
}
|
||||||
provenance = append(provenance, producerAttemptProvenance{Number: number, Kind: kind, Outcome: producerAttemptRetried})
|
provenance = append(provenance, producerAttemptProvenance{Number: number, Kind: kind, Outcome: producerAttemptRetried})
|
||||||
kind, correction = producerAttemptModuleRetry, nil
|
kind = producerAttemptModuleRetry
|
||||||
continue
|
continue
|
||||||
}
|
}
|
||||||
|
if output.Retry != nil && output.Retry.CorrectionGuidance != "" {
|
||||||
|
if _, err := moduleRetryCorrection(output); err != nil {
|
||||||
|
provenance = append(provenance, producerAttemptProvenance{Number: number, Kind: kind, Outcome: producerAttemptFailed})
|
||||||
|
return failedProducerAttempt(provenance), err
|
||||||
|
}
|
||||||
|
}
|
||||||
if output.Retry != nil {
|
if output.Retry != nil {
|
||||||
output.Diagnostics = append(output.Diagnostics, contracts.CloneProducerDiagnostics(output.Retry.FallbackDiagnostics)...)
|
output.Diagnostics = append(output.Diagnostics, contracts.CloneProducerDiagnostics(output.Retry.FallbackDiagnostics)...)
|
||||||
}
|
}
|
||||||
@@ -269,6 +284,23 @@ func runProducerAttempts(ctx context.Context, config producerAttemptConfig, prod
|
|||||||
return failedProducerAttempt(provenance), errors.New("producer attempt budget was not exhausted deterministically")
|
return failedProducerAttempt(provenance), errors.New("producer attempt budget was not exhausted deterministically")
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func moduleRetryCorrection(output producerAttemptOutput) (*contracts.SemanticCorrection, error) {
|
||||||
|
if output.Retry == nil || output.Retry.CorrectionGuidance == "" {
|
||||||
|
return nil, nil
|
||||||
|
}
|
||||||
|
if output.Candidate == nil {
|
||||||
|
return nil, errors.New("module retry correction guidance requires a model candidate")
|
||||||
|
}
|
||||||
|
if output.Candidate.Protocol != contracts.CorrectionProtocolSingleResponseV1 {
|
||||||
|
return nil, fmt.Errorf("module retry correction guidance requires protocol %q", contracts.CorrectionProtocolSingleResponseV1)
|
||||||
|
}
|
||||||
|
correction, err := contracts.NewSemanticCorrection(output.Candidate.Response, output.Retry.CorrectionGuidance)
|
||||||
|
if err != nil {
|
||||||
|
return nil, fmt.Errorf("construct module retry semantic correction: %w", err)
|
||||||
|
}
|
||||||
|
return correction, nil
|
||||||
|
}
|
||||||
|
|
||||||
func validateProducerAttemptDiagnostics(output producerAttemptOutput) error {
|
func validateProducerAttemptDiagnostics(output producerAttemptOutput) error {
|
||||||
if err := contracts.ValidateProducerDiagnostics(output.Diagnostics); err != nil {
|
if err := contracts.ValidateProducerDiagnostics(output.Diagnostics); err != nil {
|
||||||
return fmt.Errorf("producer returned invalid diagnostics: %w", err)
|
return fmt.Errorf("producer returned invalid diagnostics: %w", err)
|
||||||
|
|||||||
@@ -218,6 +218,9 @@ func TestRunProducerAttemptsUsesModuleRetryBudgetAndFallback(t *testing.T) {
|
|||||||
calls := 0
|
calls := 0
|
||||||
terminal, err := runProducerAttempts(context.Background(), producerAttemptConfig{Retries: 1, Policy: DefaultValidationPolicy()}, func(_ context.Context, request producerAttemptRequest) (producerAttemptOutput, error) {
|
terminal, err := runProducerAttempts(context.Background(), producerAttemptConfig{Retries: 1, Policy: DefaultValidationPolicy()}, func(_ context.Context, request producerAttemptRequest) (producerAttemptOutput, error) {
|
||||||
calls++
|
calls++
|
||||||
|
if request.Correction != nil {
|
||||||
|
t.Fatalf("feedback-free module retry correction = %#v, want nil", request.Correction)
|
||||||
|
}
|
||||||
if request.Number == 1 {
|
if request.Number == 1 {
|
||||||
return producerAttemptOutput{Value: "fallback", Retry: &producerRetryDirective{}}, nil
|
return producerAttemptOutput{Value: "fallback", Retry: &producerRetryDirective{}}, nil
|
||||||
}
|
}
|
||||||
@@ -234,6 +237,58 @@ func TestRunProducerAttemptsUsesModuleRetryBudgetAndFallback(t *testing.T) {
|
|||||||
}
|
}
|
||||||
})
|
})
|
||||||
|
|
||||||
|
t.Run("feedback retry", func(t *testing.T) {
|
||||||
|
const (
|
||||||
|
defective = `{"duplicate_groups":[{"candidate_ids":[1,99],"canonical_candidate_id":1}]}`
|
||||||
|
guidance = "Use only candidate IDs from the supplied candidate list. Return one complete corrected response."
|
||||||
|
)
|
||||||
|
var observed *contracts.SemanticCorrection
|
||||||
|
terminal, err := runProducerAttempts(context.Background(), producerAttemptConfig{Retries: 1, Policy: DefaultValidationPolicy()}, func(_ context.Context, request producerAttemptRequest) (producerAttemptOutput, error) {
|
||||||
|
if request.Number == 1 {
|
||||||
|
return producerAttemptOutput{Value: "safe fallback", Candidate: attemptCandidate(t, defective), Retry: &producerRetryDirective{CorrectionGuidance: guidance}}, nil
|
||||||
|
}
|
||||||
|
observed = request.Correction
|
||||||
|
return producerAttemptOutput{Value: "corrected"}, nil
|
||||||
|
}, approveAttempt)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("runProducerAttempts() error = %v", err)
|
||||||
|
}
|
||||||
|
if terminal.Action != producerTerminalAccepted || terminal.Value != "corrected" {
|
||||||
|
t.Fatalf("terminal = %#v, want corrected accepted value", terminal)
|
||||||
|
}
|
||||||
|
if observed == nil || string(observed.AssistantResponse) != defective || observed.UserGuidance != guidance {
|
||||||
|
t.Fatalf("module retry correction = %#v, want exact latest response and guidance", observed)
|
||||||
|
}
|
||||||
|
if got := attemptKinds(terminal.Provenance); !reflect.DeepEqual(got, []producerAttemptKind{producerAttemptInitial, producerAttemptModuleRetry}) {
|
||||||
|
t.Fatalf("attempt kinds = %v", got)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
|
||||||
|
t.Run("feedback-free retry clears prior correction", func(t *testing.T) {
|
||||||
|
terminal, err := runProducerAttempts(context.Background(), producerAttemptConfig{Retries: 2, Policy: DefaultValidationPolicy()}, func(_ context.Context, request producerAttemptRequest) (producerAttemptOutput, error) {
|
||||||
|
switch request.Number {
|
||||||
|
case 1:
|
||||||
|
return producerAttemptOutput{Value: "first fallback", Candidate: attemptCandidate(t, "first defective response"), Retry: &producerRetryDirective{CorrectionGuidance: "Correct the first response."}}, nil
|
||||||
|
case 2:
|
||||||
|
if request.Correction == nil || string(request.Correction.AssistantResponse) != "first defective response" {
|
||||||
|
t.Fatalf("second attempt correction = %#v", request.Correction)
|
||||||
|
}
|
||||||
|
return producerAttemptOutput{Value: "second fallback", Retry: &producerRetryDirective{}}, nil
|
||||||
|
case 3:
|
||||||
|
if request.Correction != nil {
|
||||||
|
t.Fatalf("third attempt retained stale correction %#v", request.Correction)
|
||||||
|
}
|
||||||
|
return producerAttemptOutput{Value: "accepted"}, nil
|
||||||
|
default:
|
||||||
|
t.Fatalf("unexpected producer attempt %d", request.Number)
|
||||||
|
return producerAttemptOutput{}, nil
|
||||||
|
}
|
||||||
|
}, approveAttempt)
|
||||||
|
if err != nil || terminal.Action != producerTerminalAccepted || terminal.Value != "accepted" {
|
||||||
|
t.Fatalf("terminal = %#v, error = %v", terminal, err)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
|
||||||
t.Run("fallback", func(t *testing.T) {
|
t.Run("fallback", func(t *testing.T) {
|
||||||
fallbackDiagnostic := contracts.ProducerDiagnostic{Disposition: contracts.DiagnosticDispositionWarning, Category: contracts.DiagnosticCategoryFallback, ReasonCode: "fallback", OccurrenceCount: 1, Samples: []contracts.DiagnosticSample{{Scope: "fallback", Message: "fallback warning"}}}
|
fallbackDiagnostic := contracts.ProducerDiagnostic{Disposition: contracts.DiagnosticDispositionWarning, Category: contracts.DiagnosticCategoryFallback, ReasonCode: "fallback", OccurrenceCount: 1, Samples: []contracts.DiagnosticSample{{Scope: "fallback", Message: "fallback warning"}}}
|
||||||
terminal, err := runProducerAttempts(context.Background(), producerAttemptConfig{Policy: DefaultValidationPolicy()}, func(context.Context, producerAttemptRequest) (producerAttemptOutput, error) {
|
terminal, err := runProducerAttempts(context.Background(), producerAttemptConfig{Policy: DefaultValidationPolicy()}, func(context.Context, producerAttemptRequest) (producerAttemptOutput, error) {
|
||||||
@@ -248,6 +303,24 @@ func TestRunProducerAttemptsUsesModuleRetryBudgetAndFallback(t *testing.T) {
|
|||||||
})
|
})
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func TestRunProducerAttemptsRejectsModuleCorrectionWithoutModelCandidate(t *testing.T) {
|
||||||
|
producerCalls := 0
|
||||||
|
validatorCalls := 0
|
||||||
|
terminal, err := runProducerAttempts(context.Background(), producerAttemptConfig{Retries: 1, Policy: DefaultValidationPolicy()}, func(context.Context, producerAttemptRequest) (producerAttemptOutput, error) {
|
||||||
|
producerCalls++
|
||||||
|
return producerAttemptOutput{Value: "safe", Retry: &producerRetryDirective{CorrectionGuidance: "Return a complete corrected response."}}, nil
|
||||||
|
}, func(context.Context, producerAttemptOutput) (validationReport, error) {
|
||||||
|
validatorCalls++
|
||||||
|
return validationReport{}, nil
|
||||||
|
})
|
||||||
|
if err == nil || !strings.Contains(err.Error(), "requires a model candidate") {
|
||||||
|
t.Fatalf("runProducerAttempts() error = %v, want model-candidate contract failure", err)
|
||||||
|
}
|
||||||
|
if terminal.Action != producerTerminalFailed || producerCalls != 1 || validatorCalls != 0 {
|
||||||
|
t.Fatalf("terminal = %#v, producer calls = %d, validator calls = %d", terminal, producerCalls, validatorCalls)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
func TestRunProducerAttemptsRejectsInvalidDiagnosticsWithoutRetry(t *testing.T) {
|
func TestRunProducerAttemptsRejectsInvalidDiagnosticsWithoutRetry(t *testing.T) {
|
||||||
tests := []struct {
|
tests := []struct {
|
||||||
name string
|
name string
|
||||||
|
|||||||
@@ -153,6 +153,60 @@ func TestRunnerHandlesRetryableNormalizeFallbacks(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func TestRunnerForwardsModuleRequestedNormalizeCorrection(t *testing.T) {
|
||||||
|
const (
|
||||||
|
defective = `{"duplicate_groups":[{"candidate_ids":[1,99],"canonical_candidate_id":1}]}`
|
||||||
|
guidance = "Duplicate group 1 must use only supplied candidate IDs. Return one complete corrected response."
|
||||||
|
)
|
||||||
|
prepared := preparedAttemptDebugPipeline(t)
|
||||||
|
lane := &prepared.Steps[0].lanes[0]
|
||||||
|
lane.resolved.Normalize.Retries = 1
|
||||||
|
var observed *contracts.SemanticCorrection
|
||||||
|
calls := 0
|
||||||
|
lane.typed.normalize = func(_ context.Context, _ any, request contracts.TypedNormalizeRequest[any]) (erasedTypedResult, error) {
|
||||||
|
calls++
|
||||||
|
if request.Correction != nil {
|
||||||
|
clone, err := contracts.CloneSemanticCorrection(request.Correction)
|
||||||
|
if err != nil {
|
||||||
|
return erasedTypedResult{}, err
|
||||||
|
}
|
||||||
|
observed = clone
|
||||||
|
}
|
||||||
|
if calls == 1 {
|
||||||
|
return erasedTypedResult{
|
||||||
|
Value: codecNotes{Items: []string{"safe fallback"}},
|
||||||
|
ModelCandidate: attemptCandidate(t, defective),
|
||||||
|
Retry: &contracts.NormalizeRetry{
|
||||||
|
ReasonCode: "semantic_proposal_invalid",
|
||||||
|
Message: "operator-facing proposal diagnostic",
|
||||||
|
CorrectionGuidance: guidance,
|
||||||
|
},
|
||||||
|
}, nil
|
||||||
|
}
|
||||||
|
return erasedTypedResult{Value: codecNotes{Items: []string{"corrected"}}}, nil
|
||||||
|
}
|
||||||
|
debug := newCapturedDebugRecorder()
|
||||||
|
output, err := New().Run(context.Background(), RunInput{Prepared: prepared, RawInput: []byte("input"), Debug: debug})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("Run() error = %v", err)
|
||||||
|
}
|
||||||
|
if calls != 2 || observed == nil || string(observed.AssistantResponse) != defective || observed.UserGuidance != guidance {
|
||||||
|
t.Fatalf("normalize calls = %d correction = %#v", calls, observed)
|
||||||
|
}
|
||||||
|
if len(output.NormalizeOutputs) != 1 || len(output.Rejected) != 0 {
|
||||||
|
t.Fatalf("run output = %#v, want corrected accepted output", output)
|
||||||
|
}
|
||||||
|
var retryDebug strings.Builder
|
||||||
|
for _, name := range debug.names() {
|
||||||
|
if strings.HasPrefix(name, "normalize/notes/attempt-") && strings.HasSuffix(name, ".json") {
|
||||||
|
retryDebug.Write(debug.json[name])
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if !strings.Contains(retryDebug.String(), `"correction_available":true`) || strings.Contains(retryDebug.String(), guidance) || strings.Contains(retryDebug.String(), defective) {
|
||||||
|
t.Fatalf("retry debug = %s, want safe correction metadata without content", retryDebug.String())
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
func TestRunnerValidatesNormalizeRetryDiagnostics(t *testing.T) {
|
func TestRunnerValidatesNormalizeRetryDiagnostics(t *testing.T) {
|
||||||
const (
|
const (
|
||||||
reasonSentinel = "reason-diagnostic-sentinel"
|
reasonSentinel = "reason-diagnostic-sentinel"
|
||||||
@@ -160,9 +214,11 @@ func TestRunnerValidatesNormalizeRetryDiagnostics(t *testing.T) {
|
|||||||
)
|
)
|
||||||
reasonOverLimit := strings.Repeat("r", contracts.MaxNormalizeRetryReasonCodeBytes-len(reasonSentinel)) + reasonSentinel + "x"
|
reasonOverLimit := strings.Repeat("r", contracts.MaxNormalizeRetryReasonCodeBytes-len(reasonSentinel)) + reasonSentinel + "x"
|
||||||
messageOverLimit := strings.Repeat("m", contracts.MaxNormalizeRetryMessageBytes-len(messageSentinel)) + messageSentinel + "x"
|
messageOverLimit := strings.Repeat("m", contracts.MaxNormalizeRetryMessageBytes-len(messageSentinel)) + messageSentinel + "x"
|
||||||
|
guidanceOverLimit := strings.Repeat("g", contracts.MaxNormalizeRetryCorrectionGuidanceBytes-len(messageSentinel)) + messageSentinel + "x"
|
||||||
tests := []struct {
|
tests := []struct {
|
||||||
name string
|
name string
|
||||||
retry contracts.NormalizeRetry
|
retry contracts.NormalizeRetry
|
||||||
|
candidate *contracts.ModelCandidate
|
||||||
wantError string
|
wantError string
|
||||||
hiddenValues []string
|
hiddenValues []string
|
||||||
}{
|
}{
|
||||||
@@ -173,6 +229,15 @@ func TestRunnerValidatesNormalizeRetryDiagnostics(t *testing.T) {
|
|||||||
Message: strings.Repeat("m", contracts.MaxNormalizeRetryMessageBytes),
|
Message: strings.Repeat("m", contracts.MaxNormalizeRetryMessageBytes),
|
||||||
},
|
},
|
||||||
},
|
},
|
||||||
|
{
|
||||||
|
name: "accepts correction guidance byte limit with candidate",
|
||||||
|
retry: contracts.NormalizeRetry{
|
||||||
|
ReasonCode: reasonSentinel,
|
||||||
|
Message: messageSentinel,
|
||||||
|
CorrectionGuidance: strings.Repeat("g", contracts.MaxNormalizeRetryCorrectionGuidanceBytes),
|
||||||
|
},
|
||||||
|
candidate: attemptCandidate(t, `{"duplicate_groups":[]}`),
|
||||||
|
},
|
||||||
{
|
{
|
||||||
name: "rejects oversized reason code",
|
name: "rejects oversized reason code",
|
||||||
retry: contracts.NormalizeRetry{
|
retry: contracts.NormalizeRetry{
|
||||||
@@ -209,6 +274,46 @@ func TestRunnerValidatesNormalizeRetryDiagnostics(t *testing.T) {
|
|||||||
wantError: "message has invalid UTF-8",
|
wantError: "message has invalid UTF-8",
|
||||||
hiddenValues: []string{reasonSentinel, messageSentinel},
|
hiddenValues: []string{reasonSentinel, messageSentinel},
|
||||||
},
|
},
|
||||||
|
{
|
||||||
|
name: "rejects oversized correction guidance",
|
||||||
|
retry: contracts.NormalizeRetry{
|
||||||
|
ReasonCode: reasonSentinel,
|
||||||
|
Message: messageSentinel,
|
||||||
|
CorrectionGuidance: guidanceOverLimit,
|
||||||
|
},
|
||||||
|
wantError: "correction guidance exceeds maximum length",
|
||||||
|
hiddenValues: []string{reasonSentinel, messageSentinel},
|
||||||
|
},
|
||||||
|
{
|
||||||
|
name: "rejects invalid correction guidance UTF-8",
|
||||||
|
retry: contracts.NormalizeRetry{
|
||||||
|
ReasonCode: reasonSentinel,
|
||||||
|
Message: messageSentinel,
|
||||||
|
CorrectionGuidance: messageSentinel + string([]byte{0xff}),
|
||||||
|
},
|
||||||
|
wantError: "correction guidance has invalid UTF-8",
|
||||||
|
hiddenValues: []string{reasonSentinel, messageSentinel},
|
||||||
|
},
|
||||||
|
{
|
||||||
|
name: "rejects blank correction guidance",
|
||||||
|
retry: contracts.NormalizeRetry{
|
||||||
|
ReasonCode: reasonSentinel,
|
||||||
|
Message: messageSentinel,
|
||||||
|
CorrectionGuidance: " \t\n ",
|
||||||
|
},
|
||||||
|
wantError: "correction guidance is blank",
|
||||||
|
hiddenValues: []string{reasonSentinel, messageSentinel},
|
||||||
|
},
|
||||||
|
{
|
||||||
|
name: "rejects correction guidance without candidate",
|
||||||
|
retry: contracts.NormalizeRetry{
|
||||||
|
ReasonCode: reasonSentinel,
|
||||||
|
Message: messageSentinel,
|
||||||
|
CorrectionGuidance: "Return one complete corrected response.",
|
||||||
|
},
|
||||||
|
wantError: "requires a model candidate",
|
||||||
|
hiddenValues: []string{reasonSentinel, messageSentinel},
|
||||||
|
},
|
||||||
{
|
{
|
||||||
name: "rejects blank reason code",
|
name: "rejects blank reason code",
|
||||||
retry: contracts.NormalizeRetry{
|
retry: contracts.NormalizeRetry{
|
||||||
@@ -233,7 +338,7 @@ func TestRunnerValidatesNormalizeRetryDiagnostics(t *testing.T) {
|
|||||||
t.Run(tc.name, func(t *testing.T) {
|
t.Run(tc.name, func(t *testing.T) {
|
||||||
prepared := preparedAttemptDebugPipeline(t)
|
prepared := preparedAttemptDebugPipeline(t)
|
||||||
prepared.Steps[0].lanes[0].typed.normalize = func(context.Context, any, contracts.TypedNormalizeRequest[any]) (erasedTypedResult, error) {
|
prepared.Steps[0].lanes[0].typed.normalize = func(context.Context, any, contracts.TypedNormalizeRequest[any]) (erasedTypedResult, error) {
|
||||||
return erasedTypedResult{Value: codecNotes{Items: []string{"safe"}}, Retry: &tc.retry}, nil
|
return erasedTypedResult{Value: codecNotes{Items: []string{"safe"}}, Retry: &tc.retry, ModelCandidate: tc.candidate}, nil
|
||||||
}
|
}
|
||||||
debug := newCapturedDebugRecorder()
|
debug := newCapturedDebugRecorder()
|
||||||
output, err := New().Run(context.Background(), RunInput{Prepared: prepared, RawInput: []byte("input"), Debug: debug})
|
output, err := New().Run(context.Background(), RunInput{Prepared: prepared, RawInput: []byte("input"), Debug: debug})
|
||||||
|
|||||||
@@ -451,8 +451,8 @@ func (r *Runner) runNormalizeStage(ctx context.Context, input RunInput, checkpoi
|
|||||||
return producerAttemptOutput{}, terminal.record(map[string]any{"output": debugCheckpointArtifact(serializedCandidate)}, fmt.Errorf("normalize lane %q returned invalid retry directive: %w", lane.ID, err))
|
return producerAttemptOutput{}, terminal.record(map[string]any{"output": debugCheckpointArtifact(serializedCandidate)}, fmt.Errorf("normalize lane %q returned invalid retry directive: %w", lane.ID, err))
|
||||||
}
|
}
|
||||||
anotherAttempt := request.Number <= lane.Normalize.Retries
|
anotherAttempt := request.Number <= lane.Normalize.Retries
|
||||||
attemptValue.retry = map[string]any{"reason_code": result.Retry.ReasonCode, "message": result.Retry.Message, "another_attempt": anotherAttempt, "fallback_accepted": !anotherAttempt}
|
attemptValue.retry = map[string]any{"reason_code": result.Retry.ReasonCode, "message": result.Retry.Message, "correction_available": result.Retry.CorrectionGuidance != "", "another_attempt": anotherAttempt, "fallback_accepted": !anotherAttempt}
|
||||||
directive = &producerRetryDirective{FallbackDiagnostics: contracts.CloneProducerDiagnostics(result.Retry.FallbackDiagnostics)}
|
directive = &producerRetryDirective{CorrectionGuidance: result.Retry.CorrectionGuidance, FallbackDiagnostics: contracts.CloneProducerDiagnostics(result.Retry.FallbackDiagnostics)}
|
||||||
if anotherAttempt {
|
if anotherAttempt {
|
||||||
payload := map[string]any{"output": debugCheckpointArtifact(serializedCandidate), "retry": attemptValue.retry}
|
payload := map[string]any{"output": debugCheckpointArtifact(serializedCandidate), "retry": attemptValue.retry}
|
||||||
if debugErr := terminal.record(payload, nil); debugErr != nil {
|
if debugErr := terminal.record(payload, nil); debugErr != nil {
|
||||||
|
|||||||
@@ -13,7 +13,7 @@ import (
|
|||||||
)
|
)
|
||||||
|
|
||||||
// Policy identifies the framework-owned reconciliation and assessment rules.
|
// Policy identifies the framework-owned reconciliation and assessment rules.
|
||||||
const Policy = "semantic_reconciliation.v1"
|
const Policy = "semantic_reconciliation.v2"
|
||||||
|
|
||||||
var _ contracts.ManifestMetadataProvider = (*Engine)(nil)
|
var _ contracts.ManifestMetadataProvider = (*Engine)(nil)
|
||||||
var _ pipeline.CheckpointFingerprintProvider = (*Engine)(nil)
|
var _ pipeline.CheckpointFingerprintProvider = (*Engine)(nil)
|
||||||
|
|||||||
@@ -3,6 +3,10 @@ package semanticreconcile
|
|||||||
import (
|
import (
|
||||||
"fmt"
|
"fmt"
|
||||||
"sort"
|
"sort"
|
||||||
|
"strings"
|
||||||
|
"unicode/utf8"
|
||||||
|
|
||||||
|
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
|
||||||
)
|
)
|
||||||
|
|
||||||
// ProposalResponse is the complete private structured response contract.
|
// ProposalResponse is the complete private structured response contract.
|
||||||
@@ -31,6 +35,28 @@ const (
|
|||||||
IssueOverlappingMember IssueCategory = "overlapping_member"
|
IssueOverlappingMember IssueCategory = "overlapping_member"
|
||||||
)
|
)
|
||||||
|
|
||||||
|
var allIssueCategories = []IssueCategory{
|
||||||
|
IssueMemberNonPositive,
|
||||||
|
IssueMemberUnknown,
|
||||||
|
IssueRepeatedMember,
|
||||||
|
IssueFewerThanTwoMembers,
|
||||||
|
IssueCanonicalNonPositive,
|
||||||
|
IssueCanonicalUnknown,
|
||||||
|
IssueCanonicalNotMember,
|
||||||
|
IssueOverlappingMember,
|
||||||
|
}
|
||||||
|
|
||||||
|
var issueCorrectionProse = map[IssueCategory]string{
|
||||||
|
IssueMemberNonPositive: "Use only positive candidate IDs from the supplied candidate list.",
|
||||||
|
IssueMemberUnknown: "Remove every candidate ID that is not present in the supplied candidate list.",
|
||||||
|
IssueRepeatedMember: "List each candidate ID at most once within the duplicate group.",
|
||||||
|
IssueFewerThanTwoMembers: "Include at least two distinct candidate IDs, or omit the duplicate group.",
|
||||||
|
IssueCanonicalNonPositive: "Choose a positive canonical_candidate_id from the supplied candidate list.",
|
||||||
|
IssueCanonicalUnknown: "Choose canonical_candidate_id from the supplied candidate list.",
|
||||||
|
IssueCanonicalNotMember: "Make canonical_candidate_id one of the candidate_ids in the same duplicate group.",
|
||||||
|
IssueOverlappingMember: "Place each candidate ID in at most one duplicate group.",
|
||||||
|
}
|
||||||
|
|
||||||
// Issue identifies an unsafe proposal category at its original response group
|
// Issue identifies an unsafe proposal category at its original response group
|
||||||
// index without prescribing caller diagnostic text.
|
// index without prescribing caller diagnostic text.
|
||||||
type Issue struct {
|
type Issue struct {
|
||||||
@@ -38,8 +64,8 @@ type Issue struct {
|
|||||||
Category IssueCategory
|
Category IssueCategory
|
||||||
}
|
}
|
||||||
|
|
||||||
// IssueDetails renders stable, domain-neutral proposal diagnostics for an
|
// IssueDetails renders stable, domain-neutral proposal diagnostics for
|
||||||
// adapter's retry message.
|
// operators and debug records. Its internal categories are not model guidance.
|
||||||
func IssueDetails(issues []Issue) []string {
|
func IssueDetails(issues []Issue) []string {
|
||||||
details := make([]string, len(issues))
|
details := make([]string, len(issues))
|
||||||
for index, issue := range issues {
|
for index, issue := range issues {
|
||||||
@@ -48,6 +74,94 @@ func IssueDetails(issues []Issue) []string {
|
|||||||
return details
|
return details
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// CorrectionDetails translates proposal issues into stable model-facing prose.
|
||||||
|
// The response-local group ordinals help the model find the defective group in
|
||||||
|
// the exact response appended to the correction request.
|
||||||
|
func CorrectionDetails(issues []Issue) ([]string, error) {
|
||||||
|
groupsByCategory := make(map[IssueCategory][]int)
|
||||||
|
seen := make(map[IssueCategory]map[int]struct{})
|
||||||
|
for _, issue := range issues {
|
||||||
|
if issue.GroupIndex < 0 {
|
||||||
|
return nil, fmt.Errorf("semantic reconciliation issue group index must not be negative")
|
||||||
|
}
|
||||||
|
if _, exists := issueCorrectionProse[issue.Category]; !exists {
|
||||||
|
return nil, fmt.Errorf("semantic reconciliation issue category %q has no correction guidance", issue.Category)
|
||||||
|
}
|
||||||
|
if seen[issue.Category] == nil {
|
||||||
|
seen[issue.Category] = make(map[int]struct{})
|
||||||
|
}
|
||||||
|
if _, exists := seen[issue.Category][issue.GroupIndex]; exists {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
seen[issue.Category][issue.GroupIndex] = struct{}{}
|
||||||
|
groupsByCategory[issue.Category] = append(groupsByCategory[issue.Category], issue.GroupIndex)
|
||||||
|
}
|
||||||
|
|
||||||
|
details := make([]string, 0, len(groupsByCategory))
|
||||||
|
for _, category := range allIssueCategories {
|
||||||
|
groups := groupsByCategory[category]
|
||||||
|
if len(groups) == 0 {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
sort.Ints(groups)
|
||||||
|
details = append(details, fmt.Sprintf("%s: %s", correctionGroupLabel(groups), issueCorrectionProse[category]))
|
||||||
|
}
|
||||||
|
return details, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// CorrectionGuidance builds one bounded request for a complete corrected
|
||||||
|
// proposal. Additional details let a typed owner append a domain rule without
|
||||||
|
// weakening or duplicating the shared protocol guidance.
|
||||||
|
func CorrectionGuidance(issues []Issue, additionalDetails ...string) (string, error) {
|
||||||
|
details, err := CorrectionDetails(issues)
|
||||||
|
if err != nil {
|
||||||
|
return "", err
|
||||||
|
}
|
||||||
|
for _, detail := range additionalDetails {
|
||||||
|
if !utf8.ValidString(detail) {
|
||||||
|
return "", fmt.Errorf("semantic reconciliation additional correction detail has invalid UTF-8")
|
||||||
|
}
|
||||||
|
detail = strings.TrimSpace(detail)
|
||||||
|
if detail == "" {
|
||||||
|
return "", fmt.Errorf("semantic reconciliation additional correction detail must not be blank")
|
||||||
|
}
|
||||||
|
details = append(details, detail)
|
||||||
|
}
|
||||||
|
if len(details) == 0 {
|
||||||
|
return "", fmt.Errorf("semantic reconciliation correction guidance requires at least one detail")
|
||||||
|
}
|
||||||
|
|
||||||
|
parts := make([]string, 0, len(details)+2)
|
||||||
|
parts = append(parts, "The previous semantic-duplicate proposal was invalid.")
|
||||||
|
parts = append(parts, details...)
|
||||||
|
parts = append(parts, "Return one complete corrected JSON response that follows the original instructions; do not return a patch or commentary.")
|
||||||
|
guidance := strings.Join(parts, " ")
|
||||||
|
if len(guidance) > contracts.MaxNormalizeRetryCorrectionGuidanceBytes {
|
||||||
|
return "", fmt.Errorf("semantic reconciliation correction guidance exceeds maximum length")
|
||||||
|
}
|
||||||
|
return guidance, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func correctionGroupLabel(groupIndexes []int) string {
|
||||||
|
const maximumDisplayedGroups = 12
|
||||||
|
displayed := groupIndexes
|
||||||
|
if len(displayed) > maximumDisplayedGroups {
|
||||||
|
displayed = displayed[:maximumDisplayedGroups]
|
||||||
|
}
|
||||||
|
ordinals := make([]string, len(displayed))
|
||||||
|
for index, groupIndex := range displayed {
|
||||||
|
ordinals[index] = fmt.Sprintf("%d", groupIndex+1)
|
||||||
|
}
|
||||||
|
if len(groupIndexes) == 1 {
|
||||||
|
return "Duplicate group " + ordinals[0]
|
||||||
|
}
|
||||||
|
label := "Duplicate groups " + strings.Join(ordinals, ", ")
|
||||||
|
if omitted := len(groupIndexes) - len(displayed); omitted > 0 {
|
||||||
|
label += fmt.Sprintf(", and %d additional affected group(s)", omitted)
|
||||||
|
}
|
||||||
|
return label
|
||||||
|
}
|
||||||
|
|
||||||
// PlanGroup identifies one validated group using original candidate positions.
|
// PlanGroup identifies one validated group using original candidate positions.
|
||||||
type PlanGroup struct {
|
type PlanGroup struct {
|
||||||
memberPositions []int
|
memberPositions []int
|
||||||
|
|||||||
@@ -2,9 +2,11 @@ package semanticreconcile
|
|||||||
|
|
||||||
import (
|
import (
|
||||||
"reflect"
|
"reflect"
|
||||||
|
"strings"
|
||||||
"testing"
|
"testing"
|
||||||
|
|
||||||
"gitea.maximumdirect.net/eric/notarius/internal/core/source"
|
"gitea.maximumdirect.net/eric/notarius/internal/core/source"
|
||||||
|
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
|
||||||
)
|
)
|
||||||
|
|
||||||
func TestAssessProducesAStableOriginalPositionPlan(t *testing.T) {
|
func TestAssessProducesAStableOriginalPositionPlan(t *testing.T) {
|
||||||
@@ -50,6 +52,72 @@ func TestIssueDetailsPreservesIssueOrder(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func TestCorrectionGuidanceCoversEveryIssueCategoryWithoutExposingInternalLabels(t *testing.T) {
|
||||||
|
if len(issueCorrectionProse) != len(allIssueCategories) {
|
||||||
|
t.Fatalf("correction prose entries = %d, categories = %d", len(issueCorrectionProse), len(allIssueCategories))
|
||||||
|
}
|
||||||
|
issues := make([]Issue, len(allIssueCategories))
|
||||||
|
seen := make(map[IssueCategory]struct{}, len(allIssueCategories))
|
||||||
|
for index, category := range allIssueCategories {
|
||||||
|
if _, duplicate := seen[category]; duplicate {
|
||||||
|
t.Fatalf("duplicate authoritative issue category %q", category)
|
||||||
|
}
|
||||||
|
seen[category] = struct{}{}
|
||||||
|
if strings.TrimSpace(issueCorrectionProse[category]) == "" {
|
||||||
|
t.Fatalf("issue category %q has no model-facing prose", category)
|
||||||
|
}
|
||||||
|
issues[index] = Issue{GroupIndex: index, Category: category}
|
||||||
|
}
|
||||||
|
|
||||||
|
details, err := CorrectionDetails(issues)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("CorrectionDetails() error = %v", err)
|
||||||
|
}
|
||||||
|
guidance, err := CorrectionGuidance(issues)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("CorrectionGuidance() error = %v", err)
|
||||||
|
}
|
||||||
|
if len(details) != len(allIssueCategories) || !strings.Contains(guidance, "Duplicate group 1") || !strings.Contains(guidance, "complete corrected JSON response") || len(guidance) > contracts.MaxNormalizeRetryCorrectionGuidanceBytes {
|
||||||
|
t.Fatalf("correction details = %#v guidance = %q", details, guidance)
|
||||||
|
}
|
||||||
|
for _, category := range allIssueCategories {
|
||||||
|
if strings.Contains(guidance, string(category)) {
|
||||||
|
t.Fatalf("model guidance exposed internal category %q: %q", category, guidance)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestCorrectionDetailsDeduplicatesAndBoundsAffectedGroupLists(t *testing.T) {
|
||||||
|
issues := make([]Issue, 0, 257)
|
||||||
|
for group := 0; group < 256; group++ {
|
||||||
|
issues = append(issues, Issue{GroupIndex: group, Category: IssueMemberUnknown})
|
||||||
|
}
|
||||||
|
issues = append(issues, Issue{GroupIndex: 0, Category: IssueMemberUnknown})
|
||||||
|
|
||||||
|
details, err := CorrectionDetails(issues)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("CorrectionDetails() error = %v", err)
|
||||||
|
}
|
||||||
|
if len(details) != 1 || !strings.Contains(details[0], "additional affected group") || strings.Count(details[0], "Duplicate groups") != 1 {
|
||||||
|
t.Fatalf("CorrectionDetails() = %#v, want one bounded grouped detail", details)
|
||||||
|
}
|
||||||
|
guidance, err := CorrectionGuidance(issues)
|
||||||
|
if err != nil || len(guidance) > contracts.MaxNormalizeRetryCorrectionGuidanceBytes {
|
||||||
|
t.Fatalf("CorrectionGuidance() = %q, %v", guidance, err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestCorrectionDetailsRejectsUnknownOrInvalidIssues(t *testing.T) {
|
||||||
|
for _, issues := range [][]Issue{
|
||||||
|
{{GroupIndex: 0, Category: "future_unmapped_category"}},
|
||||||
|
{{GroupIndex: -1, Category: IssueMemberUnknown}},
|
||||||
|
} {
|
||||||
|
if details, err := CorrectionDetails(issues); err == nil || details != nil {
|
||||||
|
t.Fatalf("CorrectionDetails(%#v) = %#v, %v; want fail-closed error", issues, details, err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
func TestAssessRejectsEveryUnsafeLocalGroupShape(t *testing.T) {
|
func TestAssessRejectsEveryUnsafeLocalGroupShape(t *testing.T) {
|
||||||
preparation := proposalPreparation(t)
|
preparation := proposalPreparation(t)
|
||||||
tests := []struct {
|
tests := []struct {
|
||||||
|
|||||||
@@ -23,7 +23,7 @@ const (
|
|||||||
Key = "dnd/item-registry"
|
Key = "dnd/item-registry"
|
||||||
PromptID = "dnd.item_registry.normalize"
|
PromptID = "dnd.item_registry.normalize"
|
||||||
PromptVersion = "v1"
|
PromptVersion = "v1"
|
||||||
normalizationPolicy = "dnd.item_registry.normalize.v3"
|
normalizationPolicy = "dnd.item_registry.normalize.v4"
|
||||||
NormalizationPolicy = normalizationPolicy
|
NormalizationPolicy = normalizationPolicy
|
||||||
|
|
||||||
ReasonCodeItemFieldsNormalized = "item_fields_normalized"
|
ReasonCodeItemFieldsNormalized = "item_fields_normalized"
|
||||||
@@ -129,7 +129,7 @@ func (n *Normalizer) Normalize(ctx context.Context, req contracts.TypedNormalize
|
|||||||
case semanticreconcile.SkippedInsufficientCandidates:
|
case semanticreconcile.SkippedInsufficientCandidates:
|
||||||
return normalizationResult(deterministic, findings, nil, nil)
|
return normalizationResult(deterministic, findings, nil, nil)
|
||||||
case semanticreconcile.SkippedLimitExceeded:
|
case semanticreconcile.SkippedLimitExceeded:
|
||||||
return fallbackResult(deterministic, findings, nil, semanticFallbackFinding(-1))
|
return fallbackResult(deterministic, findings, nil, semanticFallbackFinding(-1, "the candidate set exceeded the configured semantic-reconciliation limits"))
|
||||||
case semanticreconcile.RetryableInvalidStructuredOutput:
|
case semanticreconcile.RetryableInvalidStructuredOutput:
|
||||||
return n.invalidStructuredResult(deterministic, findings)
|
return n.invalidStructuredResult(deterministic, findings)
|
||||||
case semanticreconcile.Complete, semanticreconcile.RetryableDiscardedProposalGroups:
|
case semanticreconcile.Complete, semanticreconcile.RetryableDiscardedProposalGroups:
|
||||||
@@ -150,22 +150,38 @@ func (n *Normalizer) Normalize(ctx context.Context, req contracts.TypedNormalize
|
|||||||
}
|
}
|
||||||
|
|
||||||
func (n *Normalizer) invalidStructuredResult(value dnd.ItemRegistry, findings []diagnostics.Finding) (contracts.TypedNormalizeResult[dnd.ItemRegistry], error) {
|
func (n *Normalizer) invalidStructuredResult(value dnd.ItemRegistry, findings []diagnostics.Finding) (contracts.TypedNormalizeResult[dnd.ItemRegistry], error) {
|
||||||
return retryResultWithFallback(value, findings, nil, nil, ReasonCodeItemSemanticRetryProposalInvalid, "semantic proposal requires retry: invalid structured output", semanticFallbackFinding(-1))
|
return retryResultWithFallback(value, findings, nil, nil, ReasonCodeItemSemanticRetryProposalInvalid, "", "semantic proposal requires retry: invalid structured output", semanticFallbackFinding(-1, "the final model response did not satisfy the structured-output contract"))
|
||||||
}
|
}
|
||||||
|
|
||||||
func retryResult(value dnd.ItemRegistry, findings, advisoryFindings []diagnostics.Finding, reconciliation semanticreconcile.Result, rejectedGroups int) (contracts.TypedNormalizeResult[dnd.ItemRegistry], error) {
|
func retryResult(value dnd.ItemRegistry, findings, advisoryFindings []diagnostics.Finding, reconciliation semanticreconcile.Result, rejectedGroups int) (contracts.TypedNormalizeResult[dnd.ItemRegistry], error) {
|
||||||
details := semanticreconcile.IssueDetails(reconciliation.Issues())
|
issues := reconciliation.Issues()
|
||||||
|
operatorDetails := semanticreconcile.IssueDetails(issues)
|
||||||
|
correctionDetails, err := semanticreconcile.CorrectionDetails(issues)
|
||||||
|
if err != nil {
|
||||||
|
return contracts.TypedNormalizeResult[dnd.ItemRegistry]{}, normalizerErrorf("build semantic correction details: %w", err)
|
||||||
|
}
|
||||||
|
var additionalGuidance []string
|
||||||
if rejectedGroups > 0 {
|
if rejectedGroups > 0 {
|
||||||
details = append(details, "currency may only be consolidated with aliases of one denomination")
|
const currencyGuidance = "Consolidate currency only with aliases of the same denomination; keep different denominations separate and do not combine currency with non-currency items."
|
||||||
|
operatorDetails = append(operatorDetails, "currency may only be consolidated with aliases of one denomination")
|
||||||
|
correctionDetails = append(correctionDetails, currencyGuidance)
|
||||||
|
additionalGuidance = append(additionalGuidance, currencyGuidance)
|
||||||
|
}
|
||||||
|
correctionGuidance, err := semanticreconcile.CorrectionGuidance(issues, additionalGuidance...)
|
||||||
|
if err != nil {
|
||||||
|
return contracts.TypedNormalizeResult[dnd.ItemRegistry]{}, normalizerErrorf("build semantic correction guidance: %w", err)
|
||||||
}
|
}
|
||||||
discardedGroups := reconciliation.DiscardedGroupCount() + rejectedGroups
|
discardedGroups := reconciliation.DiscardedGroupCount() + rejectedGroups
|
||||||
return retryResultWithFallback(value, findings, advisoryFindings, reconciliation.ModelCandidate(), ReasonCodeItemSemanticRetryProposalInvalid, diagnostics.Aggregate("semantic proposal requires retry", details), semanticFallbackFinding(discardedGroups))
|
return retryResultWithFallback(value, findings, advisoryFindings, reconciliation.ModelCandidate(), ReasonCodeItemSemanticRetryProposalInvalid, correctionGuidance, diagnostics.Aggregate("semantic proposal requires retry", operatorDetails), semanticFallbackFinding(discardedGroups, correctionDetails...))
|
||||||
}
|
}
|
||||||
|
|
||||||
func semanticFallbackFinding(discarded int) diagnostics.Finding {
|
func semanticFallbackFinding(discarded int, details ...string) diagnostics.Finding {
|
||||||
message := "semantic proposal could not be applied"
|
message := "semantic reconciliation could not be completed; all affected candidates were preserved separately"
|
||||||
if discarded >= 0 {
|
if discarded >= 0 {
|
||||||
message = fmt.Sprintf("%d proposal group(s) omitted after semantic proposal retry exhaustion", discarded)
|
message = fmt.Sprintf("%d proposal group(s) omitted from the final semantic proposal; all affected candidates were preserved separately", discarded)
|
||||||
|
}
|
||||||
|
if len(details) > 0 {
|
||||||
|
message = diagnostics.Aggregate(message, details)
|
||||||
}
|
}
|
||||||
return diagnostics.Finding{Scope: "items", ReasonCode: ReasonCodeItemSemanticReconciliationExhausted, Message: message}
|
return diagnostics.Finding{Scope: "items", ReasonCode: ReasonCodeItemSemanticReconciliationExhausted, Message: message}
|
||||||
}
|
}
|
||||||
@@ -196,7 +212,7 @@ func fallbackResult(value dnd.ItemRegistry, findings, advisoryFindings []diagnos
|
|||||||
return result, nil
|
return result, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
func retryResultWithFallback(value dnd.ItemRegistry, findings, advisoryFindings []diagnostics.Finding, candidate *contracts.ModelCandidate, reasonCode, message string, fallback diagnostics.Finding) (contracts.TypedNormalizeResult[dnd.ItemRegistry], error) {
|
func retryResultWithFallback(value dnd.ItemRegistry, findings, advisoryFindings []diagnostics.Finding, candidate *contracts.ModelCandidate, reasonCode, correctionGuidance, message string, fallback diagnostics.Finding) (contracts.TypedNormalizeResult[dnd.ItemRegistry], error) {
|
||||||
result, err := normalizationResult(value, findings, advisoryFindings, candidate)
|
result, err := normalizationResult(value, findings, advisoryFindings, candidate)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return contracts.TypedNormalizeResult[dnd.ItemRegistry]{}, err
|
return contracts.TypedNormalizeResult[dnd.ItemRegistry]{}, err
|
||||||
@@ -205,7 +221,7 @@ func retryResultWithFallback(value dnd.ItemRegistry, findings, advisoryFindings
|
|||||||
if err != nil {
|
if err != nil {
|
||||||
return contracts.TypedNormalizeResult[dnd.ItemRegistry]{}, normalizerErrorf("collect fallback diagnostic: %w", err)
|
return contracts.TypedNormalizeResult[dnd.ItemRegistry]{}, normalizerErrorf("collect fallback diagnostic: %w", err)
|
||||||
}
|
}
|
||||||
result.Retry = &contracts.NormalizeRetry{ReasonCode: reasonCode, Message: message, FallbackDiagnostics: fallbackGroups}
|
result.Retry = &contracts.NormalizeRetry{ReasonCode: reasonCode, Message: message, CorrectionGuidance: correctionGuidance, FallbackDiagnostics: fallbackGroups}
|
||||||
return result, nil
|
return result, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|||||||
@@ -289,7 +289,10 @@ func TestNormalizeAppliesIndependentGroupAndCountsAllOmissions(t *testing.T) {
|
|||||||
if !hasDiagnostic(result.Diagnostics, ReasonCodeDuplicateItemCollapsed, contracts.DiagnosticDispositionObservation) || !hasDiagnostic(result.Diagnostics, ReasonCodeItemSemanticProposalInvalid, contracts.DiagnosticDispositionAdvisory) {
|
if !hasDiagnostic(result.Diagnostics, ReasonCodeDuplicateItemCollapsed, contracts.DiagnosticDispositionObservation) || !hasDiagnostic(result.Diagnostics, ReasonCodeItemSemanticProposalInvalid, contracts.DiagnosticDispositionAdvisory) {
|
||||||
t.Fatalf("diagnostics = %#v, want accepted and guarded-group diagnostics", result.Diagnostics)
|
t.Fatalf("diagnostics = %#v, want accepted and guarded-group diagnostics", result.Diagnostics)
|
||||||
}
|
}
|
||||||
if len(result.Retry.FallbackDiagnostics) != 1 || !strings.Contains(result.Retry.FallbackDiagnostics[0].Samples[0].Message, "2 proposal group(s)") {
|
if !strings.Contains(result.Retry.CorrectionGuidance, "aliases of the same denomination") || !strings.Contains(result.Retry.CorrectionGuidance, "do not combine currency with non-currency items") || !strings.Contains(result.Retry.CorrectionGuidance, "supplied candidate list") || strings.Contains(result.Retry.CorrectionGuidance, "member_unknown") || strings.Contains(result.Retry.CorrectionGuidance, ReasonCodeItemSemanticRetryProposalInvalid) {
|
||||||
|
t.Fatalf("correction guidance = %q, want shared and item-specific prose only", result.Retry.CorrectionGuidance)
|
||||||
|
}
|
||||||
|
if len(result.Retry.FallbackDiagnostics) != 1 || !strings.Contains(result.Retry.FallbackDiagnostics[0].Samples[0].Message, "2 proposal group(s)") || !strings.Contains(result.Retry.FallbackDiagnostics[0].Samples[0].Message, "preserved separately") || !strings.Contains(result.Retry.FallbackDiagnostics[0].Samples[0].Message, "same denomination") {
|
||||||
t.Fatalf("retry = %#v, want one guarded and one malformed group counted", result.Retry)
|
t.Fatalf("retry = %#v, want one guarded and one malformed group counted", result.Retry)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
@@ -321,6 +324,9 @@ func TestNormalizeRetryFallbackErrorsAndIdempotence(t *testing.T) {
|
|||||||
if err != nil || invalid.Retry == nil || invalid.Retry.ReasonCode != ReasonCodeItemSemanticRetryProposalInvalid {
|
if err != nil || invalid.Retry == nil || invalid.Retry.ReasonCode != ReasonCodeItemSemanticRetryProposalInvalid {
|
||||||
t.Fatalf("invalid result = %#v, %v", invalid, err)
|
t.Fatalf("invalid result = %#v, %v", invalid, err)
|
||||||
}
|
}
|
||||||
|
if invalid.Retry.CorrectionGuidance != "" || !strings.Contains(invalid.Retry.FallbackDiagnostics[0].Samples[0].Message, "structured-output contract") {
|
||||||
|
t.Fatalf("invalid structured retry = %#v, want feedback-free retry with final-cause fallback", invalid.Retry)
|
||||||
|
}
|
||||||
_, err = newNormalizer(t, &recordingNormalizerClient{err: errors.New("provider unavailable")}).Normalize(context.Background(), normalizeRequestWithSource(input, doc))
|
_, err = newNormalizer(t, &recordingNormalizerClient{err: errors.New("provider unavailable")}).Normalize(context.Background(), normalizeRequestWithSource(input, doc))
|
||||||
if err == nil || !strings.Contains(err.Error(), "provider unavailable") {
|
if err == nil || !strings.Contains(err.Error(), "provider unavailable") {
|
||||||
t.Fatalf("provider error = %v", err)
|
t.Fatalf("provider error = %v", err)
|
||||||
|
|||||||
@@ -24,7 +24,7 @@ const (
|
|||||||
Key = "dnd/location-registry"
|
Key = "dnd/location-registry"
|
||||||
PromptID = "dnd.location_registry.normalize"
|
PromptID = "dnd.location_registry.normalize"
|
||||||
PromptVersion = "v1"
|
PromptVersion = "v1"
|
||||||
normalizationPolicy = "dnd.location_registry.normalize.v3"
|
normalizationPolicy = "dnd.location_registry.normalize.v4"
|
||||||
NormalizationPolicy = normalizationPolicy
|
NormalizationPolicy = normalizationPolicy
|
||||||
|
|
||||||
ReasonCodeLocationFieldsNormalized = "location_fields_normalized"
|
ReasonCodeLocationFieldsNormalized = "location_fields_normalized"
|
||||||
@@ -129,7 +129,7 @@ func (n *Normalizer) Normalize(ctx context.Context, req contracts.TypedNormalize
|
|||||||
case semanticreconcile.SkippedInsufficientCandidates:
|
case semanticreconcile.SkippedInsufficientCandidates:
|
||||||
return normalizationResult(deterministic, findings, nil)
|
return normalizationResult(deterministic, findings, nil)
|
||||||
case semanticreconcile.SkippedLimitExceeded:
|
case semanticreconcile.SkippedLimitExceeded:
|
||||||
return fallbackResult(deterministic, findings, semanticFallbackFinding(-1))
|
return fallbackResult(deterministic, findings, semanticFallbackFinding(-1, "the candidate set exceeded the configured semantic-reconciliation limits"))
|
||||||
case semanticreconcile.RetryableInvalidStructuredOutput:
|
case semanticreconcile.RetryableInvalidStructuredOutput:
|
||||||
return n.invalidStructuredResult(deterministic, findings)
|
return n.invalidStructuredResult(deterministic, findings)
|
||||||
case semanticreconcile.Complete, semanticreconcile.RetryableDiscardedProposalGroups:
|
case semanticreconcile.Complete, semanticreconcile.RetryableDiscardedProposalGroups:
|
||||||
@@ -149,17 +149,29 @@ func (n *Normalizer) Normalize(ctx context.Context, req contracts.TypedNormalize
|
|||||||
}
|
}
|
||||||
|
|
||||||
func (n *Normalizer) invalidStructuredResult(value dnd.LocationRegistry, findings []diagnostics.Finding) (contracts.TypedNormalizeResult[dnd.LocationRegistry], error) {
|
func (n *Normalizer) invalidStructuredResult(value dnd.LocationRegistry, findings []diagnostics.Finding) (contracts.TypedNormalizeResult[dnd.LocationRegistry], error) {
|
||||||
return retryResultWithFallback(value, findings, nil, "semantic proposal requires retry: invalid structured output", semanticFallbackFinding(-1))
|
return retryResultWithFallback(value, findings, nil, "", "semantic proposal requires retry: invalid structured output", semanticFallbackFinding(-1, "the final model response did not satisfy the structured-output contract"))
|
||||||
}
|
}
|
||||||
|
|
||||||
func retryResult(value dnd.LocationRegistry, findings []diagnostics.Finding, reconciliation semanticreconcile.Result) (contracts.TypedNormalizeResult[dnd.LocationRegistry], error) {
|
func retryResult(value dnd.LocationRegistry, findings []diagnostics.Finding, reconciliation semanticreconcile.Result) (contracts.TypedNormalizeResult[dnd.LocationRegistry], error) {
|
||||||
return retryResultWithFallback(value, findings, reconciliation.ModelCandidate(), diagnostics.Aggregate("semantic proposal requires retry", semanticreconcile.IssueDetails(reconciliation.Issues())), semanticFallbackFinding(reconciliation.DiscardedGroupCount()))
|
issues := reconciliation.Issues()
|
||||||
|
correctionDetails, err := semanticreconcile.CorrectionDetails(issues)
|
||||||
|
if err != nil {
|
||||||
|
return contracts.TypedNormalizeResult[dnd.LocationRegistry]{}, normalizerErrorf("build semantic correction details: %w", err)
|
||||||
|
}
|
||||||
|
correctionGuidance, err := semanticreconcile.CorrectionGuidance(issues)
|
||||||
|
if err != nil {
|
||||||
|
return contracts.TypedNormalizeResult[dnd.LocationRegistry]{}, normalizerErrorf("build semantic correction guidance: %w", err)
|
||||||
|
}
|
||||||
|
return retryResultWithFallback(value, findings, reconciliation.ModelCandidate(), correctionGuidance, diagnostics.Aggregate("semantic proposal requires retry", semanticreconcile.IssueDetails(issues)), semanticFallbackFinding(reconciliation.DiscardedGroupCount(), correctionDetails...))
|
||||||
}
|
}
|
||||||
|
|
||||||
func semanticFallbackFinding(discarded int) diagnostics.Finding {
|
func semanticFallbackFinding(discarded int, details ...string) diagnostics.Finding {
|
||||||
message := "semantic proposal could not be applied"
|
message := "semantic reconciliation could not be completed; all affected candidates were preserved separately"
|
||||||
if discarded >= 0 {
|
if discarded >= 0 {
|
||||||
message = fmt.Sprintf("%d proposal group(s) omitted after semantic proposal retry exhaustion", discarded)
|
message = fmt.Sprintf("%d proposal group(s) omitted from the final semantic proposal; all affected candidates were preserved separately", discarded)
|
||||||
|
}
|
||||||
|
if len(details) > 0 {
|
||||||
|
message = diagnostics.Aggregate(message, details)
|
||||||
}
|
}
|
||||||
return diagnostics.Finding{Scope: "locations", ReasonCode: ReasonCodeLocationSemanticReconciliationExhausted, Message: message}
|
return diagnostics.Finding{Scope: "locations", ReasonCode: ReasonCodeLocationSemanticReconciliationExhausted, Message: message}
|
||||||
}
|
}
|
||||||
@@ -185,7 +197,7 @@ func fallbackResult(value dnd.LocationRegistry, findings []diagnostics.Finding,
|
|||||||
return result, nil
|
return result, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
func retryResultWithFallback(value dnd.LocationRegistry, findings []diagnostics.Finding, candidate *contracts.ModelCandidate, message string, fallback diagnostics.Finding) (contracts.TypedNormalizeResult[dnd.LocationRegistry], error) {
|
func retryResultWithFallback(value dnd.LocationRegistry, findings []diagnostics.Finding, candidate *contracts.ModelCandidate, correctionGuidance, message string, fallback diagnostics.Finding) (contracts.TypedNormalizeResult[dnd.LocationRegistry], error) {
|
||||||
result, err := normalizationResult(value, findings, candidate)
|
result, err := normalizationResult(value, findings, candidate)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return contracts.TypedNormalizeResult[dnd.LocationRegistry]{}, err
|
return contracts.TypedNormalizeResult[dnd.LocationRegistry]{}, err
|
||||||
@@ -194,7 +206,7 @@ func retryResultWithFallback(value dnd.LocationRegistry, findings []diagnostics.
|
|||||||
if err != nil {
|
if err != nil {
|
||||||
return contracts.TypedNormalizeResult[dnd.LocationRegistry]{}, normalizerErrorf("collect fallback diagnostic: %w", err)
|
return contracts.TypedNormalizeResult[dnd.LocationRegistry]{}, normalizerErrorf("collect fallback diagnostic: %w", err)
|
||||||
}
|
}
|
||||||
result.Retry = &contracts.NormalizeRetry{ReasonCode: ReasonCodeLocationSemanticProposalInvalid, Message: message, FallbackDiagnostics: fallbackGroups}
|
result.Retry = &contracts.NormalizeRetry{ReasonCode: ReasonCodeLocationSemanticProposalInvalid, Message: message, CorrectionGuidance: correctionGuidance, FallbackDiagnostics: fallbackGroups}
|
||||||
return result, nil
|
return result, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|||||||
@@ -128,7 +128,10 @@ func TestNormalizeRejectsUnsafeAndOverlappingGroupsWithoutLosingCandidates(t *te
|
|||||||
if err != nil || result.Retry == nil || len(result.Value.Locations) != 3 || !strings.Contains(result.Retry.Message, "overlapping_member") {
|
if err != nil || result.Retry == nil || len(result.Value.Locations) != 3 || !strings.Contains(result.Retry.Message, "overlapping_member") {
|
||||||
t.Fatalf("Normalize() = %#v, %v; want safe retry fallback", result, err)
|
t.Fatalf("Normalize() = %#v, %v; want safe retry fallback", result, err)
|
||||||
}
|
}
|
||||||
if len(result.Retry.FallbackDiagnostics) != 1 || !strings.Contains(result.Retry.FallbackDiagnostics[0].Samples[0].Message, "2 proposal group") {
|
if !strings.Contains(result.Retry.CorrectionGuidance, "Duplicate groups 1, 2") || !strings.Contains(result.Retry.CorrectionGuidance, "at most one duplicate group") || strings.Contains(result.Retry.CorrectionGuidance, "overlapping_member") {
|
||||||
|
t.Fatalf("correction guidance = %q, want model-facing overlap explanation", result.Retry.CorrectionGuidance)
|
||||||
|
}
|
||||||
|
if len(result.Retry.FallbackDiagnostics) != 1 || !strings.Contains(result.Retry.FallbackDiagnostics[0].Samples[0].Message, "2 proposal group") || !strings.Contains(result.Retry.FallbackDiagnostics[0].Samples[0].Message, "preserved separately") {
|
||||||
t.Fatalf("fallback diagnostics = %#v", result.Retry.FallbackDiagnostics)
|
t.Fatalf("fallback diagnostics = %#v", result.Retry.FallbackDiagnostics)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
@@ -188,6 +191,9 @@ func TestNormalizeHandlesRetryFallbackAndErrors(t *testing.T) {
|
|||||||
if err != nil || invalid.Retry == nil || invalid.Retry.ReasonCode != ReasonCodeLocationSemanticProposalInvalid {
|
if err != nil || invalid.Retry == nil || invalid.Retry.ReasonCode != ReasonCodeLocationSemanticProposalInvalid {
|
||||||
t.Fatalf("invalid result = %#v, %v", invalid, err)
|
t.Fatalf("invalid result = %#v, %v", invalid, err)
|
||||||
}
|
}
|
||||||
|
if invalid.Retry.CorrectionGuidance != "" || !strings.Contains(invalid.Retry.FallbackDiagnostics[0].Samples[0].Message, "structured-output contract") || !strings.Contains(invalid.Retry.FallbackDiagnostics[0].Samples[0].Message, "preserved separately") {
|
||||||
|
t.Fatalf("invalid structured retry = %#v, want feedback-free retry and informative fallback", invalid.Retry)
|
||||||
|
}
|
||||||
_, err = newNormalizer(t, &recordingLocationNormalizerClient{err: errors.New("provider unavailable")}).Normalize(context.Background(), normalizeRequestWithSource(input, doc))
|
_, err = newNormalizer(t, &recordingLocationNormalizerClient{err: errors.New("provider unavailable")}).Normalize(context.Background(), normalizeRequestWithSource(input, doc))
|
||||||
if err == nil || !strings.Contains(err.Error(), "provider unavailable") {
|
if err == nil || !strings.Contains(err.Error(), "provider unavailable") {
|
||||||
t.Fatalf("provider error = %v", err)
|
t.Fatalf("provider error = %v", err)
|
||||||
|
|||||||
@@ -23,7 +23,7 @@ const (
|
|||||||
Key = "dnd/npc-registry"
|
Key = "dnd/npc-registry"
|
||||||
PromptID = "dnd.npc_registry.normalize"
|
PromptID = "dnd.npc_registry.normalize"
|
||||||
PromptVersion = "v1"
|
PromptVersion = "v1"
|
||||||
normalizationPolicy = "dnd.npc_registry.normalize.v5"
|
normalizationPolicy = "dnd.npc_registry.normalize.v6"
|
||||||
NormalizationPolicy = normalizationPolicy
|
NormalizationPolicy = normalizationPolicy
|
||||||
|
|
||||||
ReasonCodeNPCFieldsNormalized = "npc_fields_normalized"
|
ReasonCodeNPCFieldsNormalized = "npc_fields_normalized"
|
||||||
@@ -128,7 +128,7 @@ func (n *Normalizer) Normalize(ctx context.Context, req contracts.TypedNormalize
|
|||||||
case semanticreconcile.SkippedInsufficientCandidates:
|
case semanticreconcile.SkippedInsufficientCandidates:
|
||||||
return normalizationResult(deterministic, findings, nil)
|
return normalizationResult(deterministic, findings, nil)
|
||||||
case semanticreconcile.SkippedLimitExceeded:
|
case semanticreconcile.SkippedLimitExceeded:
|
||||||
return fallbackResult(deterministic, findings, semanticFallbackFinding(-1))
|
return fallbackResult(deterministic, findings, semanticFallbackFinding(-1, "the candidate set exceeded the configured semantic-reconciliation limits"))
|
||||||
case semanticreconcile.RetryableInvalidStructuredOutput:
|
case semanticreconcile.RetryableInvalidStructuredOutput:
|
||||||
return n.invalidStructuredResult(deterministic, findings)
|
return n.invalidStructuredResult(deterministic, findings)
|
||||||
case semanticreconcile.Complete, semanticreconcile.RetryableDiscardedProposalGroups:
|
case semanticreconcile.Complete, semanticreconcile.RetryableDiscardedProposalGroups:
|
||||||
@@ -148,17 +148,29 @@ func (n *Normalizer) Normalize(ctx context.Context, req contracts.TypedNormalize
|
|||||||
}
|
}
|
||||||
|
|
||||||
func (n *Normalizer) invalidStructuredResult(value dnd.NPCRegistry, findings []diagnostics.Finding) (contracts.TypedNormalizeResult[dnd.NPCRegistry], error) {
|
func (n *Normalizer) invalidStructuredResult(value dnd.NPCRegistry, findings []diagnostics.Finding) (contracts.TypedNormalizeResult[dnd.NPCRegistry], error) {
|
||||||
return retryResultWithFallback(value, findings, nil, "semantic proposal requires retry: invalid structured output", semanticFallbackFinding(-1))
|
return retryResultWithFallback(value, findings, nil, "", "semantic proposal requires retry: invalid structured output", semanticFallbackFinding(-1, "the final model response did not satisfy the structured-output contract"))
|
||||||
}
|
}
|
||||||
|
|
||||||
func retryResult(value dnd.NPCRegistry, findings []diagnostics.Finding, reconciliation semanticreconcile.Result) (contracts.TypedNormalizeResult[dnd.NPCRegistry], error) {
|
func retryResult(value dnd.NPCRegistry, findings []diagnostics.Finding, reconciliation semanticreconcile.Result) (contracts.TypedNormalizeResult[dnd.NPCRegistry], error) {
|
||||||
return retryResultWithFallback(value, findings, reconciliation.ModelCandidate(), diagnostics.Aggregate("semantic proposal requires retry", semanticreconcile.IssueDetails(reconciliation.Issues())), semanticFallbackFinding(reconciliation.DiscardedGroupCount()))
|
issues := reconciliation.Issues()
|
||||||
|
correctionDetails, err := semanticreconcile.CorrectionDetails(issues)
|
||||||
|
if err != nil {
|
||||||
|
return contracts.TypedNormalizeResult[dnd.NPCRegistry]{}, normalizerErrorf("build semantic correction details: %w", err)
|
||||||
|
}
|
||||||
|
correctionGuidance, err := semanticreconcile.CorrectionGuidance(issues)
|
||||||
|
if err != nil {
|
||||||
|
return contracts.TypedNormalizeResult[dnd.NPCRegistry]{}, normalizerErrorf("build semantic correction guidance: %w", err)
|
||||||
|
}
|
||||||
|
return retryResultWithFallback(value, findings, reconciliation.ModelCandidate(), correctionGuidance, diagnostics.Aggregate("semantic proposal requires retry", semanticreconcile.IssueDetails(issues)), semanticFallbackFinding(reconciliation.DiscardedGroupCount(), correctionDetails...))
|
||||||
}
|
}
|
||||||
|
|
||||||
func semanticFallbackFinding(discardedGroups int) diagnostics.Finding {
|
func semanticFallbackFinding(discardedGroups int, details ...string) diagnostics.Finding {
|
||||||
message := "semantic proposal could not be applied"
|
message := "semantic reconciliation could not be completed; all affected candidates were preserved separately"
|
||||||
if discardedGroups >= 0 {
|
if discardedGroups >= 0 {
|
||||||
message = fmt.Sprintf("%d proposal group(s) omitted after semantic proposal retry exhaustion", discardedGroups)
|
message = fmt.Sprintf("%d proposal group(s) omitted from the final semantic proposal; all affected candidates were preserved separately", discardedGroups)
|
||||||
|
}
|
||||||
|
if len(details) > 0 {
|
||||||
|
message = diagnostics.Aggregate(message, details)
|
||||||
}
|
}
|
||||||
return diagnostics.Finding{Scope: "npcs", ReasonCode: ReasonCodeNPCSemanticReconciliationExhausted, Message: message}
|
return diagnostics.Finding{Scope: "npcs", ReasonCode: ReasonCodeNPCSemanticReconciliationExhausted, Message: message}
|
||||||
}
|
}
|
||||||
@@ -184,7 +196,7 @@ func fallbackResult(value dnd.NPCRegistry, findings []diagnostics.Finding, fallb
|
|||||||
return result, nil
|
return result, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
func retryResultWithFallback(value dnd.NPCRegistry, findings []diagnostics.Finding, candidate *contracts.ModelCandidate, message string, fallback diagnostics.Finding) (contracts.TypedNormalizeResult[dnd.NPCRegistry], error) {
|
func retryResultWithFallback(value dnd.NPCRegistry, findings []diagnostics.Finding, candidate *contracts.ModelCandidate, correctionGuidance, message string, fallback diagnostics.Finding) (contracts.TypedNormalizeResult[dnd.NPCRegistry], error) {
|
||||||
result, err := normalizationResult(value, findings, candidate)
|
result, err := normalizationResult(value, findings, candidate)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return contracts.TypedNormalizeResult[dnd.NPCRegistry]{}, err
|
return contracts.TypedNormalizeResult[dnd.NPCRegistry]{}, err
|
||||||
@@ -193,7 +205,7 @@ func retryResultWithFallback(value dnd.NPCRegistry, findings []diagnostics.Findi
|
|||||||
if err != nil {
|
if err != nil {
|
||||||
return contracts.TypedNormalizeResult[dnd.NPCRegistry]{}, normalizerErrorf("collect fallback diagnostic: %w", err)
|
return contracts.TypedNormalizeResult[dnd.NPCRegistry]{}, normalizerErrorf("collect fallback diagnostic: %w", err)
|
||||||
}
|
}
|
||||||
result.Retry = &contracts.NormalizeRetry{ReasonCode: ReasonCodeNPCSemanticProposalInvalid, Message: message, FallbackDiagnostics: fallbackGroups}
|
result.Retry = &contracts.NormalizeRetry{ReasonCode: ReasonCodeNPCSemanticProposalInvalid, Message: message, CorrectionGuidance: correctionGuidance, FallbackDiagnostics: fallbackGroups}
|
||||||
return result, nil
|
return result, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|||||||
@@ -115,10 +115,13 @@ func TestNormalizeUnsafeProposalReturnsSafeRetryFallback(t *testing.T) {
|
|||||||
if result.Retry.ReasonCode != ReasonCodeNPCSemanticProposalInvalid || !strings.Contains(result.Retry.Message, "group 1: member_unknown") {
|
if result.Retry.ReasonCode != ReasonCodeNPCSemanticProposalInvalid || !strings.Contains(result.Retry.Message, "group 1: member_unknown") {
|
||||||
t.Fatalf("retry = %#v, want bounded invalid-proposal diagnostics", result.Retry)
|
t.Fatalf("retry = %#v, want bounded invalid-proposal diagnostics", result.Retry)
|
||||||
}
|
}
|
||||||
|
if !strings.Contains(result.Retry.CorrectionGuidance, "Duplicate group 2") || !strings.Contains(result.Retry.CorrectionGuidance, "supplied candidate list") || strings.Contains(result.Retry.CorrectionGuidance, "member_unknown") || strings.Contains(result.Retry.CorrectionGuidance, ReasonCodeNPCSemanticProposalInvalid) {
|
||||||
|
t.Fatalf("correction guidance = %q, want contextual prose without internal labels", result.Retry.CorrectionGuidance)
|
||||||
|
}
|
||||||
if len(result.Value.NPCs) != 2 || result.Value.NPCs[0].Name != "Mira Thorn" || result.Value.NPCs[1].Name != "Captain Vale" {
|
if len(result.Value.NPCs) != 2 || result.Value.NPCs[0].Name != "Mira Thorn" || result.Value.NPCs[1].Name != "Captain Vale" {
|
||||||
t.Fatalf("fallback NPCs = %#v, want independently safe group applied", result.Value.NPCs)
|
t.Fatalf("fallback NPCs = %#v, want independently safe group applied", result.Value.NPCs)
|
||||||
}
|
}
|
||||||
if len(result.Retry.FallbackDiagnostics) != 1 || result.Retry.FallbackDiagnostics[0].ReasonCode != ReasonCodeNPCSemanticReconciliationExhausted || !strings.Contains(result.Retry.FallbackDiagnostics[0].Samples[0].Message, "1 proposal group") {
|
if len(result.Retry.FallbackDiagnostics) != 1 || result.Retry.FallbackDiagnostics[0].ReasonCode != ReasonCodeNPCSemanticReconciliationExhausted || !strings.Contains(result.Retry.FallbackDiagnostics[0].Samples[0].Message, "1 proposal group") || !strings.Contains(result.Retry.FallbackDiagnostics[0].Samples[0].Message, "preserved separately") || !strings.Contains(result.Retry.FallbackDiagnostics[0].Samples[0].Message, "supplied candidate list") {
|
||||||
t.Fatalf("fallback diagnostics = %#v, want exact omitted-group warning", result.Retry.FallbackDiagnostics)
|
t.Fatalf("fallback diagnostics = %#v, want exact omitted-group warning", result.Retry.FallbackDiagnostics)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
@@ -134,6 +137,9 @@ func TestNormalizeInvalidStructuredOutputAndOperationalErrorsRemainDistinct(t *t
|
|||||||
if err != nil || result.Retry == nil || result.Retry.ReasonCode != ReasonCodeNPCSemanticProposalInvalid || len(result.Value.NPCs) != 2 {
|
if err != nil || result.Retry == nil || result.Retry.ReasonCode != ReasonCodeNPCSemanticProposalInvalid || len(result.Value.NPCs) != 2 {
|
||||||
t.Fatalf("invalid structured result = %#v, %v; want deterministic retry fallback", result, err)
|
t.Fatalf("invalid structured result = %#v, %v; want deterministic retry fallback", result, err)
|
||||||
}
|
}
|
||||||
|
if result.Retry.CorrectionGuidance != "" || !strings.Contains(result.Retry.FallbackDiagnostics[0].Samples[0].Message, "structured-output contract") || !strings.Contains(result.Retry.FallbackDiagnostics[0].Samples[0].Message, "preserved separately") {
|
||||||
|
t.Fatalf("invalid structured retry = %#v, want feedback-free retry and informative fallback", result.Retry)
|
||||||
|
}
|
||||||
operational := errors.New("provider unavailable")
|
operational := errors.New("provider unavailable")
|
||||||
if _, err := newNormalizer(t, &recordingNPCNormalizerClient{err: operational}).Normalize(context.Background(), normalizeRequestWithSource(input, doc)); !errors.Is(err, operational) || errors.Is(err, contracts.ErrInvalidStructuredOutput) {
|
if _, err := newNormalizer(t, &recordingNPCNormalizerClient{err: operational}).Normalize(context.Background(), normalizeRequestWithSource(input, doc)); !errors.Is(err, operational) || errors.Is(err, contracts.ErrInvalidStructuredOutput) {
|
||||||
t.Fatalf("operational completion error = %v, want ordinary error", err)
|
t.Fatalf("operational completion error = %v, want ordinary error", err)
|
||||||
|
|||||||
Reference in New Issue
Block a user