Improve item occurrence holder corrections
This commit is contained in:
@@ -15,7 +15,7 @@ diagnostics. The remaining near-term work applies those completed foundations
|
||||
to domain review and empirical evaluation.
|
||||
|
||||
The active D&D reliability work is defined by
|
||||
[D&D Source-Reference Endpoint Canonicalization](source-reference-canonicalization.md).
|
||||
[D&D Item-Occurrence Holder Reliability](item-occurrence-holder-reliability.md).
|
||||
|
||||
## Near-Term D&D Pipeline
|
||||
|
||||
|
||||
95
docs/roadmap/item-occurrence-holder-reliability.md
Normal file
95
docs/roadmap/item-occurrence-holder-reliability.md
Normal file
@@ -0,0 +1,95 @@
|
||||
# D&D Item-Occurrence Holder Reliability
|
||||
|
||||
## Purpose
|
||||
|
||||
Improve item-occurrence extraction reliability for smaller models by making
|
||||
holder rules easier to follow and semantic retry feedback specific enough to
|
||||
correct a rejected response.
|
||||
|
||||
## Problem
|
||||
|
||||
The item-occurrence prompt correctly defines the permitted `from` and `to`
|
||||
values for every occurrence kind, but presents the rules in dense prose. The
|
||||
private schema requires both nullable fields but cannot express their
|
||||
cross-field relationship. The deterministic shape validator correctly rejects
|
||||
incompatible combinations, yet its model-facing correction guidance only asks
|
||||
for generally valid holder combinations. Notarius therefore resends the
|
||||
defective response without telling the model which contextual records failed or
|
||||
what their corrected holder shape must be.
|
||||
|
||||
## Target State
|
||||
|
||||
- The module instructions present one compact, unambiguous holder matrix:
|
||||
`discovered` uses two nulls; `acquired` uses a null `from` and non-null `to`;
|
||||
`lost` and `consumed` use a non-null `from` and null `to`; and `transferred`
|
||||
uses two distinct named party members.
|
||||
- The private LLM schema continues requiring `quantity`, `from`, and `to` with
|
||||
nullable types, and constrains `kind` to the five supported values. The
|
||||
durable v1 artifact contract remains unchanged.
|
||||
- A holder-combination rejection supplies bounded, semantically meaningful
|
||||
correction guidance that identifies affected occurrences by contextual item
|
||||
name and transcript evidence where useful, states the required JSON
|
||||
null/non-null shape, and requests a complete corrected replacement.
|
||||
- Correction guidance never exposes durable item IDs, hashes, validator keys,
|
||||
reason codes, or other opaque implementation identifiers. It does not rely
|
||||
solely on mapped array indexes, because canonical extraction ordering may
|
||||
differ from the raw response order appended to the retry prompt.
|
||||
- Operator-facing validation messages remain specific and include enough
|
||||
information to distinguish a missing holder from an extra or misplaced
|
||||
holder.
|
||||
- The existing validator remains strict. Holder combinations are not silently
|
||||
rewritten: unlike reversed evidence endpoints, changing a holder can conceal
|
||||
a misclassified discovery, acquisition, loss, consumption, or transfer.
|
||||
|
||||
## Required Work
|
||||
|
||||
1. Rewrite the holder paragraph in
|
||||
`assets/dnd/item-occurrences/prompts/instructions.md` as a compact matrix,
|
||||
retaining the existing classification and currency rules without adding
|
||||
contradictory repetition.
|
||||
2. Add the five supported `kind` values as an enum in the private item-occurrence
|
||||
response schema. Keep every listed field required and keep `quantity`,
|
||||
`from`, and `to` explicitly nullable.
|
||||
3. Refactor the item-occurrence shape validator to build detailed bounded
|
||||
operator issues and separate actionable correction guidance from the same
|
||||
evaluated candidate. Group repeated holder failures where practical while
|
||||
retaining contextual record identification and every distinct correction
|
||||
rule needed by the model.
|
||||
4. Preserve the generic validation retry protocol: the complete original
|
||||
prompt, exact defective assistant response, and one user correction message.
|
||||
Do not add a module-local retry loop.
|
||||
5. Ensure the changed prompt and schema produce new computed fingerprints, and
|
||||
bump the validator policy fingerprint so stale checkpoints cannot be reused
|
||||
across the behavior change. Update internal documentation where needed; the
|
||||
external item-occurrence contract requires no semantic change.
|
||||
6. Add lean offline regression coverage for the holder matrix, schema enum,
|
||||
contextual correction guidance, bounded aggregation, absence of opaque IDs,
|
||||
and a rejected-then-corrected extraction attempt. Avoid exact prose snapshots
|
||||
and live-provider tests.
|
||||
|
||||
## Non-Goals
|
||||
|
||||
- Automatically deleting, moving, or inferring holder values.
|
||||
- Changing the durable item-occurrence schema or event taxonomy.
|
||||
- Encoding the complete semantic holder matrix through provider-sensitive
|
||||
conditional JSON Schema constructs.
|
||||
- Adding warnings for ordinary semantic rejection or successful correction.
|
||||
- Tuning unrelated D&D extraction prompts or validators.
|
||||
|
||||
## Acceptance Criteria
|
||||
|
||||
- The maintained item-occurrence prompt shows the clear holder matrix,
|
||||
and the private schema rejects unsupported kind strings while remaining
|
||||
compatible with strict structured-output providers.
|
||||
- A rejected `acquired` occurrence receives guidance that explicitly requires
|
||||
`from: null` and a non-null `to`; equivalent exact guidance exists for the
|
||||
other four kinds.
|
||||
- Multiple invalid occurrences yield one bounded correction request containing
|
||||
all distinct required fixes and contextual identifiers that can be matched to
|
||||
the appended raw response.
|
||||
- No correction message contains an item ID, hash, validator key, or reason
|
||||
code, and the raw model candidate remains byte-faithful.
|
||||
- Valid candidates and the durable output contract are unchanged; invalid
|
||||
holder combinations still fail after configured retries are exhausted.
|
||||
- Focused tests, `go test ./...`, `go vet ./...`, and `go build ./cmd/notarius`
|
||||
pass.
|
||||
Reference in New Issue
Block a user