96 lines
5.0 KiB
Markdown
96 lines
5.0 KiB
Markdown
# D&D Item-Occurrence Holder Reliability
|
|
|
|
## Purpose
|
|
|
|
Improve item-occurrence extraction reliability for smaller models by making
|
|
holder rules easier to follow and semantic retry feedback specific enough to
|
|
correct a rejected response.
|
|
|
|
## Problem
|
|
|
|
The item-occurrence prompt correctly defines the permitted `from` and `to`
|
|
values for every occurrence kind, but presents the rules in dense prose. The
|
|
private schema requires both nullable fields but cannot express their
|
|
cross-field relationship. The deterministic shape validator correctly rejects
|
|
incompatible combinations, yet its model-facing correction guidance only asks
|
|
for generally valid holder combinations. Notarius therefore resends the
|
|
defective response without telling the model which contextual records failed or
|
|
what their corrected holder shape must be.
|
|
|
|
## Target State
|
|
|
|
- The module instructions present one compact, unambiguous holder matrix:
|
|
`discovered` uses two nulls; `acquired` uses a null `from` and non-null `to`;
|
|
`lost` and `consumed` use a non-null `from` and null `to`; and `transferred`
|
|
uses two distinct named party members.
|
|
- The private LLM schema continues requiring `quantity`, `from`, and `to` with
|
|
nullable types, and constrains `kind` to the five supported values. The
|
|
durable v1 artifact contract remains unchanged.
|
|
- A holder-combination rejection supplies bounded, semantically meaningful
|
|
correction guidance that identifies affected occurrences by contextual item
|
|
name and transcript evidence where useful, states the required JSON
|
|
null/non-null shape, and requests a complete corrected replacement.
|
|
- Correction guidance never exposes durable item IDs, hashes, validator keys,
|
|
reason codes, or other opaque implementation identifiers. It does not rely
|
|
solely on mapped array indexes, because canonical extraction ordering may
|
|
differ from the raw response order appended to the retry prompt.
|
|
- Operator-facing validation messages remain specific and include enough
|
|
information to distinguish a missing holder from an extra or misplaced
|
|
holder.
|
|
- The existing validator remains strict. Holder combinations are not silently
|
|
rewritten: unlike reversed evidence endpoints, changing a holder can conceal
|
|
a misclassified discovery, acquisition, loss, consumption, or transfer.
|
|
|
|
## Required Work
|
|
|
|
1. Rewrite the holder paragraph in
|
|
`assets/dnd/item-occurrences/prompts/instructions.md` as a compact matrix,
|
|
retaining the existing classification and currency rules without adding
|
|
contradictory repetition.
|
|
2. Add the five supported `kind` values as an enum in the private item-occurrence
|
|
response schema. Keep every listed field required and keep `quantity`,
|
|
`from`, and `to` explicitly nullable.
|
|
3. Refactor the item-occurrence shape validator to build detailed bounded
|
|
operator issues and separate actionable correction guidance from the same
|
|
evaluated candidate. Group repeated holder failures where practical while
|
|
retaining contextual record identification and every distinct correction
|
|
rule needed by the model.
|
|
4. Preserve the generic validation retry protocol: the complete original
|
|
prompt, exact defective assistant response, and one user correction message.
|
|
Do not add a module-local retry loop.
|
|
5. Ensure the changed prompt and schema produce new computed fingerprints, and
|
|
bump the validator policy fingerprint so stale checkpoints cannot be reused
|
|
across the behavior change. Update internal documentation where needed; the
|
|
external item-occurrence contract requires no semantic change.
|
|
6. Add lean offline regression coverage for the holder matrix, schema enum,
|
|
contextual correction guidance, bounded aggregation, absence of opaque IDs,
|
|
and a rejected-then-corrected extraction attempt. Avoid exact prose snapshots
|
|
and live-provider tests.
|
|
|
|
## Non-Goals
|
|
|
|
- Automatically deleting, moving, or inferring holder values.
|
|
- Changing the durable item-occurrence schema or event taxonomy.
|
|
- Encoding the complete semantic holder matrix through provider-sensitive
|
|
conditional JSON Schema constructs.
|
|
- Adding warnings for ordinary semantic rejection or successful correction.
|
|
- Tuning unrelated D&D extraction prompts or validators.
|
|
|
|
## Acceptance Criteria
|
|
|
|
- The maintained item-occurrence prompt shows the clear holder matrix,
|
|
and the private schema rejects unsupported kind strings while remaining
|
|
compatible with strict structured-output providers.
|
|
- A rejected `acquired` occurrence receives guidance that explicitly requires
|
|
`from: null` and a non-null `to`; equivalent exact guidance exists for the
|
|
other four kinds.
|
|
- Multiple invalid occurrences yield one bounded correction request containing
|
|
all distinct required fixes and contextual identifiers that can be matched to
|
|
the appended raw response.
|
|
- No correction message contains an item ID, hash, validator key, or reason
|
|
code, and the raw model candidate remains byte-faithful.
|
|
- Valid candidates and the durable output contract are unchanged; invalid
|
|
holder combinations still fail after configured retries are exhausted.
|
|
- Focused tests, `go test ./...`, `go vet ./...`, and `go build ./cmd/notarius`
|
|
pass.
|