Document completed spell catalog baseline
This commit is contained in:
@@ -2,9 +2,31 @@
|
||||
|
||||
## Status
|
||||
|
||||
The immediate baseline feature is accepted and pending implementation.
|
||||
Feedback-aware repair and semantic LLM validation are deferred until the
|
||||
baseline has been evaluated.
|
||||
The accepted baseline implementation is complete. It provides effective
|
||||
SRD-plus-overlay catalog composition, canonical-name prompt grounding,
|
||||
deterministic catalog validation, the maintained `retries: 2` policy, and
|
||||
provenance and checkpoint identity coverage through the assembled offline
|
||||
workflow.
|
||||
|
||||
External quality evaluation is pending. This repository contains the
|
||||
maintained example and offline fake-LLM coverage, but no approved
|
||||
human-reviewed transcript corpus or authorized live-model evaluation was
|
||||
available for this implementation run. The maintained example can be run
|
||||
from the repository root with:
|
||||
|
||||
```sh
|
||||
go run ./cmd/notarius run dnd-session \
|
||||
--config examples/dnd-spells-production.config.yml \
|
||||
--input examples/seriatim-minimal-transcript.json \
|
||||
--chunk_cache bypass
|
||||
```
|
||||
|
||||
That command requires the configured Scriptorium profile credential. When an
|
||||
approved profile and reviewed corpus are available, record anonymized
|
||||
aggregate false positives, false negatives, unknown-name rejections,
|
||||
acceptance by attempt, and model-call cost here without committing transcripts
|
||||
or raw prompts. Feedback-aware repair and semantic LLM validation remain
|
||||
deferred until that baseline evaluation is available.
|
||||
|
||||
## Objective
|
||||
|
||||
|
||||
@@ -9,14 +9,9 @@ not as committed release dates.
|
||||
|
||||
### Solidify Spell Extraction
|
||||
|
||||
- Implement the immediate overlay-capable catalog-grounding and
|
||||
deterministic-validation baseline in
|
||||
[D&D Spell Extraction Quality](dnd-spell-extraction.md), using ordinary
|
||||
`retries: 2` behavior as the comparison point.
|
||||
- After evaluating that baseline, reconsider the roadmap's deferred structured
|
||||
diagnostics, retryability classification, repair-capable extractor contract,
|
||||
hybrid repair/fresh retry policy, and narrowly scoped semantic LLM
|
||||
validation.
|
||||
- Evaluate the implemented baseline in
|
||||
[D&D Spell Extraction Quality](dnd-spell-extraction.md), then reconsider its
|
||||
[deferred retry and validation work](dnd-spell-extraction.md#deferred-retry-and-validation-work).
|
||||
- Replace the no-op spell normalizer with a D&D-specific implementation that
|
||||
canonicalizes recognized spell names, performs only high-confidence repairs,
|
||||
retains uncertain values for review, and emits scoped warnings.
|
||||
|
||||
@@ -2,9 +2,10 @@
|
||||
|
||||
## Status
|
||||
|
||||
Ready for implementation. Follow the stages in order. This plan covers the
|
||||
accepted baseline in [D&D Spell Extraction Quality](dnd-spell-extraction.md),
|
||||
not its deferred repair-aware retry or semantic LLM-validation work.
|
||||
The accepted baseline implementation is complete through the assembled
|
||||
offline workflow. External quality evaluation remains pending as recorded in
|
||||
[D&D Spell Extraction Quality](dnd-spell-extraction.md). This plan covers that
|
||||
baseline, not its deferred repair-aware retry or semantic LLM-validation work.
|
||||
|
||||
This active plan replaces the completed test-suite implementation record that
|
||||
previously occupied this filename. That review remains documented in
|
||||
|
||||
Reference in New Issue
Block a user