eric/notarius
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Add feature roadmap and implementation plan for D&D spell normalization module

Browse Source
This commit is contained in:
Eric Rakestraw
2026-07-20 15:31:57 -05:00
parent 0f25e7339f
commit d2eb763b9b
5 changed files with 142 additions and 1108 deletions
Download Patch File Download Diff File Expand all files Collapse all files

458
docs/roadmap/implementation.md
View File

@@ -1,458 +0,0 @@
# D&D Spell Extraction Quality Implementation Plan
## Status
The accepted baseline implementation is complete through the assembled
offline workflow. External quality evaluation remains pending as recorded in
[D&D Spell Extraction Quality](dnd-spell-extraction.md). This plan covers that
baseline, not its deferred repair-aware retry or semantic LLM-validation work.
This active plan replaces the completed test-suite implementation record that
previously occupied this filename. That review remains documented in
[Test Suite Policy Review](tests.md) and repository history; its stages are not
instructions for this feature.
## Objective
Implement an overlay-capable effective spell catalog, ground every
`dnd/spells` extraction request with its canonical names, reject extracted
names that do not resolve through that catalog, and establish the existing
same-request behavior with `retries: 2` as the measured baseline.
The completed feature must fail malformed catalog configuration before source
parsing or any LLM call, keep aliases out of the prompt, preserve the current
private LLM response schema, and use one effective catalog policy for both the
extractor and validator.
## Governing Documents
Before changing code, read and follow:
- [D&D Spell Extraction Quality](dnd-spell-extraction.md) for accepted product
intent, target state, and deferred work;
- [Architecture](../policy/architecture.md) for dependency direction,
preparation, domain ownership, validation, reference provenance, and LLM
boundaries;
- [Documentation Policy](../policy/documentation.md) for canonical document
ownership and current-versus-future behavior;
- [Testing Policy](../policy/testing.md) for behavioral test ownership and
offline/default-suite requirements;
- [Pipeline Internals](../internal/pipeline.md), [Module Internals](../internal/modules.md),
and [LLM Runtime](../internal/llm.md) for the implementation boundaries being
changed; and
- [Configuration](../config.md) and the existing D&D integration contracts for
current user-visible contracts.
Preserve unrelated worktree changes. Do not implement anything listed under
the feature roadmap's deferred retry and validation section.
## Fixed Design Decisions
### Configuration and overlay transport
Add one optional extract-reference slot named `spell_catalog` to `dnd/spells`.
It accepts one UTF-8 `application/json` file with a maximum size of 1 MiB. Use
the normal reference binding and materialization path, so configuration-relative
paths, CLI overrides, redaction, provenance, and checkpoint reference digests
continue to work without a D&D-specific filesystem reader.
The slot binds an overlay bundle rather than one catalog. One file can therefore
carry multiple sourcebook, campaign, or homebrew catalogs without adding list
values to the general configuration reference schema. The canonical lane-local
form is:
```yaml
artifacts:
spells:
extract:
module: dnd/spells
retries: 2
references:
spell_catalog: ./campaign-spells.json
```
The existing pipeline-level reference-default behavior may also bind the slot.
Do not add module options, environment variables, or a new CLI flag for catalog
overlays.
### Overlay bundle contract
Define and document this strict JSON shape as version 1:
```json
{
"schema_version": "notarius.dnd.spell-catalog-overlay.v1",
"catalogs": [
{
"id": "campaign.example",
"ruleset": "dnd-5e-2014",
"source": {
"title": "Example campaign spells",
"version": "1",
"url": "",
"license": ""
},
"spells": [
{
"name": "Aegis of Emberfall",
"aliases": ["Emberfall Aegis"]
}
]
}
]
}
```
Use `encoding/json` with unknown-field rejection and reject trailing JSON
values. Require `schema_version` and `catalogs` at the bundle level;
`id`, `ruleset`, `source`, and `spells` on each catalog; `title` on each source;
and `name` on each spell. Require the exact schema version, at least one
catalog, unique trimmed nonempty catalog IDs, the `dnd-5e-2014` ruleset, a
trimmed nonempty source title, at least one spell per catalog, and trimmed
nonempty canonical names. `source.version`, `source.url`, and `source.license`
are optional strings and may be empty so private homebrew material is not
forced to claim a publication or license. `aliases` may be omitted or supplied
as an array, but it must not be JSON `null`, and every supplied alias must be
trimmed and nonempty. The integration contract must state that catalog content
is contextual domain data and never source evidence for a cast.
Overlay spells intentionally carry names and aliases only. Do not invent
unknown spell levels or class memberships, and do not weaken the richer
embedded SRD catalog contract to accommodate overlay-only recognition data.
### Effective catalog composition
Keep the existing embedded SRD catalog and introduce a separate immutable
effective recognition-catalog type in `internal/modules/dnd/spells/catalog`.
It must expose defensive-copy access to globally sorted canonical names,
normalized lookup returning the canonical display name, the ruleset, overlay
IDs, and a deterministic semantic digest.
Compose the effective catalog using these rules:
1. Start with every canonical SRD name and its aliases.
2. Sort overlay catalogs by ID before composition; file order must not choose a
winner.
3. Normalize lookup keys using the package's existing case, whitespace, and
apostrophe rules.
4. A new canonical key adds a spell. An overlay canonical key equal to an
existing canonical key augments that same spell; retain the established
canonical display spelling and merge aliases.
5. If equal normalized canonical keys have different display spellings, reject
the bundle instead of selecting one by order.
6. Repeated aliases mapping to the same canonical spell are idempotent.
Canonical-name or alias collisions mapping to different spells are errors,
including collisions with the embedded catalog.
7. Return canonical names and aliases in deterministic sorted order regardless
of input ordering. Do not expose mutable backing slices or maps.
Compute the effective-catalog digest from a canonical semantic representation
that includes the base catalog identity, ruleset, sorted overlay identity and
source metadata, and sorted effective canonical-name/alias mappings. Formatting
or object-key order alone must not change this semantic digest. Raw reference
digests continue to protect checkpoint identity when overlay file bytes change.
### Construction-time reference availability
Extend `pipeline.BuildRequest` with a defensively cloned
`contracts.ReferenceSet`. `pipeline.Prepare` must provide the materialized
references belonging to each constructed stage implementation and to every
validator in that stage's chain:
- chunk references to the chunker and chunk validators;
- lane extract references to the extractor and extract validators;
- lane merge references to the merger and merge validators; and
- lane normalize references to the normalizer and normalize validators.
Input and output receive an empty set because they cannot declare references.
Keep runtime request references unchanged. Construction-time references exist
so immutable modules can validate and retain derived data before source parsing;
runtime references remain available for run operations and validator context.
Do not put D&D catalog types into framework contracts or shared module
dependencies.
Both the spell extractor and catalog validator must construct their immutable
effective catalog from the same `spell_catalog` reference bytes. Centralize
overlay selection and catalog composition in the D&D catalog package so the two
implementations cannot drift. Reject more than one materialized item in the
slot even though current configuration produces only one.
### Prompt grounding
Add one required structured-completion input named `spell_catalog` with media
type `application/json`. Its generated content is exactly a small object whose
`spell_names` value is the globally sorted array of effective canonical names.
Do not include aliases, levels, classes, source metadata, raw overlay JSON, or
spell descriptions.
Add a package-owned prompt fragment that presents this input as the allowed
spell-name catalog. Update task instructions to require canonical catalog names
and prohibit returning names absent from the catalog. Keep campaign references
separate and retain the rule that references cannot establish that a cast
occurred. Do not add an enum to `dnd_spells_llm.v1.json` and do not change the
durable `dnd/spell-list` artifact schema.
Use the digest of the generated prompt-input bytes as its input digest. Extend
extractor manifest metadata with the effective catalog semantic digest, base
catalog ID, and sorted overlay IDs; do not expose catalog content or local file
content in manifests or debug summaries. Existing reference provenance owns
overlay origin URI, media type, byte size, and raw digest.
### Deterministic validation
Add typed validator key `extract/dnd/spells/catalog` under
`internal/modules/dnd/validate/spells/catalog`. Construct it from the effective
catalog at preparation time and give it no module options or LLM profile.
If spell shape is invalid, approve without catalog diagnostics so the shape
validator remains the owner of shape rejection. Otherwise, check every
nonempty spell name through normalized effective-catalog lookup. Approve
canonical names and aliases without mutating the artifact. If any names are
unknown, reject the complete extraction result with stable reason code
`unknown_spell` and a bounded message identifying all affected spell indices
and names. Sort diagnostics by artifact index and cap displayed issues at 20,
reporting the number omitted. Do not silently delete casts, convert aliases to
canonical names, or emit future structured validation issues in this feature.
Register the validator immediately after `extract/dnd/spells/shape` in the
production default chain. Explicit validator overrides remain authoritative.
### Retry baseline
Make no runner, retry, rejection, or validation-result contract changes. The
maintained production example continues to set `extract.retries: 2`, meaning
one initial extraction plus at most two identical-request attempts around the
complete validator chain. Rejections exhausted after three attempts remain
nonfatal rejected outputs under current runner semantics.
Do not add repair prompts, previous candidates, rejection feedback, structured
validation issues, error retryability classification, or an LLM validator.
## Stage 0 — Baseline and scope control
1. Read the governing documents and inspect the current worktree. Record and
preserve changes not owned by this plan.
2. Confirm that the embedded catalog loads 319 spells and 779 class
memberships and that the maintained production example already contains
`extract.retries: 2`.
3. Run the baseline commands:
```sh
go test ./...
go vet ./...
go build ./cmd/notarius
```
4. If the baseline fails for an unrelated reason, stop and report it. Do not
modify unrelated behavior or tests to obtain a clean baseline.
Stage 0 is complete when the baseline and preserved worktree scope are known.
## Stage 1 — Inject materialized references during preparation
1. Add `References contracts.ReferenceSet` to `pipeline.BuildRequest` and clone
it in `cloneBuildRequest`.
2. Refactor `pipeline.Prepare`, `prepareLane`, and `prepareValidatorChain` so
each builder receives the correct materialized reference target described in
the fixed decisions. Do not change operation-request delivery.
3. Update builder/preparation tests to prove:
- construction receives the correct target-specific reference content;
- extractor and extract validators receive independent clones of the same
logical set;
- mutation by one builder cannot affect another builder, the resolved
pipeline, or runtime requests; and
- empty/unmaterialized reference targets remain valid for callers that use
`Prepare` directly.
4. Update [Pipeline Internals](../internal/pipeline.md) and
[Module Internals](../internal/modules.md) to describe construction-time
reference delivery as current behavior once the code lands.
5. Run focused framework tests plus all repository-wide validation commands.
Do not change configuration syntax, reference multiplicity, reference
materialization, manifests, or checkpoint formats in this stage.
Stage 1 is complete when all stage builders receive isolated target references
at preparation and existing pipelines behave unchanged.
## Stage 2 — Implement overlay parsing and effective catalog composition
1. Add package-owned overlay DTOs, strict decoding, semantic validation, and
immutable effective-catalog composition under
`internal/modules/dnd/spells/catalog`. Keep overlay DTOs private; expose only
the minimal effective-catalog API needed by extraction and validation.
2. Preserve `LoadSRD5E2014`, existing SRD metadata, and the existing `Spell`
API. Do not represent overlay-only entries as incomplete SRD `Spell` values.
3. Add a helper that resolves the optional `spell_catalog` slot from a cloned
`ReferenceSet`, enforces zero-or-one JSON item, and returns the base-only or
base-plus-overlay effective catalog. Keep all filesystem access in framework
reference materialization.
4. Add focused behavioral tests for:
- base-only canonical names, normalized lookup, and immutability;
- a new overlay spell and an alias accepted only by lookup;
- augmentation of an existing canonical spell;
- deterministic results and semantic digest under reordered JSON catalogs,
spells, aliases, and object keys;
- duplicate IDs, wrong ruleset/schema version, empty required values,
unknown fields, trailing JSON, and empty bundles;
- same-key display conflicts and every cross-spell canonical/alias collision
category; and
- rejection of multiple items or a non-JSON item in the reference slot.
5. Add the external overlay format contract under `docs/integrations/` and
update the internal catalog inventory only after the behavior exists.
6. Run the catalog package tests, import-boundary tests, and repository-wide
validation commands.
Prefer table-driven coverage for meaningful invalid-input categories, but do
not duplicate every parser case across higher layers.
Stage 2 is complete when one immutable effective catalog deterministically
represents the embedded base plus a valid overlay bundle and rejects ambiguous
composition before execution.
## Stage 3 — Ground the spell extractor
1. Declare the optional `spell_catalog` reference slot on both the extractor
runtime and `ModuleSpec`, accepting only `application/json`, `Multiple:
false`, and `MaxBytes: 1048576`. Keep the existing campaign-reference slots
unchanged.
2. Update the extractor builder and constructor to resolve and retain the
effective catalog at preparation. A malformed overlay must make `Prepare`
fail before raw input is read or the LLM is called.
3. Generate the canonical-name-only JSON prompt material from the retained
catalog and add it to every structured completion request. Extend the
Scriptorium prompt declaration and package-owned prompt assets accordingly.
4. Update prompt wording to distinguish the allowed spell-name catalog from
campaign references and source evidence. Leave both response schemas
unchanged.
5. Add effective catalog identity to extractor manifest metadata without
exposing names, aliases, or raw overlay content. Include the new prompt
fragment in prompt hashing.
6. Update focused tests to protect:
- base-only requests contain all and only sorted canonical names;
- aliases and non-name metadata do not appear in prompt input;
- overlay canonical names do appear;
- prompt/schema preparation succeeds offline with the required input;
- prompt and manifest diagnostics omit source, reference, overlay, alias,
and catalog-name content; and
- malformed overlays fail construction without invoking the fake LLM.
7. Update the configuration reference-slot catalog, internal module docs, LLM
internals where needed, and the external overlay contract in the same stage.
8. Add a safe maintained overlay example file and bind it from the production
D&D example. Keep the minimal example base-only. Ensure example tests load
and materialize the referenced file, rather than checking YAML syntax alone.
9. Run focused extractor, prompt-asset, CLI example, and integration tests plus
all repository-wide validation commands.
Stage 3 is complete when every spell extraction is grounded by the immutable
effective catalog and all overlay failures occur before paid work.
## Stage 4 — Add catalog validation and production policy
1. Implement and register `extract/dnd/spells/catalog` as specified above,
reusing the same catalog resolver as the extractor.
2. Insert it immediately after the shape validator in
`internal/modules/dnd/register`. Do not reorder or otherwise change the
remaining default validators.
3. Add package-level tests for canonical names, lookup normalization, aliases,
overlay spells, multiple unknown casts, the 20-issue message bound, shape
deferral, immutability, strict empty options, execution class, and
registration.
4. Update production composition tests to protect registration, exact default
chain placement, and construction from the same materialized overlay.
5. Add one assembled offline retry test using the production D&D registration
and a fake structured LLM, with two subcases: one remains unknown through
exhaustion, and one becomes catalog-valid on a retry. Configure a test-local
retry count and assert calls are bounded by `retries + 1`, exhausted
rejection is nonfatal, no rejected attempt advances to merge, and only the
accepted attempt contributes warnings and output. Do not repeat generic
retry cases already owned by framework tests.
6. Update the implemented-validator catalog in Configuration and the D&D
validator inventory in Module Internals.
7. Run focused validator, registrar, integration, and CLI tests plus all
repository-wide validation commands.
Stage 4 is complete when no artifact containing a name outside the effective
catalog can pass the production spell extraction chain.
## Stage 5 — Verify provenance, identity, and assembled behavior
1. Add or extend the narrowest existing CLI/checkpoint contract tests to prove
that changing overlay file bytes changes checkpoint identity through
reference provenance, while semantically reordered overlay content retains
the same extractor-reported effective catalog digest. Do not alter
checkpoint schemas or path formats.
2. Verify that changing the configured overlay binding changes resolved
pipeline identity, and that manifest reference provenance records the
overlay origin, media type, size, and raw digest without content.
3. Exercise a representative production-composed run with an overlay-only
canonical spell and confirm prompt grounding, validation acceptance, output,
manifest catalog metadata, and reference provenance agree.
4. Verify base-only configuration remains valid and produces no overlay
provenance or overlay IDs.
5. Run:
```sh
go test ./...
go vet ./...
go build ./cmd/notarius
go test -race ./internal/framework/pipeline ./internal/cli ./internal/modules/dnd/extract/spells ./internal/modules/dnd/validate/spells/catalog ./internal/modules/integration
```
Stage 5 is complete when the assembled offline workflow protects catalog
grounding, validation, provenance, retry count, and checkpoint invalidation at
their appropriate test boundaries.
## Stage 6 — Documentation, evaluation handoff, and completion
1. Review every affected current-behavior owner and remove roadmap-style
wording from implemented contracts:
- `docs/config.md` owns the slot name, media type, byte limit, binding form,
validator key, and default chain;
- the new `docs/integrations/` document owns overlay JSON shape and
compatibility rules;
- `docs/internal/pipeline.md`, `docs/internal/modules.md`, and
`docs/internal/overview.md` own construction and component behavior;
- maintained copyable files remain under `examples/`; and
- operations documentation changes only if runtime handling actually
changes.
2. Check all links and ensure examples contain no credentials, private paths,
transcripts, or copyrighted spell descriptions.
3. If an approved LLM profile and human-reviewed transcript corpus are
available, run the base-only and overlay-capable baseline with
`retries: 2`. Record anonymized aggregate false positives, false negatives,
unknown-name rejections, acceptance by attempt, and model-call cost in the
feature roadmap without committing sensitive transcripts or raw prompts.
4. If those external evaluation inputs are unavailable, mark evaluation as
pending external execution and provide the operator with the exact config
and command used by the maintained example. Do not fabricate quality
results, invoke a paid model without authorization, or make live evaluation
part of the default Go test suite.
5. Update [D&D Spell Extraction Quality](dnd-spell-extraction.md) status to
distinguish completed implementation from pending or completed evaluation.
Remove completed baseline items from [Future Work](future.md), retaining a
concise link for deferred repair-aware retry and LLM-validator work.
6. Run `git diff --check` and the full Stage 5 validation set once more after
documentation and example changes.
Stage 6 is complete when current behavior is documented in its canonical
owners, the feature roadmap accurately records implementation/evaluation
status, deferred work remains unimplemented, and all required checks pass.
## Explicit Non-Goals
- Changing the framework-wide retry default or retry orchestration.
- Supplying rejected candidates or validation feedback to retry attempts.
- Structured validation-issue contracts or retryability classification.
- An LLM-backed spell validator.
- Spell-name mutation or catalog-aware normalization.
- Multiple reference bindings per generic slot or list-valued reference
configuration.
- Spell descriptions, mechanics, unknown overlay levels, or class metadata.
- A configuration-version bump, environment override, checkpoint migration,
or checkpoint-format change.
## Open Questions
None. The overlay transport, file contract, composition policy, preparation
boundary, prompt content, validator behavior, retry baseline, provenance, and
test ownership are fixed by this plan.