notarius/docs/roadmap/implementation.md

D&D Spell Extraction Quality Implementation Plan

Status

The accepted baseline implementation is complete through the assembled offline workflow. External quality evaluation remains pending as recorded in D&D Spell Extraction Quality. This plan covers that baseline, not its deferred repair-aware retry or semantic LLM-validation work.

This active plan replaces the completed test-suite implementation record that previously occupied this filename. That review remains documented in Test Suite Policy Review and repository history; its stages are not instructions for this feature.

Objective

Implement an overlay-capable effective spell catalog, ground every dnd/spells extraction request with its canonical names, reject extracted names that do not resolve through that catalog, and establish the existing same-request behavior with retries: 2 as the measured baseline.

The completed feature must fail malformed catalog configuration before source parsing or any LLM call, keep aliases out of the prompt, preserve the current private LLM response schema, and use one effective catalog policy for both the extractor and validator.

Governing Documents

Before changing code, read and follow:

• D&D Spell Extraction Quality for accepted product intent, target state, and deferred work;
• Architecture for dependency direction, preparation, domain ownership, validation, reference provenance, and LLM boundaries;
• Documentation Policy for canonical document ownership and current-versus-future behavior;
• Testing Policy for behavioral test ownership and offline/default-suite requirements;
• Pipeline Internals, Module Internals, and LLM Runtime for the implementation boundaries being changed; and
• Configuration and the existing D&D integration contracts for current user-visible contracts.

Preserve unrelated worktree changes. Do not implement anything listed under the feature roadmap's deferred retry and validation section.

Fixed Design Decisions

Configuration and overlay transport

Add one optional extract-reference slot named spell_catalog to dnd/spells. It accepts one UTF-8 application/json file with a maximum size of 1 MiB. Use the normal reference binding and materialization path, so configuration-relative paths, CLI overrides, redaction, provenance, and checkpoint reference digests continue to work without a D&D-specific filesystem reader.

The slot binds an overlay bundle rather than one catalog. One file can therefore carry multiple sourcebook, campaign, or homebrew catalogs without adding list values to the general configuration reference schema. The canonical lane-local form is:

artifacts:
  spells:
    extract:
      module: dnd/spells
      retries: 2
      references:
        spell_catalog: ./campaign-spells.json

The existing pipeline-level reference-default behavior may also bind the slot. Do not add module options, environment variables, or a new CLI flag for catalog overlays.

Overlay bundle contract

Define and document this strict JSON shape as version 1:

{
  "schema_version": "notarius.dnd.spell-catalog-overlay.v1",
  "catalogs": [
    {
      "id": "campaign.example",
      "ruleset": "dnd-5e-2014",
      "source": {
        "title": "Example campaign spells",
        "version": "1",
        "url": "",
        "license": ""
      },
      "spells": [
        {
          "name": "Aegis of Emberfall",
          "aliases": ["Emberfall Aegis"]
        }
      ]
    }
  ]
}

Use encoding/json with unknown-field rejection and reject trailing JSON values. Require schema_version and catalogs at the bundle level; id, ruleset, source, and spells on each catalog; title on each source; and name on each spell. Require the exact schema version, at least one catalog, unique trimmed nonempty catalog IDs, the dnd-5e-2014 ruleset, a trimmed nonempty source title, at least one spell per catalog, and trimmed nonempty canonical names. source.version, source.url, and source.license are optional strings and may be empty so private homebrew material is not forced to claim a publication or license. aliases may be omitted or supplied as an array, but it must not be JSON null, and every supplied alias must be trimmed and nonempty. The integration contract must state that catalog content is contextual domain data and never source evidence for a cast.

Overlay spells intentionally carry names and aliases only. Do not invent unknown spell levels or class memberships, and do not weaken the richer embedded SRD catalog contract to accommodate overlay-only recognition data.

Effective catalog composition

Keep the existing embedded SRD catalog and introduce a separate immutable effective recognition-catalog type in internal/modules/dnd/spells/catalog. It must expose defensive-copy access to globally sorted canonical names, normalized lookup returning the canonical display name, the ruleset, overlay IDs, and a deterministic semantic digest.

Compose the effective catalog using these rules:

1. Start with every canonical SRD name and its aliases.
2. Sort overlay catalogs by ID before composition; file order must not choose a winner.
3. Normalize lookup keys using the package's existing case, whitespace, and apostrophe rules.
4. A new canonical key adds a spell. An overlay canonical key equal to an existing canonical key augments that same spell; retain the established canonical display spelling and merge aliases.
5. If equal normalized canonical keys have different display spellings, reject the bundle instead of selecting one by order.
6. Repeated aliases mapping to the same canonical spell are idempotent. Canonical-name or alias collisions mapping to different spells are errors, including collisions with the embedded catalog.
7. Return canonical names and aliases in deterministic sorted order regardless of input ordering. Do not expose mutable backing slices or maps.

Compute the effective-catalog digest from a canonical semantic representation that includes the base catalog identity, ruleset, sorted overlay identity and source metadata, and sorted effective canonical-name/alias mappings. Formatting or object-key order alone must not change this semantic digest. Raw reference digests continue to protect checkpoint identity when overlay file bytes change.

Construction-time reference availability

Extend pipeline.BuildRequest with a defensively cloned contracts.ReferenceSet. pipeline.Prepare must provide the materialized references belonging to each constructed stage implementation and to every validator in that stage's chain:

• chunk references to the chunker and chunk validators;
• lane extract references to the extractor and extract validators;
• lane merge references to the merger and merge validators; and
• lane normalize references to the normalizer and normalize validators.

Input and output receive an empty set because they cannot declare references. Keep runtime request references unchanged. Construction-time references exist so immutable modules can validate and retain derived data before source parsing; runtime references remain available for run operations and validator context. Do not put D&D catalog types into framework contracts or shared module dependencies.

Both the spell extractor and catalog validator must construct their immutable effective catalog from the same spell_catalog reference bytes. Centralize overlay selection and catalog composition in the D&D catalog package so the two implementations cannot drift. Reject more than one materialized item in the slot even though current configuration produces only one.

Prompt grounding

Add one required structured-completion input named spell_catalog with media type application/json. Its generated content is exactly a small object whose spell_names value is the globally sorted array of effective canonical names. Do not include aliases, levels, classes, source metadata, raw overlay JSON, or spell descriptions.

Add a package-owned prompt fragment that presents this input as the allowed spell-name catalog. Update task instructions to require canonical catalog names and prohibit returning names absent from the catalog. Keep campaign references separate and retain the rule that references cannot establish that a cast occurred. Do not add an enum to dnd_spells_llm.v1.json and do not change the durable dnd/spell-list artifact schema.

Use the digest of the generated prompt-input bytes as its input digest. Extend extractor manifest metadata with the effective catalog semantic digest, base catalog ID, and sorted overlay IDs; do not expose catalog content or local file content in manifests or debug summaries. Existing reference provenance owns overlay origin URI, media type, byte size, and raw digest.

Deterministic validation

Add typed validator key extract/dnd/spells/catalog under internal/modules/dnd/validate/spells/catalog. Construct it from the effective catalog at preparation time and give it no module options or LLM profile.

If spell shape is invalid, approve without catalog diagnostics so the shape validator remains the owner of shape rejection. Otherwise, check every nonempty spell name through normalized effective-catalog lookup. Approve canonical names and aliases without mutating the artifact. If any names are unknown, reject the complete extraction result with stable reason code unknown_spell and a bounded message identifying all affected spell indices and names. Sort diagnostics by artifact index and cap displayed issues at 20, reporting the number omitted. Do not silently delete casts, convert aliases to canonical names, or emit future structured validation issues in this feature.

Register the validator immediately after extract/dnd/spells/shape in the production default chain. Explicit validator overrides remain authoritative.

Retry baseline

Make no runner, retry, rejection, or validation-result contract changes. The maintained production example continues to set extract.retries: 2, meaning one initial extraction plus at most two identical-request attempts around the complete validator chain. Rejections exhausted after three attempts remain nonfatal rejected outputs under current runner semantics.

Do not add repair prompts, previous candidates, rejection feedback, structured validation issues, error retryability classification, or an LLM validator.

Stage 0 — Baseline and scope control

1. Read the governing documents and inspect the current worktree. Record and preserve changes not owned by this plan.

2. Confirm that the embedded catalog loads 319 spells and 779 class memberships and that the maintained production example already contains extract.retries: 2.

3. Run the baseline commands:

   go test ./...
   go vet ./...
   go build ./cmd/notarius
   

4. If the baseline fails for an unrelated reason, stop and report it. Do not modify unrelated behavior or tests to obtain a clean baseline.

Stage 0 is complete when the baseline and preserved worktree scope are known.

Stage 1 — Inject materialized references during preparation

1. Add References contracts.ReferenceSet to pipeline.BuildRequest and clone it in cloneBuildRequest.
2. Refactor pipeline.Prepare, prepareLane, and prepareValidatorChain so each builder receives the correct materialized reference target described in the fixed decisions. Do not change operation-request delivery.
3. Update builder/preparation tests to prove:
   • construction receives the correct target-specific reference content;
   • extractor and extract validators receive independent clones of the same logical set;
   • mutation by one builder cannot affect another builder, the resolved pipeline, or runtime requests; and
   • empty/unmaterialized reference targets remain valid for callers that use Prepare directly.
4. Update Pipeline Internals and Module Internals to describe construction-time reference delivery as current behavior once the code lands.
5. Run focused framework tests plus all repository-wide validation commands.

Do not change configuration syntax, reference multiplicity, reference materialization, manifests, or checkpoint formats in this stage.

Stage 1 is complete when all stage builders receive isolated target references at preparation and existing pipelines behave unchanged.

Stage 2 — Implement overlay parsing and effective catalog composition

1. Add package-owned overlay DTOs, strict decoding, semantic validation, and immutable effective-catalog composition under internal/modules/dnd/spells/catalog. Keep overlay DTOs private; expose only the minimal effective-catalog API needed by extraction and validation.
2. Preserve LoadSRD5E2014, existing SRD metadata, and the existing Spell API. Do not represent overlay-only entries as incomplete SRD Spell values.
3. Add a helper that resolves the optional spell_catalog slot from a cloned ReferenceSet, enforces zero-or-one JSON item, and returns the base-only or base-plus-overlay effective catalog. Keep all filesystem access in framework reference materialization.
4. Add focused behavioral tests for:
   • base-only canonical names, normalized lookup, and immutability;
   • a new overlay spell and an alias accepted only by lookup;
   • augmentation of an existing canonical spell;
   • deterministic results and semantic digest under reordered JSON catalogs, spells, aliases, and object keys;
   • duplicate IDs, wrong ruleset/schema version, empty required values, unknown fields, trailing JSON, and empty bundles;
   • same-key display conflicts and every cross-spell canonical/alias collision category; and
   • rejection of multiple items or a non-JSON item in the reference slot.
5. Add the external overlay format contract under docs/integrations/ and update the internal catalog inventory only after the behavior exists.
6. Run the catalog package tests, import-boundary tests, and repository-wide validation commands.

Prefer table-driven coverage for meaningful invalid-input categories, but do not duplicate every parser case across higher layers.

Stage 2 is complete when one immutable effective catalog deterministically represents the embedded base plus a valid overlay bundle and rejects ambiguous composition before execution.

Stage 3 — Ground the spell extractor

1. Declare the optional spell_catalog reference slot on both the extractor runtime and ModuleSpec, accepting only application/json, Multiple: false, and MaxBytes: 1048576. Keep the existing campaign-reference slots unchanged.
2. Update the extractor builder and constructor to resolve and retain the effective catalog at preparation. A malformed overlay must make Prepare fail before raw input is read or the LLM is called.
3. Generate the canonical-name-only JSON prompt material from the retained catalog and add it to every structured completion request. Extend the Scriptorium prompt declaration and package-owned prompt assets accordingly.
4. Update prompt wording to distinguish the allowed spell-name catalog from campaign references and source evidence. Leave both response schemas unchanged.
5. Add effective catalog identity to extractor manifest metadata without exposing names, aliases, or raw overlay content. Include the new prompt fragment in prompt hashing.
6. Update focused tests to protect:
   • base-only requests contain all and only sorted canonical names;
   • aliases and non-name metadata do not appear in prompt input;
   • overlay canonical names do appear;
   • prompt/schema preparation succeeds offline with the required input;
   • prompt and manifest diagnostics omit source, reference, overlay, alias, and catalog-name content; and
   • malformed overlays fail construction without invoking the fake LLM.
7. Update the configuration reference-slot catalog, internal module docs, LLM internals where needed, and the external overlay contract in the same stage.
8. Add a safe maintained overlay example file and bind it from the production D&D example. Keep the minimal example base-only. Ensure example tests load and materialize the referenced file, rather than checking YAML syntax alone.
9. Run focused extractor, prompt-asset, CLI example, and integration tests plus all repository-wide validation commands.

Stage 3 is complete when every spell extraction is grounded by the immutable effective catalog and all overlay failures occur before paid work.

Stage 4 — Add catalog validation and production policy

1. Implement and register extract/dnd/spells/catalog as specified above, reusing the same catalog resolver as the extractor.
2. Insert it immediately after the shape validator in internal/modules/dnd/register. Do not reorder or otherwise change the remaining default validators.
3. Add package-level tests for canonical names, lookup normalization, aliases, overlay spells, multiple unknown casts, the 20-issue message bound, shape deferral, immutability, strict empty options, execution class, and registration.
4. Update production composition tests to protect registration, exact default chain placement, and construction from the same materialized overlay.
5. Add one assembled offline retry test using the production D&D registration and a fake structured LLM, with two subcases: one remains unknown through exhaustion, and one becomes catalog-valid on a retry. Configure a test-local retry count and assert calls are bounded by retries + 1, exhausted rejection is nonfatal, no rejected attempt advances to merge, and only the accepted attempt contributes warnings and output. Do not repeat generic retry cases already owned by framework tests.
6. Update the implemented-validator catalog in Configuration and the D&D validator inventory in Module Internals.
7. Run focused validator, registrar, integration, and CLI tests plus all repository-wide validation commands.

Stage 4 is complete when no artifact containing a name outside the effective catalog can pass the production spell extraction chain.

Stage 5 — Verify provenance, identity, and assembled behavior

1. Add or extend the narrowest existing CLI/checkpoint contract tests to prove that changing overlay file bytes changes checkpoint identity through reference provenance, while semantically reordered overlay content retains the same extractor-reported effective catalog digest. Do not alter checkpoint schemas or path formats.

2. Verify that changing the configured overlay binding changes resolved pipeline identity, and that manifest reference provenance records the overlay origin, media type, size, and raw digest without content.

3. Exercise a representative production-composed run with an overlay-only canonical spell and confirm prompt grounding, validation acceptance, output, manifest catalog metadata, and reference provenance agree.

4. Verify base-only configuration remains valid and produces no overlay provenance or overlay IDs.

5. Run:

   go test ./...
   go vet ./...
   go build ./cmd/notarius
   go test -race ./internal/framework/pipeline ./internal/cli ./internal/modules/dnd/extract/spells ./internal/modules/dnd/validate/spells/catalog ./internal/modules/integration
   

Stage 5 is complete when the assembled offline workflow protects catalog grounding, validation, provenance, retry count, and checkpoint invalidation at their appropriate test boundaries.

Stage 6 — Documentation, evaluation handoff, and completion

1. Review every affected current-behavior owner and remove roadmap-style wording from implemented contracts:
   • docs/config.md owns the slot name, media type, byte limit, binding form, validator key, and default chain;
   • the new docs/integrations/ document owns overlay JSON shape and compatibility rules;
   • docs/internal/pipeline.md, docs/internal/modules.md, and docs/internal/overview.md own construction and component behavior;
   • maintained copyable files remain under examples/; and
   • operations documentation changes only if runtime handling actually changes.
2. Check all links and ensure examples contain no credentials, private paths, transcripts, or copyrighted spell descriptions.
3. If an approved LLM profile and human-reviewed transcript corpus are available, run the base-only and overlay-capable baseline with retries: 2. Record anonymized aggregate false positives, false negatives, unknown-name rejections, acceptance by attempt, and model-call cost in the feature roadmap without committing sensitive transcripts or raw prompts.
4. If those external evaluation inputs are unavailable, mark evaluation as pending external execution and provide the operator with the exact config and command used by the maintained example. Do not fabricate quality results, invoke a paid model without authorization, or make live evaluation part of the default Go test suite.
5. Update D&D Spell Extraction Quality status to distinguish completed implementation from pending or completed evaluation. Remove completed baseline items from Future Work, retaining a concise link for deferred repair-aware retry and LLM-validator work.
6. Run git diff --check and the full Stage 5 validation set once more after documentation and example changes.

Stage 6 is complete when current behavior is documented in its canonical owners, the feature roadmap accurately records implementation/evaluation status, deferred work remains unimplemented, and all required checks pass.

Explicit Non-Goals

• Changing the framework-wide retry default or retry orchestration.
• Supplying rejected candidates or validation feedback to retry attempts.
• Structured validation-issue contracts or retryability classification.
• An LLM-backed spell validator.
• Spell-name mutation or catalog-aware normalization.
• Multiple reference bindings per generic slot or list-valued reference configuration.
• Spell descriptions, mechanics, unknown overlay levels, or class metadata.
• A configuration-version bump, environment override, checkpoint migration, or checkpoint-format change.

Open Questions

None. The overlay transport, file contract, composition policy, preparation boundary, prompt content, validator behavior, retry baseline, provenance, and test ownership are fixed by this plan.