notarius/docs/roadmap/implementation.md

D&D NPC Extraction And Registry Implementation Plan

Status: Ready for implementation.

Implement this plan in order. The feature policy and target state are defined in D&D NPC Extraction And Registry; this document owns implementation sequencing and concrete technical decisions.

Do not introduce a DAG, implicit prior-run discovery, an LLM-backed validator, or generic semantic deduplication. Preserve the fixed input -> chunk -> extract -> merge -> normalize -> output architecture and the existing framework retry, warning, rejection, checkpoint, debug, and output contracts.

Cross-Stage Decisions

Artifact And Module Identities

Use these exact production identities:

• artifact kind: dnd/npc-list;
• extractor key: dnd/npcs;
• normalizer key: dnd/npcs;
• extractor capability: dnd.npcs;
• prompt ID: dnd.npcs, version v1;
• private LLM response-schema key: dnd_npcs_llm;
• private schema ID: notarius.dnd.npcs.llm;
• private schema name: notarius_dnd_npcs_llm_v1;
• durable codec schema ID: notarius.dnd.npcs;
• durable codec schema name: notarius_dnd_npcs_v1;
• durable schema version: v1; and
• durable media type: application/json.

The private LLM schema omits framework-assigned id and source_id fields. The durable codec schema includes both.

Durable Go And JSON Shape

Add the following canonical D&D types in internal/modules/dnd/types.go:

type NPCList struct {
    NPCs []NPC `json:"npcs"`
}

type NPC struct {
    ID            string            `json:"id"`
    Name          string            `json:"name"`
    Aliases       []string          `json:"aliases"`
    Description   string            `json:"description"`
    Relationships []NPCRelationship `json:"relationships"`
    SourceRefs    []source.SourceRef `json:"source_refs"`
}

type NPCRelationship struct {
    Target       string `json:"target"`
    Relationship string `json:"relationship"`
}

All six NPC fields and both relationship fields are required in durable JSON. npcs, aliases, and relationships must be present arrays; the first may be empty and the latter two may be empty for an individual record. source_refs must contain at least one item. Required strings must be non-empty after trimming. Unknown JSON fields are rejected.

One NPC-level source_refs collection collectively supports the canonical name, aliases, description, and relationships. Do not add per-alias or per-relationship evidence in this version.

Canonical Identity And IDs

Create internal/modules/dnd/npcs/identity as the sole owner of NPC identity comparison and ID derivation. Define comparison keys by applying, in order:

1. Unicode NFKC normalization;
2. replacement of U+2018, U+2019, and U+02BC with ASCII apostrophe;
3. Unicode whitespace collapsing with strings.Fields and one ASCII space;
4. Unicode case folding with golang.org/x/text/cases.Fold.

Do not alter display punctuation merely to match a comparison key. Display normalization trims and collapses whitespace but otherwise retains the first observed spelling.

For a non-empty comparison key, derive the ID as:

npc:sha256:<64 lowercase hexadecimal SHA-256 characters>

The hash input is the UTF-8 comparison key with no prefix, suffix, separator, or salt. An empty identity produces an empty ID and is rejected by shape validation. Export a stable identity-policy value dnd.npcs.identity.v1; components that depend on these rules use it as their identity_policy checkpoint fingerprint. A future semantic identity change must change this policy value.

Deterministic Consolidation

Normalizer identity components are built in merged input order. Union two records only when:

• their canonical-name comparison keys match; or
• either record's canonical-name key appears in the other record's alias keys.

Do not union records merely because their alias sets intersect. This prevents a shared title from silently merging otherwise distinct NPCs. After consolidation, an alias owned by more than one retained component, or an alias matching another retained canonical name, is an identity collision and causes normalize-stage validation rejection.

The first record in a component supplies the retained canonical display name, description, and output position. Add every distinct later canonical display name to its aliases, followed by later aliases in encounter order. Union relationships by the pair of target and relationship comparison keys, and union exact source references. Recompute the retained ID from the retained canonical name. Never ask an LLM to choose merged prose.

After components are known, rewrite a relationship target to a retained NPC's canonical display name when its comparison key matches exactly one retained canonical name or alias. Preserve targets that have no NPC match; they may name a PC, group, place, or other non-registry entity. An ambiguous target remains unchanged and is covered by identity-collision validation when the ambiguity is an alias collision.

Validation And Diagnostics

All new production validators are deterministic. Use these exact keys and reason codes:

Validator key	Rejection/warning reason
extract/dnd/npcs/shape	invalid_npc_shape
extract/dnd/npcs/source_refs	invalid_npc_source_refs
extract/dnd/npcs/source_relatedness	warning npc_not_near_source
normalize/dnd/npcs/identity	invalid_npc_identity

Shape validation owns required fields and non-empty values. Source-reference validation owns document identity, unit existence, and range ordering through source.ValidateRef. Identity validation owns ID syntax and recomputation, duplicate canonical identities, duplicate IDs, duplicate aliases, aliases equal to their own canonical name, cross-record alias/canonical collisions, and cross-record alias ownership.

Relatedness approves the artifact and emits at most one warning per NPC when neither its canonical name nor any alias appears in the combined cited source text after the same Unicode comparison normalization. Opaque campaign references may legitimately explain such a warning; do not treat them as source evidence or attempt to parse them deterministically.

Bound diagnostics using the established spell-validator policy: display at most 20 issues, truncate displayed identity values to 128 Unicode code points with an ellipsis, keep encoded rejection messages at or below 4,096 bytes, and report the total omitted issue count. Preserve valid UTF-8 and use Go quoting for control characters. Warnings must likewise use bounded displayed values.

Each new validator implements CheckpointFingerprintProvider with one local policy fingerprint. Use these exact values:

• shape: dnd.npcs.validator.shape.v1;
• source references: dnd.npcs.validator.source_refs.v1;
• source relatedness: dnd.npcs.validator.source_relatedness.v1; and
• identity: dnd.npcs.identity.v1.

Bump only the affected value when validator semantics change.

Testing Rules

Follow docs/policy/testing.md. Tests must be offline and deterministic. Use a fake structured LLM only at the external completion boundary. Protect durable artifact contracts, identity and normalization invariants, validation outcomes, registration, checkpoint identity, and representative assembled workflows.

Do not add tests that require specific words or phrases to remain in prompt prose. Prompt-asset tests may verify that assets register, required inputs are wired, the selected prompt and schema identities are correct, and raw material does not leak into diagnostics. Human review, not exact model-output fixtures, owns semantic extraction quality.

Stage 1: NPC Domain Contract, Identity Policy, And Codec

Goal

Establish the typed artifact, deterministic identity primitives, and durable serialization contract without registering a selectable production module.

Changes

• Add NPCListKind, NPCList, NPC, and NPCRelationship to the canonical D&D model using the exact shape above.
• Add internal/modules/dnd/npcs/identity with defensive, concurrency-safe pure functions for display normalization, comparison keys, ID derivation, ID syntax checks, and whole-registry identity validation.
• Identity validation must return structured or otherwise inspectable issues so the normalize validator can build bounded aggregate diagnostics. Do not make it depend on pipeline validator types.
• Add internal/modules/dnd/codec/npcs, following the spell codec's candidate versus approved encode/decode boundary:
  • strict single-value JSON decoding with unknown-field rejection;
  • EncodeCandidate and DecodeCandidate preserve validator-visible invalid values;
  • approved Encode and Decode enforce structural validity;
  • schema and metadata are defensive copies; and
  • metadata reports npc_count only.
• Add the durable JSON Schema at internal/modules/dnd/codec/npcs/assets/schemas/dnd_npcs.v1.json with the exact required fields, ID pattern, array rules, source-reference shape, and additionalProperties: false at every object level.
• Create docs/integrations/dnd-npc-artifacts.md as the canonical durable contract. At this stage describe the implemented artifact and codec only; do not yet claim that production pipelines can select it.

Tests

• Test identity equivalence across case, Unicode compatibility forms, whitespace, and supported apostrophes; different names must remain distinct.
• Test the exact ID format, deterministic repeatability, empty-name behavior, and registry identity issues for every collision category.
• Test codec candidate preservation, approved round trips, schema identity and defensive copies, nil-versus-empty arrays, malformed/trailing/unknown JSON, required strings, relationship shape, ID pattern, and source-reference structure.
• Keep one small intentional durable JSON fixture for round-trip compatibility; do not snapshot incidental error text.

Completion Check

Run go test ./internal/modules/dnd/npcs/... ./internal/modules/dnd/codec/npcs and git diff --check.

Stage 2: NPC Extractor And Extraction Validators

Goal

Add a package-complete LLM-backed NPC extractor and deterministic extraction validation, still without composing it into the production D&D registrar.

Changes

• Add internal/modules/dnd/extract/npcs, mirroring the established spell extractor boundaries without sharing spell-specific code.
• The module spec requires chunks and source.transcript, provides dnd.npcs, uses artifact kind dnd/npc-list, accepts no options, and declares the existing optional players, party, glossary, and deprecated roster campaign slots with the existing shared media types.
• Embed and register package-owned prompt assets and a private LLM JSON Schema. Reuse the shared D&D system, transcript, and campaign-reference prompt fragments through shared.ModulePromptFS.
• The prompt implements the feature-roadmap inclusion and exclusion policy, requires canonical in-world names rather than speakers, keeps descriptions concise, requires every claim to be supported by transcript citations, and treats auxiliary references only as disambiguation.
• The private response has top-level npcs; each item contains name, aliases, description, relationships, and source ranges without source_id. Require aliases and relationships arrays even when empty.
• Preserve malformed model values for validators. Canonicalize and de-duplicate exact source ranges, order response records by earliest valid cited unit, map every range to the current source ID, and assign IDs with the identity package. Do not merge NPC records in the extractor.
• Validate constructor dependencies and embedded prompt/schema metadata during preparation. Expose manifest-safe prompt and private-schema identities plus identity_policy; expose prompt, response_schema, and identity_policy checkpoint fingerprints.
• Add typed validators under internal/modules/dnd/validate/npcs/shape, source_refs, and source_relatedness using the contracts above. Validators defer appropriately after shape failure rather than emitting misleading secondary diagnostics.
• Give each validator a strict empty-options decoder, immutable spec, correct execution class, typed builder, policy fingerprint, and nil/dependency error behavior consistent with existing production validators.

Tests

• Through a fake LLM, test request identity, profile/session propagation, chunk-scoped transcript bytes, campaign reference inputs, response mapping, deterministic IDs, source-ID assignment, evidence ordering, and preservation of invalid candidates for retries.
• Test cancellation and contextual provider errors without asserting entire messages.
• Test prompt and schema asset registration, input wiring, metadata redaction, and checkpoint fingerprints without asserting prompt prose.
• Test each validator's approval, rejection, warning, malformed-shape deferral, strict options, typed registration, diagnostic limits, Unicode truncation, and immutability.
• Test source relatedness using canonical names, aliases, case/whitespace/ apostrophe variants, invalid ranges, and names resolved only through opaque references.

Completion Check

Run go test ./internal/modules/dnd/extract/npcs ./internal/modules/dnd/validate/npcs/... and git diff --check.

Stage 3: Deterministic NPC Normalization

Goal

Implement conservative cross-chunk identity consolidation and final identity validation without changing framework normalization contracts.

Changes

• Add internal/modules/dnd/normalize/npcs with key dnd/npcs, artifact kind dnd/npc-list, no options, no references, requirements merged, and capability normalized.
• Deep-clone all nested slices before mutation. A result must never alias the merged input or another retained record.
• Per record, normalize display whitespace, trim description and relationship strings, de-duplicate aliases and relationships by comparison keys, remove aliases equal to the canonical name, sort and exactly de-duplicate source references, and recompute the ID.
• Build and merge identity components with the exact cross-stage algorithm. Preserve component and member encounter order. Add removed canonical names as aliases, union provenance, retain only the first description, and canonicalize unambiguous relationship targets after all components exist.
• Use these stable warning reason codes:
  • npc_fields_normalized;
  • npc_id_recomputed;
  • source_references_normalized;
  • duplicate_npc_collapsed; and
  • relationship_target_canonicalized.
• Warning scopes use merged input indexes such as npcs[3]. A collapsed-group warning is scoped to the retained input index and reports bounded removed indexes. Accepted-only warning promotion remains framework policy.
• Expose manifest metadata identity_policy and normalization_policy: dnd.npcs.normalize.v1. Provide those same two local checkpoint fingerprints. Bump the normalization value whenever consolidation or retained-field semantics change.
• Add internal/modules/dnd/validate/npcs/identity. It delegates identity issue detection to the domain identity package, builds bounded diagnostics, and is registered only in the normalize default chain during Stage 4.

Tests

• Use table-driven domain cases for per-record normalization and alias, relationship, source-reference, and ID behavior.
• Use a compact accepted-output fixture for representative multi-record consolidation: exact canonical match, canonical-to-alias match, transitive unambiguous match, shared-alias non-merge, first-description retention, relationship target rewriting, and stable ordering.
• Prove ambiguous shared aliases remain separate for identity-validator rejection and that invalid evidence is not made valid by normalization.
• Prove input immutability and independent nested output storage.
• Test warning scopes/reason codes, normalization and identity fingerprints, strict options, registration, cancellation, nil input, and bounded identity rejection diagnostics.
• Do not separately test private union-find or graph helpers when the public normalizer cases already protect the behavior.

Completion Check

Run go test ./internal/modules/dnd/normalize/npcs ./internal/modules/dnd/validate/npcs/identity and git diff --check.

Stage 4: Production Composition And NPC Pipeline

Goal

Make the complete NPC lane selectable and verify the assembled framework path before adding it as a reference consumer elsewhere.

Changes

• Extend internal/modules/dnd/register to register:
  • the NPC codec;
  • NPC extractor and prompt assets;
  • typed appendorder merger specialization for dnd.NPCList;
  • NPC normalizer and typed noop specialization;
  • all four NPC validators;
  • typed always-accept and always-reject validator variants; and
  • default validator-chain mappings.
• Add a small appendNPCLists function that preserves chunk and record order and distinguishes nil from present-empty output consistently with the spell merger.
• Register the extract default chain in this order: generic/valid_json, generic/valid_json_schema, NPC shape, NPC source refs, NPC source relatedness.
• Register the normalize default chain in this order: generic/valid_json, generic/valid_json_schema, NPC shape, NPC identity, NPC source refs, NPC source relatedness.
• Do not register an NPC merge default chain and do not change global framework defaults.
• Add a synthetic, non-sensitive Seriatim integration fixture with a PC, a repeated named NPC, an alias, an unnamed but distinguishable NPC, and an interchangeable group that the fake response omits.
• Add examples/dnd-npcs.config.yml as a complete version-3 configuration. Use a generic chunker, an npcs artifact lane, dnd/npcs extraction with retries: 2, dnd/npcs normalization, explicit checkpoint enabled: false, and safe relative output/debug paths.

Tests

• Extend family registration tests for keys, artifact variants, default chain order, prompt/schema assets, nil registry failures, and duplicate registration errors. Test the public catalog outcome, not private registrar call order.
• Add config-resolution coverage for the assembled NPC lane, capabilities, typed codec/merger/normalizer compatibility, strict options, references, and invalid validator placement.
• Add one runner integration test using a fake LLM that exercises Seriatim -> chunks -> NPC extract -> validators -> append merge -> NPC normalize -> validators -> JSON output. Assert normalized identities, provenance, warnings, manifest metadata, and durable lane schema.
• Add one representative retry/rejection case at the assembled boundary only if existing generic runner tests do not already protect the same mechanism; do not duplicate the framework retry matrix.
• Verify the maintained example through the existing config/example contract tests.

Completion Check

Run go test ./internal/modules/dnd/register ./internal/modules/integration ./internal/core/config and git diff --check.

Stage 5: NPC Registry Reference For Spell Extraction

Goal

Complete the operator-driven sequential workflow by making normalized NPC JSON an explicit, validated spell-extractor reference.

Changes

• Add an optional npcs reference slot to the spell extractor only. Accept exactly one application/json item with a maximum size of 1,048,576 bytes. Do not add it to the shared scene or NPC-extractor campaign slots.
• During spell-extractor preparation, when npcs is bound:
  • require exactly one item;
  • decode it with the approved NPC codec;
  • validate its IDs and alias ownership through the NPC identity package;
  • re-encode canonical durable JSON for the prompt; and
  • compute a semantic SHA-256 digest over those canonical bytes.
• Permit source references inside the registry to identify another session; they are registry provenance, not spell evidence. Do not validate them against the current transcript.
• When the slot is unbound, supply the prompt with exactly {"npcs":[]} as the empty structured value. Do not create reference manifest provenance or an npc_registry fingerprint for the absent slot.
• Declare an optional application/json npcs prompt input and render it in a spell-owned prompt message. Do not modify the shared D&D reference fragment, because the scene and NPC prompts do not consume this registry.
• Instruct the spell model to prefer exact canonical NPC names, recognize aliases, and never treat registry content as proof that a spell was cast or as source evidence. Preserve the existing party/player policy for PCs.
• When bound, add manifest metadata npc_registry_digest and npc_count, and add local checkpoint fingerprint npc_registry with the semantic digest. Never expose names, aliases, reference content, paths, or raw bytes in metadata or fingerprints. Existing raw reference provenance continues to participate independently in checkpoint identity.
• Add examples/dnd-npc-spell-sequential.config.yml with separate dnd-npcs and dnd-spells pipeline definitions over the same Seriatim input shape. The NPC output remains intentionally unbound in the static spell definition because its run directory is dynamic; demonstrate the explicit runtime binding in documentation.
• Update current-behavior documentation in canonical locations:
  • docs/config.md: module and validator catalogs, NPC reference slot, limits, default chains, and maintained examples;
  • docs/cli.md: one explicit selector example using spells.extract.npcs=<npc-run>/lanes/npcs.json;
  • docs/operations.md: the two-invocation sequential workflow and state sensitivity;
  • docs/integrations/dnd-npc-artifacts.md: production identity, final normalization, warnings, manifest metadata, and reference-consumer semantics;
  • docs/integrations/dnd-spell-artifacts.md: optional NPC grounding and its non-evidence rule;
  • docs/integrations/json-output.md: link the NPC lane payload contract;
  • docs/internal/overview.md, docs/internal/modules.md, and docs/internal/llm.md: implemented package inventory and internal prompt, preparation, fingerprint, and registration behavior.
• At feature completion, mark the feature roadmap and this implementation plan complete. Remove the now-implemented NPC item from future.md; keep combat turns as the next future sequential artifact. Do not leave future behavior in current-behavior documentation.

Tests

• Test spell preparation with an absent registry, a valid normalized registry, malformed JSON, unknown fields, invalid IDs, alias collisions, multiple items, wrong media type, and the configured byte limit. Materialization-limit failures must occur before pipeline execution and checkpoint construction.
• Through the fake spell client, assert canonical NPC JSON input, input metadata, manifest-safe count/digest, and absence of content leakage. Do not assert prompt wording.
• Test that identical semantic registries produce the same component fingerprint, semantic changes change it, and returned fingerprints cannot be mutated through caller-owned slices. Do not repeat generic checkpoint ordering tests already owned by the framework.
• Add one sequential integration case: run the assembled NPC pipeline with a fake NPC response, serialize its normalized lane payload, materialize that payload as the spell extractor's npcs reference, and run the spell pipeline with a fake spell response. Verify reference provenance, prompt input, typed output, and that no NPC source reference becomes spell evidence.
• Validate all new documentation links and maintained example configurations.

Completion Check

Run the final verification suite below.

Final Verification

Run:

git diff --check
go test ./...
go vet ./...
go build ./cmd/notarius
go test -race ./internal/modules/dnd/... ./internal/framework/pipeline \
  ./internal/cli ./internal/modules/integration

Review the final diff for:

• accidental framework or provider-specific D&D behavior;
• mutation of caller-owned artifacts, schemas, references, metadata, or fingerprints;
• prompt/schema/content leakage into manifests, errors, checkpoint identity, or redacted summaries;
• current documentation claiming behavior before its implementing stage exists;
• redundant tests or prompt-prose change detectors; and
• unrelated changes in the worktree.

Open Questions

None. The artifact shape, identity algorithm, merge policy, validator placement, reference parsing, checkpoint semantics, and documentation ownership are fixed by this plan.