notarius/docs/roadmap/implementation.md

# D&D NPC Extraction And Registry Implementation Plan

Status: Complete.

Implement this plan in order. The feature policy and target state are defined
in [D&D NPC Extraction And Registry](dnd-npc-extraction.md); this document owns
implementation sequencing and concrete technical decisions.

Do not introduce a DAG, implicit prior-run discovery, an LLM-backed validator,
or generic semantic deduplication. Preserve the fixed
`input -> chunk -> extract -> merge -> normalize -> output` architecture and
the existing framework retry, warning, rejection, checkpoint, debug, and
output contracts.

## Cross-Stage Decisions

### Artifact And Module Identities

Use these exact production identities:

- artifact kind: `dnd/npc-list`;
- extractor key: `dnd/npcs`;
- normalizer key: `dnd/npcs`;
- extractor capability: `dnd.npcs`;
- prompt ID: `dnd.npcs`, version `v1`;
- private LLM response-schema key: `dnd_npcs_llm`;
- private schema ID: `notarius.dnd.npcs.llm`;
- private schema name: `notarius_dnd_npcs_llm_v1`;
- durable codec schema ID: `notarius.dnd.npcs`;
- durable codec schema name: `notarius_dnd_npcs_v1`;
- durable schema version: `v1`; and
- durable media type: `application/json`.

The private LLM schema omits framework-assigned `id` and `source_id` fields.
The durable codec schema includes both.

### Durable Go And JSON Shape

Add the following canonical D&D types in `internal/modules/dnd/types.go`:

```go
type NPCList struct {
    NPCs []NPC `json:"npcs"`
}

type NPC struct {
    ID            string            `json:"id"`
    Name          string            `json:"name"`
    Aliases       []string          `json:"aliases"`
    Description   string            `json:"description"`
    Relationships []NPCRelationship `json:"relationships"`
    SourceRefs    []source.SourceRef `json:"source_refs"`
}

type NPCRelationship struct {
    Target       string `json:"target"`
    Relationship string `json:"relationship"`
}
```

All six NPC fields and both relationship fields are required in durable JSON.
`npcs`, `aliases`, and `relationships` must be present arrays; the first may be
empty and the latter two may be empty for an individual record. `source_refs`
must contain at least one item. Required strings must be non-empty after
trimming. Unknown JSON fields are rejected.

One NPC-level `source_refs` collection collectively supports the canonical
name, aliases, description, and relationships. Do not add per-alias or
per-relationship evidence in this version.

### Canonical Identity And IDs

Create `internal/modules/dnd/npcs/identity` as the sole owner of NPC identity
comparison and ID derivation. Define comparison keys by applying, in order:

1. Unicode NFKC normalization;
2. replacement of U+2018, U+2019, and U+02BC with ASCII apostrophe;
3. Unicode whitespace collapsing with `strings.Fields` and one ASCII space;
4. Unicode case folding with `golang.org/x/text/cases.Fold`.

Do not alter display punctuation merely to match a comparison key. Display
normalization trims and collapses whitespace but otherwise retains the first
observed spelling.

For a non-empty comparison key, derive the ID as:

```text
npc:sha256:<64 lowercase hexadecimal SHA-256 characters>
```

The hash input is the UTF-8 comparison key with no prefix, suffix, separator,
or salt. An empty identity produces an empty ID and is rejected by shape
validation. Export a stable identity-policy value
`dnd.npcs.identity.v1`; components that depend on these rules use it as their
`identity_policy` checkpoint fingerprint. A future semantic identity change
must change this policy value.

### Deterministic Consolidation

Normalizer identity components are built in merged input order. Union two
records only when:

- their canonical-name comparison keys match; or
- either record's canonical-name key appears in the other record's alias keys.

Do not union records merely because their alias sets intersect. This prevents a
shared title from silently merging otherwise distinct NPCs. After
consolidation, an alias owned by more than one retained component, or an alias
matching another retained canonical name, is an identity collision and causes
normalize-stage validation rejection.

The first record in a component supplies the retained canonical display name,
description, and output position. Add every distinct later canonical display
name to its aliases, followed by later aliases in encounter order. Union
relationships by the pair of target and relationship comparison keys, and
union exact source references. Recompute the retained ID from the retained
canonical name. Never ask an LLM to choose merged prose.

After components are known, rewrite a relationship target to a retained NPC's
canonical display name when its comparison key matches exactly one retained
canonical name or alias. Preserve targets that have no NPC match; they may name
a PC, group, place, or other non-registry entity. An ambiguous target remains
unchanged and is covered by identity-collision validation when the ambiguity is
an alias collision.

### Validation And Diagnostics

All new production validators are deterministic. Use these exact keys and
reason codes:

| Validator key | Rejection/warning reason |
| --- | --- |
| `extract/dnd/npcs/shape` | `invalid_npc_shape` |
| `extract/dnd/npcs/source_refs` | `invalid_npc_source_refs` |
| `extract/dnd/npcs/source_relatedness` | warning `npc_not_near_source` |
| `normalize/dnd/npcs/identity` | `invalid_npc_identity` |

Shape validation owns required fields and non-empty values. Source-reference
validation owns document identity, unit existence, and range ordering through
`source.ValidateRef`. Identity validation owns ID syntax and recomputation,
duplicate canonical identities, duplicate IDs, duplicate aliases, aliases
equal to their own canonical name, cross-record alias/canonical collisions, and
cross-record alias ownership.

Relatedness approves the artifact and emits at most one warning per NPC when
neither its canonical name nor any alias appears in the combined cited source
text after the same Unicode comparison normalization. Opaque campaign
references may legitimately explain such a warning; do not treat them as
source evidence or attempt to parse them deterministically.

Bound diagnostics using the established spell-validator policy: display at
most 20 issues, truncate displayed identity values to 128 Unicode code points
with an ellipsis, keep encoded rejection messages at or below 4,096 bytes, and
report the total omitted issue count. Preserve valid UTF-8 and use Go quoting
for control characters. Warnings must likewise use bounded displayed values.

Each new validator implements `CheckpointFingerprintProvider` with one local
`policy` fingerprint. Use these exact values:

- shape: `dnd.npcs.validator.shape.v1`;
- source references: `dnd.npcs.validator.source_refs.v1`;
- source relatedness: `dnd.npcs.validator.source_relatedness.v1`; and
- identity: `dnd.npcs.identity.v1`.

Bump only the affected value when validator semantics change.

### Testing Rules

Follow `docs/policy/testing.md`. Tests must be offline and deterministic. Use a
fake structured LLM only at the external completion boundary. Protect durable
artifact contracts, identity and normalization invariants, validation outcomes,
registration, checkpoint identity, and representative assembled workflows.

Do not add tests that require specific words or phrases to remain in prompt
prose. Prompt-asset tests may verify that assets register, required inputs are
wired, the selected prompt and schema identities are correct, and raw material
does not leak into diagnostics. Human review, not exact model-output fixtures,
owns semantic extraction quality.

## Stage 1: NPC Domain Contract, Identity Policy, And Codec

### Goal

Establish the typed artifact, deterministic identity primitives, and durable
serialization contract without registering a selectable production module.

### Changes

- Add `NPCListKind`, `NPCList`, `NPC`, and `NPCRelationship` to the canonical
  D&D model using the exact shape above.
- Add `internal/modules/dnd/npcs/identity` with defensive, concurrency-safe pure
  functions for display normalization, comparison keys, ID derivation, ID
  syntax checks, and whole-registry identity validation.
- Identity validation must return structured or otherwise inspectable issues so
  the normalize validator can build bounded aggregate diagnostics. Do not make
  it depend on pipeline validator types.
- Add `internal/modules/dnd/codec/npcs`, following the spell codec's candidate
  versus approved encode/decode boundary:
  - strict single-value JSON decoding with unknown-field rejection;
  - `EncodeCandidate` and `DecodeCandidate` preserve validator-visible invalid
    values;
  - approved `Encode` and `Decode` enforce structural validity;
  - schema and metadata are defensive copies; and
  - metadata reports `npc_count` only.
- Add the durable JSON Schema at
  `internal/modules/dnd/codec/npcs/assets/schemas/dnd_npcs.v1.json` with the
  exact required fields, ID pattern, array rules, source-reference shape, and
  `additionalProperties: false` at every object level.
- Create `docs/integrations/dnd-npc-artifacts.md` as the canonical durable
  contract. At this stage describe the implemented artifact and codec only; do
  not yet claim that production pipelines can select it.

### Tests

- Test identity equivalence across case, Unicode compatibility forms,
  whitespace, and supported apostrophes; different names must remain distinct.
- Test the exact ID format, deterministic repeatability, empty-name behavior,
  and registry identity issues for every collision category.
- Test codec candidate preservation, approved round trips, schema identity and
  defensive copies, nil-versus-empty arrays, malformed/trailing/unknown JSON,
  required strings, relationship shape, ID pattern, and source-reference
  structure.
- Keep one small intentional durable JSON fixture for round-trip compatibility;
  do not snapshot incidental error text.

### Completion Check

Run `go test ./internal/modules/dnd/npcs/... ./internal/modules/dnd/codec/npcs` and
`git diff --check`.

## Stage 2: NPC Extractor And Extraction Validators

### Goal

Add a package-complete LLM-backed NPC extractor and deterministic extraction
validation, still without composing it into the production D&D registrar.

### Changes

- Add `internal/modules/dnd/extract/npcs`, mirroring the established spell
  extractor boundaries without sharing spell-specific code.
- The module spec requires `chunks` and `source.transcript`, provides
  `dnd.npcs`, uses artifact kind `dnd/npc-list`, accepts no options, and declares
  the existing optional `players`, `party`, `glossary`, and deprecated `roster`
  campaign slots with the existing shared media types.
- Embed and register package-owned prompt assets and a private LLM JSON Schema.
  Reuse the shared D&D system, transcript, and campaign-reference prompt
  fragments through `shared.ModulePromptFS`.
- The prompt implements the feature-roadmap inclusion and exclusion policy,
  requires canonical in-world names rather than speakers, keeps descriptions
  concise, requires every claim to be supported by transcript citations, and
  treats auxiliary references only as disambiguation.
- The private response has top-level `npcs`; each item contains `name`,
  `aliases`, `description`, `relationships`, and source ranges without
  `source_id`. Require `aliases` and `relationships` arrays even when empty.
- Preserve malformed model values for validators. Canonicalize and de-duplicate
  exact source ranges, order response records by earliest valid cited unit, map
  every range to the current source ID, and assign IDs with the identity
  package. Do not merge NPC records in the extractor.
- Validate constructor dependencies and embedded prompt/schema metadata during
  preparation. Expose manifest-safe prompt and private-schema identities plus
  `identity_policy`; expose `prompt`, `response_schema`, and `identity_policy`
  checkpoint fingerprints.
- Add typed validators under `internal/modules/dnd/validate/npcs/shape`,
  `source_refs`, and `source_relatedness` using the contracts above. Validators
  defer appropriately after shape failure rather than emitting misleading
  secondary diagnostics.
- Give each validator a strict empty-options decoder, immutable spec, correct
  execution class, typed builder, policy fingerprint, and nil/dependency error
  behavior consistent with existing production validators.

### Tests

- Through a fake LLM, test request identity, profile/session propagation,
  chunk-scoped transcript bytes, campaign reference inputs, response mapping,
  deterministic IDs, source-ID assignment, evidence ordering, and preservation
  of invalid candidates for retries.
- Test cancellation and contextual provider errors without asserting entire
  messages.
- Test prompt and schema asset registration, input wiring, metadata redaction,
  and checkpoint fingerprints without asserting prompt prose.
- Test each validator's approval, rejection, warning, malformed-shape deferral,
  strict options, typed registration, diagnostic limits, Unicode truncation,
  and immutability.
- Test source relatedness using canonical names, aliases, case/whitespace/
  apostrophe variants, invalid ranges, and names resolved only through opaque
  references.

### Completion Check

Run `go test ./internal/modules/dnd/extract/npcs ./internal/modules/dnd/validate/npcs/...`
and `git diff --check`.

## Stage 3: Deterministic NPC Normalization

### Goal

Implement conservative cross-chunk identity consolidation and final identity
validation without changing framework normalization contracts.

### Changes

- Add `internal/modules/dnd/normalize/npcs` with key `dnd/npcs`, artifact kind
  `dnd/npc-list`, no options, no references, requirements `merged`, and
  capability `normalized`.
- Deep-clone all nested slices before mutation. A result must never alias the
  merged input or another retained record.
- Per record, normalize display whitespace, trim description and relationship
  strings, de-duplicate aliases and relationships by comparison keys, remove
  aliases equal to the canonical name, sort and exactly de-duplicate source
  references, and recompute the ID.
- Build and merge identity components with the exact cross-stage algorithm.
  Preserve component and member encounter order. Add removed canonical names as
  aliases, union provenance, retain only the first description, and
  canonicalize unambiguous relationship targets after all components exist.
- Use these stable warning reason codes:
  - `npc_fields_normalized`;
  - `npc_id_recomputed`;
  - `source_references_normalized`;
  - `duplicate_npc_collapsed`; and
  - `relationship_target_canonicalized`.
- Warning scopes use merged input indexes such as `npcs[3]`. A collapsed-group
  warning is scoped to the retained input index and reports bounded removed
  indexes. Accepted-only warning promotion remains framework policy.
- Expose manifest metadata `identity_policy` and
  `normalization_policy: dnd.npcs.normalize.v1`. Provide those same two local
  checkpoint fingerprints. Bump the normalization value whenever consolidation
  or retained-field semantics change.
- Add `internal/modules/dnd/validate/npcs/identity`. It delegates identity issue
  detection to the domain identity package, builds bounded diagnostics, and is
  registered only in the normalize default chain during Stage 4.

### Tests

- Use table-driven domain cases for per-record normalization and alias,
  relationship, source-reference, and ID behavior.
- Use a compact accepted-output fixture for representative multi-record
  consolidation: exact canonical match, canonical-to-alias match, transitive
  unambiguous match, shared-alias non-merge, first-description retention,
  relationship target rewriting, and stable ordering.
- Prove ambiguous shared aliases remain separate for identity-validator
  rejection and that invalid evidence is not made valid by normalization.
- Prove input immutability and independent nested output storage.
- Test warning scopes/reason codes, normalization and identity fingerprints,
  strict options, registration, cancellation, nil input, and bounded identity
  rejection diagnostics.
- Do not separately test private union-find or graph helpers when the public
  normalizer cases already protect the behavior.

### Completion Check

Run `go test ./internal/modules/dnd/normalize/npcs ./internal/modules/dnd/validate/npcs/identity`
and `git diff --check`.

## Stage 4: Production Composition And NPC Pipeline

### Goal

Make the complete NPC lane selectable and verify the assembled framework path
before adding it as a reference consumer elsewhere.

### Changes

- Extend `internal/modules/dnd/register` to register:
  - the NPC codec;
  - NPC extractor and prompt assets;
  - typed `appendorder` merger specialization for `dnd.NPCList`;
  - NPC normalizer and typed `noop` specialization;
  - all four NPC validators;
  - typed always-accept and always-reject validator variants; and
  - default validator-chain mappings.
- Add a small `appendNPCLists` function that preserves chunk and record order
  and distinguishes nil from present-empty output consistently with the spell
  merger.
- Register the extract default chain in this order:
  `generic/valid_json`, `generic/valid_json_schema`, NPC shape, NPC source refs,
  NPC source relatedness.
- Register the normalize default chain in this order:
  `generic/valid_json`, `generic/valid_json_schema`, NPC shape, NPC identity,
  NPC source refs, NPC source relatedness.
- Do not register an NPC merge default chain and do not change global framework
  defaults.
- Add a synthetic, non-sensitive Seriatim integration fixture with a PC, a
  repeated named NPC, an alias, an unnamed but distinguishable NPC, and an
  interchangeable group that the fake response omits.
- Add `examples/dnd-npcs.config.yml` as a complete version-3 configuration. Use
  a generic chunker, an `npcs` artifact lane, `dnd/npcs` extraction with
  `retries: 2`, `dnd/npcs` normalization, explicit checkpoint `enabled: false`,
  and safe relative output/debug paths.

### Tests

- Extend family registration tests for keys, artifact variants, default chain
  order, prompt/schema assets, nil registry failures, and duplicate
  registration errors. Test the public catalog outcome, not private registrar
  call order.
- Add config-resolution coverage for the assembled NPC lane, capabilities,
  typed codec/merger/normalizer compatibility, strict options, references, and
  invalid validator placement.
- Add one runner integration test using a fake LLM that exercises
  Seriatim -> chunks -> NPC extract -> validators -> append merge -> NPC
  normalize -> validators -> JSON output. Assert normalized identities,
  provenance, warnings, manifest metadata, and durable lane schema.
- Add one representative retry/rejection case at the assembled boundary only
  if existing generic runner tests do not already protect the same mechanism;
  do not duplicate the framework retry matrix.
- Verify the maintained example through the existing config/example contract
  tests.

### Completion Check

Run `go test ./internal/modules/dnd/register ./internal/modules/integration ./internal/core/config`
and `git diff --check`.

## Stage 5: NPC Registry Reference For Spell Extraction

### Goal

Complete the operator-driven sequential workflow by making normalized NPC JSON
an explicit, validated spell-extractor reference.

### Changes

- Add an optional `npcs` reference slot to the spell extractor only. Accept
  exactly one `application/json` item with a maximum size of 1,048,576 bytes.
  Do not add it to the shared scene or NPC-extractor campaign slots.
- During spell-extractor preparation, when `npcs` is bound:
  - require exactly one item;
  - decode it with the approved NPC codec;
  - validate its IDs and alias ownership through the NPC identity package;
  - re-encode canonical durable JSON for the prompt; and
  - compute a semantic SHA-256 digest over those canonical bytes.
- Permit source references inside the registry to identify another session;
  they are registry provenance, not spell evidence. Do not validate them
  against the current transcript.
- When the slot is unbound, supply the prompt with exactly `{"npcs":[]}` as the
  empty structured value. Do not create reference manifest provenance or an
  `npc_registry` fingerprint for the absent slot.
- Declare an optional `application/json` `npcs` prompt input and render it in a
  spell-owned prompt message. Do not modify the shared D&D reference fragment,
  because the scene and NPC prompts do not consume this registry.
- Instruct the spell model to prefer exact canonical NPC names, recognize
  aliases, and never treat registry content as proof that a spell was cast or
  as source evidence. Preserve the existing party/player policy for PCs.
- When bound, add manifest metadata `npc_registry_digest` and `npc_count`, and
  add local checkpoint fingerprint `npc_registry` with the semantic digest.
  Never expose names, aliases, reference content, paths, or raw bytes in
  metadata or fingerprints. Existing raw reference provenance continues to
  participate independently in checkpoint identity.
- Add `examples/dnd-npc-spell-sequential.config.yml` with separate `dnd-npcs`
  and `dnd-spells` pipeline definitions over the same Seriatim input shape. The
  NPC output remains intentionally unbound in the static spell definition
  because its run directory is dynamic; demonstrate the explicit runtime
  binding in documentation.
- Update current-behavior documentation in canonical locations:
  - `docs/config.md`: module and validator catalogs, NPC reference slot, limits,
    default chains, and maintained examples;
  - `docs/cli.md`: one explicit selector example using
    `spells.extract.npcs=<npc-run>/lanes/npcs.json`;
  - `docs/operations.md`: the two-invocation sequential workflow and state
    sensitivity;
  - `docs/integrations/dnd-npc-artifacts.md`: production identity, final
    normalization, warnings, manifest metadata, and reference-consumer
    semantics;
  - `docs/integrations/dnd-spell-artifacts.md`: optional NPC grounding and its
    non-evidence rule;
  - `docs/integrations/json-output.md`: link the NPC lane payload contract;
  - `docs/internal/overview.md`, `docs/internal/modules.md`, and
    `docs/internal/llm.md`: implemented package inventory and internal prompt,
    preparation, fingerprint, and registration behavior.
- At feature completion, mark the feature roadmap and this implementation plan
  complete. Remove the now-implemented NPC item from `future.md`; keep combat
  turns as the next future sequential artifact. Do not leave future behavior in
  current-behavior documentation.

### Tests

- Test spell preparation with an absent registry, a valid normalized registry,
  malformed JSON, unknown fields, invalid IDs, alias collisions, multiple
  items, wrong media type, and the configured byte limit. Materialization-limit
  failures must occur before pipeline execution and checkpoint construction.
- Through the fake spell client, assert canonical NPC JSON input, input
  metadata, manifest-safe count/digest, and absence of content leakage. Do not
  assert prompt wording.
- Test that identical semantic registries produce the same component
  fingerprint, semantic changes change it, and returned fingerprints cannot be
  mutated through caller-owned slices. Do not repeat generic checkpoint ordering
  tests already owned by the framework.
- Add one sequential integration case: run the assembled NPC pipeline with a
  fake NPC response, serialize its normalized lane payload, materialize that
  payload as the spell extractor's `npcs` reference, and run the spell pipeline
  with a fake spell response. Verify reference provenance, prompt input, typed
  output, and that no NPC source reference becomes spell evidence.
- Validate all new documentation links and maintained example configurations.

### Completion Check

Run the final verification suite below.

## Final Verification

Run:

```sh
git diff --check
go test ./...
go vet ./...
go build ./cmd/notarius
go test -race ./internal/modules/dnd/... ./internal/framework/pipeline \
  ./internal/cli ./internal/modules/integration
```

Review the final diff for:

- accidental framework or provider-specific D&D behavior;
- mutation of caller-owned artifacts, schemas, references, metadata, or
  fingerprints;
- prompt/schema/content leakage into manifests, errors, checkpoint identity,
  or redacted summaries;
- current documentation claiming behavior before its implementing stage exists;
- redundant tests or prompt-prose change detectors; and
- unrelated changes in the worktree.

## Open Questions

None. The artifact shape, identity algorithm, merge policy, validator placement,
reference parsing, checkpoint semantics, and documentation ownership are fixed
by this plan.