Add feature roadmap and implementation plan for a D&D NPC extraction module

docs/roadmap/implementation.md

@@ -0,0 +1,528 @@
+# D&D NPC Extraction And Registry Implementation Plan
+
+Status: Ready for implementation.
+
+Implement this plan in order. The feature policy and target state are defined
+in [D&D NPC Extraction And Registry](dnd-npc-extraction.md); this document owns
+implementation sequencing and concrete technical decisions.
+
+Do not introduce a DAG, implicit prior-run discovery, an LLM-backed validator,
+or generic semantic deduplication. Preserve the fixed
+`input -> chunk -> extract -> merge -> normalize -> output` architecture and
+the existing framework retry, warning, rejection, checkpoint, debug, and
+output contracts.
+
+## Cross-Stage Decisions
+
+### Artifact And Module Identities
+
+Use these exact production identities:
+
+- artifact kind: `dnd/npc-list`;
+- extractor key: `dnd/npcs`;
+- normalizer key: `dnd/npcs`;
+- extractor capability: `dnd.npcs`;
+- prompt ID: `dnd.npcs`, version `v1`;
+- private LLM response-schema key: `dnd_npcs_llm`;
+- private schema ID: `notarius.dnd.npcs.llm`;
+- private schema name: `notarius_dnd_npcs_llm_v1`;
+- durable codec schema ID: `notarius.dnd.npcs`;
+- durable codec schema name: `notarius_dnd_npcs_v1`;
+- durable schema version: `v1`; and
+- durable media type: `application/json`.
+
+The private LLM schema omits framework-assigned `id` and `source_id` fields.
+The durable codec schema includes both.
+
+### Durable Go And JSON Shape
+
+Add the following canonical D&D types in `internal/modules/dnd/types.go`:
+
+```go
+type NPCList struct {
+    NPCs []NPC `json:"npcs"`
+}
+
+type NPC struct {
+    ID            string            `json:"id"`
+    Name          string            `json:"name"`
+    Aliases       []string          `json:"aliases"`
+    Description   string            `json:"description"`
+    Relationships []NPCRelationship `json:"relationships"`
+    SourceRefs    []source.SourceRef `json:"source_refs"`
+}
+
+type NPCRelationship struct {
+    Target       string `json:"target"`
+    Relationship string `json:"relationship"`
+}
+```
+
+All six NPC fields and both relationship fields are required in durable JSON.
+`npcs`, `aliases`, and `relationships` must be present arrays; the first may be
+empty and the latter two may be empty for an individual record. `source_refs`
+must contain at least one item. Required strings must be non-empty after
+trimming. Unknown JSON fields are rejected.
+
+One NPC-level `source_refs` collection collectively supports the canonical
+name, aliases, description, and relationships. Do not add per-alias or
+per-relationship evidence in this version.
+
+### Canonical Identity And IDs
+
+Create `internal/modules/dnd/npcs/identity` as the sole owner of NPC identity
+comparison and ID derivation. Define comparison keys by applying, in order:
+
+1. Unicode NFKC normalization;
+2. replacement of U+2018, U+2019, and U+02BC with ASCII apostrophe;
+3. Unicode whitespace collapsing with `strings.Fields` and one ASCII space;
+4. Unicode case folding with `golang.org/x/text/cases.Fold`.
+
+Do not alter display punctuation merely to match a comparison key. Display
+normalization trims and collapses whitespace but otherwise retains the first
+observed spelling.
+
+For a non-empty comparison key, derive the ID as:
+
+```text
+npc:sha256:<64 lowercase hexadecimal SHA-256 characters>
+```
+
+The hash input is the UTF-8 comparison key with no prefix, suffix, separator,
+or salt. An empty identity produces an empty ID and is rejected by shape
+validation. Export a stable identity-policy value
+`dnd.npcs.identity.v1`; components that depend on these rules use it as their
+`identity_policy` checkpoint fingerprint. A future semantic identity change
+must change this policy value.
+
+### Deterministic Consolidation
+
+Normalizer identity components are built in merged input order. Union two
+records only when:
+
+- their canonical-name comparison keys match; or
+- either record's canonical-name key appears in the other record's alias keys.
+
+Do not union records merely because their alias sets intersect. This prevents a
+shared title from silently merging otherwise distinct NPCs. After
+consolidation, an alias owned by more than one retained component, or an alias
+matching another retained canonical name, is an identity collision and causes
+normalize-stage validation rejection.
+
+The first record in a component supplies the retained canonical display name,
+description, and output position. Add every distinct later canonical display
+name to its aliases, followed by later aliases in encounter order. Union
+relationships by the pair of target and relationship comparison keys, and
+union exact source references. Recompute the retained ID from the retained
+canonical name. Never ask an LLM to choose merged prose.
+
+After components are known, rewrite a relationship target to a retained NPC's
+canonical display name when its comparison key matches exactly one retained
+canonical name or alias. Preserve targets that have no NPC match; they may name
+a PC, group, place, or other non-registry entity. An ambiguous target remains
+unchanged and is covered by identity-collision validation when the ambiguity is
+an alias collision.
+
+### Validation And Diagnostics
+
+All new production validators are deterministic. Use these exact keys and
+reason codes:
+
+| Validator key | Rejection/warning reason |
+| --- | --- |
+| `extract/dnd/npcs/shape` | `invalid_npc_shape` |
+| `extract/dnd/npcs/source_refs` | `invalid_npc_source_refs` |
+| `extract/dnd/npcs/source_relatedness` | warning `npc_not_near_source` |
+| `normalize/dnd/npcs/identity` | `invalid_npc_identity` |
+
+Shape validation owns required fields and non-empty values. Source-reference
+validation owns document identity, unit existence, and range ordering through
+`source.ValidateRef`. Identity validation owns ID syntax and recomputation,
+duplicate canonical identities, duplicate IDs, duplicate aliases, aliases
+equal to their own canonical name, cross-record alias/canonical collisions, and
+cross-record alias ownership.
+
+Relatedness approves the artifact and emits at most one warning per NPC when
+neither its canonical name nor any alias appears in the combined cited source
+text after the same Unicode comparison normalization. Opaque campaign
+references may legitimately explain such a warning; do not treat them as
+source evidence or attempt to parse them deterministically.
+
+Bound diagnostics using the established spell-validator policy: display at
+most 20 issues, truncate displayed identity values to 128 Unicode code points
+with an ellipsis, keep encoded rejection messages at or below 4,096 bytes, and
+report the total omitted issue count. Preserve valid UTF-8 and use Go quoting
+for control characters. Warnings must likewise use bounded displayed values.
+
+Each new validator implements `CheckpointFingerprintProvider` with one local
+`policy` fingerprint. Use these exact values:
+
+- shape: `dnd.npcs.validator.shape.v1`;
+- source references: `dnd.npcs.validator.source_refs.v1`;
+- source relatedness: `dnd.npcs.validator.source_relatedness.v1`; and
+- identity: `dnd.npcs.identity.v1`.
+
+Bump only the affected value when validator semantics change.
+
+### Testing Rules
+
+Follow `docs/policy/testing.md`. Tests must be offline and deterministic. Use a
+fake structured LLM only at the external completion boundary. Protect durable
+artifact contracts, identity and normalization invariants, validation outcomes,
+registration, checkpoint identity, and representative assembled workflows.
+
+Do not add tests that require specific words or phrases to remain in prompt
+prose. Prompt-asset tests may verify that assets register, required inputs are
+wired, the selected prompt and schema identities are correct, and raw material
+does not leak into diagnostics. Human review, not exact model-output fixtures,
+owns semantic extraction quality.
+
+## Stage 1: NPC Domain Contract, Identity Policy, And Codec
+
+### Goal
+
+Establish the typed artifact, deterministic identity primitives, and durable
+serialization contract without registering a selectable production module.
+
+### Changes
+
+- Add `NPCListKind`, `NPCList`, `NPC`, and `NPCRelationship` to the canonical
+  D&D model using the exact shape above.
+- Add `internal/modules/dnd/npcs/identity` with defensive, concurrency-safe pure
+  functions for display normalization, comparison keys, ID derivation, ID
+  syntax checks, and whole-registry identity validation.
+- Identity validation must return structured or otherwise inspectable issues so
+  the normalize validator can build bounded aggregate diagnostics. Do not make
+  it depend on pipeline validator types.
+- Add `internal/modules/dnd/codec/npcs`, following the spell codec's candidate
+  versus approved encode/decode boundary:
+  - strict single-value JSON decoding with unknown-field rejection;
+  - `EncodeCandidate` and `DecodeCandidate` preserve validator-visible invalid
+    values;
+  - approved `Encode` and `Decode` enforce structural validity;
+  - schema and metadata are defensive copies; and
+  - metadata reports `npc_count` only.
+- Add the durable JSON Schema at
+  `internal/modules/dnd/codec/npcs/assets/schemas/dnd_npcs.v1.json` with the
+  exact required fields, ID pattern, array rules, source-reference shape, and
+  `additionalProperties: false` at every object level.
+- Create `docs/integrations/dnd-npc-artifacts.md` as the canonical durable
+  contract. At this stage describe the implemented artifact and codec only; do
+  not yet claim that production pipelines can select it.
+
+### Tests
+
+- Test identity equivalence across case, Unicode compatibility forms,
+  whitespace, and supported apostrophes; different names must remain distinct.
+- Test the exact ID format, deterministic repeatability, empty-name behavior,
+  and registry identity issues for every collision category.
+- Test codec candidate preservation, approved round trips, schema identity and
+  defensive copies, nil-versus-empty arrays, malformed/trailing/unknown JSON,
+  required strings, relationship shape, ID pattern, and source-reference
+  structure.
+- Keep one small intentional durable JSON fixture for round-trip compatibility;
+  do not snapshot incidental error text.
+
+### Completion Check
+
+Run `go test ./internal/modules/dnd/npcs/... ./internal/modules/dnd/codec/npcs` and
+`git diff --check`.
+
+## Stage 2: NPC Extractor And Extraction Validators
+
+### Goal
+
+Add a package-complete LLM-backed NPC extractor and deterministic extraction
+validation, still without composing it into the production D&D registrar.
+
+### Changes
+
+- Add `internal/modules/dnd/extract/npcs`, mirroring the established spell
+  extractor boundaries without sharing spell-specific code.
+- The module spec requires `chunks` and `source.transcript`, provides
+  `dnd.npcs`, uses artifact kind `dnd/npc-list`, accepts no options, and declares
+  the existing optional `players`, `party`, `glossary`, and deprecated `roster`
+  campaign slots with the existing shared media types.
+- Embed and register package-owned prompt assets and a private LLM JSON Schema.
+  Reuse the shared D&D system, transcript, and campaign-reference prompt
+  fragments through `shared.ModulePromptFS`.
+- The prompt implements the feature-roadmap inclusion and exclusion policy,
+  requires canonical in-world names rather than speakers, keeps descriptions
+  concise, requires every claim to be supported by transcript citations, and
+  treats auxiliary references only as disambiguation.
+- The private response has top-level `npcs`; each item contains `name`,
+  `aliases`, `description`, `relationships`, and source ranges without
+  `source_id`. Require `aliases` and `relationships` arrays even when empty.
+- Preserve malformed model values for validators. Canonicalize and de-duplicate
+  exact source ranges, order response records by earliest valid cited unit, map
+  every range to the current source ID, and assign IDs with the identity
+  package. Do not merge NPC records in the extractor.
+- Validate constructor dependencies and embedded prompt/schema metadata during
+  preparation. Expose manifest-safe prompt and private-schema identities plus
+  `identity_policy`; expose `prompt`, `response_schema`, and `identity_policy`
+  checkpoint fingerprints.
+- Add typed validators under `internal/modules/dnd/validate/npcs/shape`,
+  `source_refs`, and `source_relatedness` using the contracts above. Validators
+  defer appropriately after shape failure rather than emitting misleading
+  secondary diagnostics.
+- Give each validator a strict empty-options decoder, immutable spec, correct
+  execution class, typed builder, policy fingerprint, and nil/dependency error
+  behavior consistent with existing production validators.
+
+### Tests
+
+- Through a fake LLM, test request identity, profile/session propagation,
+  chunk-scoped transcript bytes, campaign reference inputs, response mapping,
+  deterministic IDs, source-ID assignment, evidence ordering, and preservation
+  of invalid candidates for retries.
+- Test cancellation and contextual provider errors without asserting entire
+  messages.
+- Test prompt and schema asset registration, input wiring, metadata redaction,
+  and checkpoint fingerprints without asserting prompt prose.
+- Test each validator's approval, rejection, warning, malformed-shape deferral,
+  strict options, typed registration, diagnostic limits, Unicode truncation,
+  and immutability.
+- Test source relatedness using canonical names, aliases, case/whitespace/
+  apostrophe variants, invalid ranges, and names resolved only through opaque
+  references.
+
+### Completion Check
+
+Run `go test ./internal/modules/dnd/extract/npcs ./internal/modules/dnd/validate/npcs/...`
+and `git diff --check`.
+
+## Stage 3: Deterministic NPC Normalization
+
+### Goal
+
+Implement conservative cross-chunk identity consolidation and final identity
+validation without changing framework normalization contracts.
+
+### Changes
+
+- Add `internal/modules/dnd/normalize/npcs` with key `dnd/npcs`, artifact kind
+  `dnd/npc-list`, no options, no references, requirements `merged`, and
+  capability `normalized`.
+- Deep-clone all nested slices before mutation. A result must never alias the
+  merged input or another retained record.
+- Per record, normalize display whitespace, trim description and relationship
+  strings, de-duplicate aliases and relationships by comparison keys, remove
+  aliases equal to the canonical name, sort and exactly de-duplicate source
+  references, and recompute the ID.
+- Build and merge identity components with the exact cross-stage algorithm.
+  Preserve component and member encounter order. Add removed canonical names as
+  aliases, union provenance, retain only the first description, and
+  canonicalize unambiguous relationship targets after all components exist.
+- Use these stable warning reason codes:
+  - `npc_fields_normalized`;
+  - `npc_id_recomputed`;
+  - `source_references_normalized`;
+  - `duplicate_npc_collapsed`; and
+  - `relationship_target_canonicalized`.
+- Warning scopes use merged input indexes such as `npcs[3]`. A collapsed-group
+  warning is scoped to the retained input index and reports bounded removed
+  indexes. Accepted-only warning promotion remains framework policy.
+- Expose manifest metadata `identity_policy` and
+  `normalization_policy: dnd.npcs.normalize.v1`. Provide those same two local
+  checkpoint fingerprints. Bump the normalization value whenever consolidation
+  or retained-field semantics change.
+- Add `internal/modules/dnd/validate/npcs/identity`. It delegates identity issue
+  detection to the domain identity package, builds bounded diagnostics, and is
+  registered only in the normalize default chain during Stage 4.
+
+### Tests
+
+- Use table-driven domain cases for per-record normalization and alias,
+  relationship, source-reference, and ID behavior.
+- Use a compact accepted-output fixture for representative multi-record
+  consolidation: exact canonical match, canonical-to-alias match, transitive
+  unambiguous match, shared-alias non-merge, first-description retention,
+  relationship target rewriting, and stable ordering.
+- Prove ambiguous shared aliases remain separate for identity-validator
+  rejection and that invalid evidence is not made valid by normalization.
+- Prove input immutability and independent nested output storage.
+- Test warning scopes/reason codes, normalization and identity fingerprints,
+  strict options, registration, cancellation, nil input, and bounded identity
+  rejection diagnostics.
+- Do not separately test private union-find or graph helpers when the public
+  normalizer cases already protect the behavior.
+
+### Completion Check
+
+Run `go test ./internal/modules/dnd/normalize/npcs ./internal/modules/dnd/validate/npcs/identity`
+and `git diff --check`.
+
+## Stage 4: Production Composition And NPC Pipeline
+
+### Goal
+
+Make the complete NPC lane selectable and verify the assembled framework path
+before adding it as a reference consumer elsewhere.
+
+### Changes
+
+- Extend `internal/modules/dnd/register` to register:
+  - the NPC codec;
+  - NPC extractor and prompt assets;
+  - typed `appendorder` merger specialization for `dnd.NPCList`;
+  - NPC normalizer and typed `noop` specialization;
+  - all four NPC validators;
+  - typed always-accept and always-reject validator variants; and
+  - default validator-chain mappings.
+- Add a small `appendNPCLists` function that preserves chunk and record order
+  and distinguishes nil from present-empty output consistently with the spell
+  merger.
+- Register the extract default chain in this order:
+  `generic/valid_json`, `generic/valid_json_schema`, NPC shape, NPC source refs,
+  NPC source relatedness.
+- Register the normalize default chain in this order:
+  `generic/valid_json`, `generic/valid_json_schema`, NPC shape, NPC identity,
+  NPC source refs, NPC source relatedness.
+- Do not register an NPC merge default chain and do not change global framework
+  defaults.
+- Add a synthetic, non-sensitive Seriatim integration fixture with a PC, a
+  repeated named NPC, an alias, an unnamed but distinguishable NPC, and an
+  interchangeable group that the fake response omits.
+- Add `examples/dnd-npcs.config.yml` as a complete version-3 configuration. Use
+  a generic chunker, an `npcs` artifact lane, `dnd/npcs` extraction with
+  `retries: 2`, `dnd/npcs` normalization, explicit checkpoint `enabled: false`,
+  and safe relative output/debug paths.
+
+### Tests
+
+- Extend family registration tests for keys, artifact variants, default chain
+  order, prompt/schema assets, nil registry failures, and duplicate
+  registration errors. Test the public catalog outcome, not private registrar
+  call order.
+- Add config-resolution coverage for the assembled NPC lane, capabilities,
+  typed codec/merger/normalizer compatibility, strict options, references, and
+  invalid validator placement.
+- Add one runner integration test using a fake LLM that exercises
+  Seriatim -> chunks -> NPC extract -> validators -> append merge -> NPC
+  normalize -> validators -> JSON output. Assert normalized identities,
+  provenance, warnings, manifest metadata, and durable lane schema.
+- Add one representative retry/rejection case at the assembled boundary only
+  if existing generic runner tests do not already protect the same mechanism;
+  do not duplicate the framework retry matrix.
+- Verify the maintained example through the existing config/example contract
+  tests.
+
+### Completion Check
+
+Run `go test ./internal/modules/dnd/register ./internal/modules/integration ./internal/core/config`
+and `git diff --check`.
+
+## Stage 5: NPC Registry Reference For Spell Extraction
+
+### Goal
+
+Complete the operator-driven sequential workflow by making normalized NPC JSON
+an explicit, validated spell-extractor reference.
+
+### Changes
+
+- Add an optional `npcs` reference slot to the spell extractor only. Accept
+  exactly one `application/json` item with a maximum size of 1,048,576 bytes.
+  Do not add it to the shared scene or NPC-extractor campaign slots.
+- During spell-extractor preparation, when `npcs` is bound:
+  - require exactly one item;
+  - decode it with the approved NPC codec;
+  - validate its IDs and alias ownership through the NPC identity package;
+  - re-encode canonical durable JSON for the prompt; and
+  - compute a semantic SHA-256 digest over those canonical bytes.
+- Permit source references inside the registry to identify another session;
+  they are registry provenance, not spell evidence. Do not validate them
+  against the current transcript.
+- When the slot is unbound, supply the prompt with exactly `{"npcs":[]}` as the
+  empty structured value. Do not create reference manifest provenance or an
+  `npc_registry` fingerprint for the absent slot.
+- Declare an optional `application/json` `npcs` prompt input and render it in a
+  spell-owned prompt message. Do not modify the shared D&D reference fragment,
+  because the scene and NPC prompts do not consume this registry.
+- Instruct the spell model to prefer exact canonical NPC names, recognize
+  aliases, and never treat registry content as proof that a spell was cast or
+  as source evidence. Preserve the existing party/player policy for PCs.
+- When bound, add manifest metadata `npc_registry_digest` and `npc_count`, and
+  add local checkpoint fingerprint `npc_registry` with the semantic digest.
+  Never expose names, aliases, reference content, paths, or raw bytes in
+  metadata or fingerprints. Existing raw reference provenance continues to
+  participate independently in checkpoint identity.
+- Add `examples/dnd-npc-spell-sequential.config.yml` with separate `dnd-npcs`
+  and `dnd-spells` pipeline definitions over the same Seriatim input shape. The
+  NPC output remains intentionally unbound in the static spell definition
+  because its run directory is dynamic; demonstrate the explicit runtime
+  binding in documentation.
+- Update current-behavior documentation in canonical locations:
+  - `docs/config.md`: module and validator catalogs, NPC reference slot, limits,
+    default chains, and maintained examples;
+  - `docs/cli.md`: one explicit selector example using
+    `spells.extract.npcs=<npc-run>/lanes/npcs.json`;
+  - `docs/operations.md`: the two-invocation sequential workflow and state
+    sensitivity;
+  - `docs/integrations/dnd-npc-artifacts.md`: production identity, final
+    normalization, warnings, manifest metadata, and reference-consumer
+    semantics;
+  - `docs/integrations/dnd-spell-artifacts.md`: optional NPC grounding and its
+    non-evidence rule;
+  - `docs/integrations/json-output.md`: link the NPC lane payload contract;
+  - `docs/internal/overview.md`, `docs/internal/modules.md`, and
+    `docs/internal/llm.md`: implemented package inventory and internal prompt,
+    preparation, fingerprint, and registration behavior.
+- At feature completion, mark the feature roadmap and this implementation plan
+  complete. Remove the now-implemented NPC item from `future.md`; keep combat
+  turns as the next future sequential artifact. Do not leave future behavior in
+  current-behavior documentation.
+
+### Tests
+
+- Test spell preparation with an absent registry, a valid normalized registry,
+  malformed JSON, unknown fields, invalid IDs, alias collisions, multiple
+  items, wrong media type, and the configured byte limit. Materialization-limit
+  failures must occur before pipeline execution and checkpoint construction.
+- Through the fake spell client, assert canonical NPC JSON input, input
+  metadata, manifest-safe count/digest, and absence of content leakage. Do not
+  assert prompt wording.
+- Test that identical semantic registries produce the same component
+  fingerprint, semantic changes change it, and returned fingerprints cannot be
+  mutated through caller-owned slices. Do not repeat generic checkpoint ordering
+  tests already owned by the framework.
+- Add one sequential integration case: run the assembled NPC pipeline with a
+  fake NPC response, serialize its normalized lane payload, materialize that
+  payload as the spell extractor's `npcs` reference, and run the spell pipeline
+  with a fake spell response. Verify reference provenance, prompt input, typed
+  output, and that no NPC source reference becomes spell evidence.
+- Validate all new documentation links and maintained example configurations.
+
+### Completion Check
+
+Run the final verification suite below.
+
+## Final Verification
+
+Run:
+
+```sh
+git diff --check
+go test ./...
+go vet ./...
+go build ./cmd/notarius
+go test -race ./internal/modules/dnd/... ./internal/framework/pipeline \
+  ./internal/cli ./internal/modules/integration
+```
+
+Review the final diff for:
+
+- accidental framework or provider-specific D&D behavior;
+- mutation of caller-owned artifacts, schemas, references, metadata, or
+  fingerprints;
+- prompt/schema/content leakage into manifests, errors, checkpoint identity,
+  or redacted summaries;
+- current documentation claiming behavior before its implementing stage exists;
+- redundant tests or prompt-prose change detectors; and
+- unrelated changes in the worktree.
+
+## Open Questions
+
+None. The artifact shape, identity algorithm, merge policy, validator placement,
+reference parsing, checkpoint semantics, and documentation ownership are fixed
+by this plan.