Add D&D NPC interaction artifact codec

docs/roadmap/implementation.md

@@ -0,0 +1,570 @@
+# D&D NPC Interactions Implementation Plan
+
+Status: Ready for implementation
+
+## Objective
+
+Implement the accepted [D&D NPC Interactions](dnd-npc-interactions.md)
+roadmap as a production D&D artifact lane. The finished lane must extract an
+ordered list of minimal, evidence-grounded NPC interaction occurrences from
+each accepted transcript chunk, ground every NPC against a required accepted
+NPC registry, and preserve the existing fixed pipeline and generated-reference
+architecture.
+
+This plan is the implementation authority for sequencing and file-level work.
+The feature roadmap remains authoritative for product intent, category
+semantics, occurrence boundaries, evidence policy, and non-goals. Follow
+[ADR-0009](../adr/0009-minimal-evidence-grounded-extraction-artifacts.md)
+throughout: do not add descriptive, analytical, relationship, or state fields.
+
+## Fixed Decisions
+
+The implementation must use these identities:
+
+| Concern | Identity |
+| --- | --- |
+| Extractor key | `dnd/npc-interactions` |
+| Normalizer key | `dnd/npc-interactions` |
+| Artifact kind | `dnd/npc-interaction-list` |
+| Durable schema ID | `notarius.dnd.npc_interactions` |
+| Durable schema name | `notarius_dnd_npc_interactions_v1` |
+| Durable schema version | `v1` |
+| Durable media type | `application/json` |
+| Private prompt ID | `dnd.npc_interactions` |
+| Private response schema ID | `notarius.dnd.npc_interactions.llm` |
+| Reference slot | `npcs` |
+
+Use a top-level `interactions` array. Each durable record has exactly `name`,
+`kind`, and `source_refs`. Do not add an interaction ID. The closed kind
+vocabulary is `mentioned`, `noncombat_presence`, `dialogue`, `combat_ally`,
+`combat_opponent`, and `other`.
+
+The `npcs` slot is required for both extraction and normalization. It accepts
+only the existing `dnd/npc-list` JSON artifact within the existing NPC registry
+size limit. A bound registry with an empty `npcs` array is valid and requires an
+empty interaction artifact. An absent binding, malformed registry, failed
+generated handoff, or unknown interaction name is not permission to perform
+ungrounded extraction.
+
+Use the existing `internal/modules/dnd/npcs/registry.Resolver` for construction-
+time external references and operation-time generated references. The
+extractor presents only `Registry.PromptInput()` to the model. Name lookup uses
+the registry's established comparison-key policy before normalization; the
+normalizer replaces a recognized variant with the exact canonical registry
+display name. Registry source references and campaign references never become
+interaction evidence.
+
+An occurrence is one NPC, one kind, and one coherent passage within one
+accepted chunk. The model must split a category or combat-alignment transition.
+No occurrence spans chunks. Merge preserves chunk order. Normalization:
+
+1. replaces recognized names with canonical registry display names;
+2. sorts and exact-deduplicates each record's source references;
+3. orders records by earliest valid source position, then canonical name,
+   interaction kind, and the canonical source-reference sequence;
+4. removes records only when canonical name, kind, and the complete canonical
+   source-reference sequence are all identical; and
+5. never semantically combines nearby, overlapping, or cross-chunk records.
+
+The private model response remains structural and contains name, kind, and
+integer start/end unit candidates. Mapping attaches the current source ID.
+Invalid strings, kinds, and ranges must survive mapping far enough for the
+deterministic validator that owns the rule to reject them.
+
+For the final record comparator, use source-document unit position for the
+earliest valid reference. Compare canonical names first by the existing NPC
+identity comparison key and then by exact display string; compare kinds by
+their string values; and compare canonical reference sequences
+lexicographically by `source_id`, start-unit document position, and end-unit
+document position. Records with valid evidence precede records without it;
+normalized output validation rejects the latter before durable encoding.
+
+No framework change, new identity subsystem, DAG behavior, LLM-assisted
+deduplication, or scene dependency is part of this plan.
+
+## Testing And Documentation Rules
+
+Apply [Testing Policy](../policy/testing.md) at every stage:
+
+- Protect artifact round trips, required fields, enum values, source
+  provenance, registry grounding, deterministic ordering, exact
+  deduplication, reference immutability, and generated handoff behavior.
+- Use the real codec, registry, resolver, normalizer, and pipeline machinery
+  when they are fast and deterministic. Stub only the structured LLM boundary.
+- Keep provider calls offline and deterministic.
+- Test each policy at its narrowest stable owner. Do not repeat every codec,
+  validator, and normalizer case in an end-to-end test.
+- Do not add a prompt-cache “change detector” based on exact message count,
+  shared-prefix length, token count, or prompt hash. Reuse the shared prompt
+  assets, test rendered behavior and manifest composition, and document the
+  ordering policy instead.
+- Do not use broad golden snapshots for prompts or diagnostics. A maintained
+  durable JSON fixture is appropriate because the complete serialized artifact
+  is an intentional integration contract.
+
+Until the final stage, keep unimplemented behavior only in roadmap
+documentation. Update current-behavior documentation and examples in the same
+stage that makes the production lane selectable.
+
+## Stage 1 — Add The Typed Artifact And Durable Codec
+
+### Goal
+
+Establish the domain type and strict serialized boundary without changing the
+production module catalog.
+
+### Changes
+
+1. Extend `internal/modules/dnd/types.go` with:
+   - `NPCInteractionListKind`;
+   - a string-backed `NPCInteractionKind`;
+   - exported constants for all six accepted values;
+   - `NPCInteractionList` with `Interactions []NPCInteraction`; and
+   - `NPCInteraction` with `Name string`, `Kind NPCInteractionKind`, and
+     `SourceRefs []source.SourceRef`.
+2. Add `internal/modules/dnd/codec/npcinteractions` following the existing NPC
+   and combat-turn codec boundary:
+   - expose the fixed schema constants and `MediaType`;
+   - implement the typed artifact codec for `dnd.NPCInteractionList`;
+   - keep `EncodeCandidate` and `DecodeCandidate` strict about one JSON value
+     and unknown fields while preserving typed semantic candidates;
+   - make approved `Encode` and `Decode` enforce non-empty names, the closed
+     kind enum, at least one structurally valid source reference, and the
+     durable schema shape; and
+   - report only `interaction_count` in codec metadata.
+3. Add
+   `internal/modules/dnd/codec/npcinteractions/assets/schemas/dnd_npc_interactions.v1.json`.
+   It must require the top-level array and all three record fields, reject
+   unknown fields at every object level, encode the six-value enum, require at
+   least one source reference, and use the established durable source-reference
+   shape.
+4. Add a small maintained fixture under
+   `internal/modules/dnd/codec/npcinteractions/testdata/` containing at least
+   two ordered records with distinct kinds and source ranges.
+
+### Tests
+
+At the codec package boundary, cover:
+
+- fixture decode/encode round trip and canonical compact JSON;
+- exact kind, schema identity, version, media type, and Go type registration;
+- empty-list support and nil-versus-present-empty behavior where the existing
+  codecs distinguish it;
+- strict rejection of malformed JSON, trailing values, and unknown fields;
+- approved-boundary rejection of each meaningful required-field, enum, and
+  source-reference violation;
+- candidate-boundary preservation of semantic values for later validators; and
+- defensive copies for schema bytes and metadata.
+
+Do not duplicate source-document existence checks in the codec; those belong to
+the source-reference validator.
+
+### Completion Criteria
+
+- The new types and codec compile and pass focused tests.
+- The codec can be registered into an isolated artifact codec registry with
+  exact type `dnd.NPCInteractionList`.
+- No production registrar or configuration catalog exposes the new kind yet.
+
+## Stage 2 — Implement The Chunk-Scoped Extractor And Prompt
+
+### Goal
+
+Add an independently testable LLM-backed extractor that requires NPC grounding,
+maps only minimal private output, and follows the established D&D prompt-cache
+layout.
+
+### Changes
+
+1. Add `internal/modules/dnd/extract/npcinteractions` using the current D&D
+   extractor organization:
+   - `assets.go`;
+   - `canonicalize.go`;
+   - `extractor.go`;
+   - `model.go`;
+   - `schema.go`;
+   - `scriptorium_assets.go`;
+   - corresponding focused tests; and
+   - package-local embedded prompt and schema assets.
+2. Give the extractor strict empty options, required capabilities `chunks` and
+   `source.transcript`, provided capability `dnd.npc_interactions`, and artifact
+   kind `dnd.NPCInteractionListKind`.
+3. Build its reference slots from `shared.ReferenceSlots(...)`, then append the
+   `npcs` slot with:
+   - `Required: true`;
+   - accepted media type `application/json`;
+   - accepted artifact kind `dnd.NPCListKind`;
+   - the existing registry maximum byte count; and
+   - a description stating that the accepted registry is required identity
+     grounding, not evidence.
+   Return defensive, consistently sorted slot declarations from both
+   `ModuleSpec()` and the constructed extractor.
+4. Construct an `npcregistry.Resolver` from the preparation-time reference set.
+   Resolve operation references inside `Extract`, require `Registry.Bound()`,
+   and fail before the LLM call if the required registry is absent or invalid.
+   Preserve the resolver's content-safe error behavior.
+5. Use `shared.ChunkPromptMaterial` and `shared.PromptInputs`. Replace the
+   ordinary `npcs` input with the resolved names-only `Registry.PromptInput()`.
+   Do not place full registry bytes, NPC IDs, registry evidence, or provenance
+   in prompt inputs.
+6. Add these assets:
+   - `assets/prompts/dnd.npc_interactions.yaml`;
+   - `assets/prompts/task.md`;
+   - `assets/prompts/instructions.md`; and
+   - `assets/schemas/dnd_npc_interactions_llm.v1.json`.
+7. Use the existing shared prompt assets in this exact semantic order:
+   common system; common extraction evidence; common identity; common campaign
+   references; common NPC grounding; lane task; lane instructions; variable
+   transcript. Preserve the established cache-control placement used by the
+   spell and combat-turn manifests.
+8. Revise the shared `common-dnd-npcs.md` wording once so it truthfully applies
+   to all three consumers: a normalized registry is always presented to the
+   prompt and may be empty. Keep its identity, context-only rule, and
+   prohibition on treating registry provenance as event evidence. Do not fork a
+   nearly identical interaction-specific grounding asset.
+9. The lane prompt must state the category definitions, precedence, occurrence
+   splitting rules, registry-only name restriction, empty-output behavior, and
+   exclusions from the feature roadmap. Keep the response schema structural:
+   it requires the envelope and fields but leaves enum membership, non-empty
+   values, and source semantics to deterministic validators.
+10. Map private records to `dnd.NPCInteractionList`, attach the current source
+    ID, sort and exact-deduplicate ranges within each candidate, and stable-sort
+    candidates with valid evidence by earliest source-document position.
+    Preserve invalid candidate fields rather than repairing or dropping them.
+11. Expose prompt, response-schema, and mapping-policy identities through
+    manifest metadata and checkpoint fingerprints, following the current D&D
+    extractors. Include the NPC names-only projection digest in local checkpoint
+    identity. For a preparation-time external registry, expose only bounded
+    digest/count metadata; do not place names or source content in metadata.
+    Generated-reference provenance remains framework-owned.
+
+### Tests
+
+At stable package boundaries, cover:
+
+- nil/cancelled/invalid extraction requests and provider failures;
+- required-slot declaration and rejection of an unbound or malformed registry
+  before any LLM call;
+- operation-time generated registry resolution by an extractor prepared
+  without generated bytes;
+- exact names-only registry prompt input and separation from transcript
+  evidence;
+- empty registry plus empty response;
+- mapping of every category, current-source attachment, evidence
+  canonicalization, and source-position ordering;
+- preservation of invalid names, kinds, and ranges for validators;
+- strict private JSON shape and unknown-field rejection;
+- prompt rendering with required inputs, category policy, shared assets, and
+  transcript-last ordering;
+- prompt/schema registration and content-safe metadata/fingerprints; and
+- defensive module specifications and strict rejection of unknown options.
+
+Run the existing spell and combat-turn prompt/asset tests after changing the
+shared NPC prompt fragment. Do not assert an exact shared-prefix length.
+
+### Completion Criteria
+
+- The extractor is constructible and testable through the typed extractor
+  contract.
+- A valid request makes one scheduled structured-completion call and returns a
+  typed candidate with current-source provenance.
+- Missing required NPC grounding cannot reach the model.
+- The extractor remains unregistered in the production D&D family until later
+  stages provide the rest of the lane.
+
+## Stage 3 — Add Deterministic Artifact Validators
+
+### Goal
+
+Give each semantic invariant one clear validation owner and make invalid model
+output a rejection rather than a normalization repair or framework error.
+
+### Changes
+
+Add these typed validator packages, each with strict empty options, bounded
+diagnostics, typed registration, and the fixed artifact kind:
+
+1. `internal/modules/dnd/validate/npcinteractions/shape`
+   (`extract/dnd/npc-interactions/shape`):
+   - require a non-empty trimmed name without mutating it;
+   - require one of the six kinds; and
+   - require at least one source-reference candidate with structurally required
+     values.
+2. `internal/modules/dnd/validate/npcinteractions/registry`
+   (`extract/dnd/npc-interactions/registry`):
+   - use `npcregistry.Resolver` at construction and operation time;
+   - require a bound registry;
+   - approve a name when `Registry.Lookup` recognizes it under the established
+     comparison policy;
+   - reject unknown names in stable record order; and
+   - never use registry evidence as transcript evidence.
+3. `internal/modules/dnd/validate/npcinteractions/source_refs`
+   (`extract/dnd/npc-interactions/source_refs`):
+   - require the current source identity;
+   - require referenced units to exist; and
+   - require valid inclusive start/end order through the current document.
+   Follow the established D&D citation helpers and rejection aggregation
+   limits rather than introducing a generic framework dependency on the
+   interaction type.
+4. `internal/modules/dnd/validate/npcinteractions/source_relatedness`
+   (`extract/dnd/npc-interactions/source_relatedness`):
+   - inspect only current transcript text covered by the cited ranges;
+   - emit at most one bounded warning per record when the NPC name cannot be
+     related to that evidence;
+   - do not claim to validate category agreement; and
+   - ignore campaign and registry content for relatedness.
+5. `internal/modules/dnd/validate/npcinteractions/invariants`
+   (`normalize/dnd/npc-interactions/invariants`):
+   - require the exact canonical display name returned by registry lookup;
+   - require canonical source-reference order with no duplicate ranges;
+   - require the complete list order defined in Fixed Decisions; and
+   - reject exact duplicate normalized records.
+
+The registry and invariants validators must support operation-time generated
+references. Construction-time metadata and fingerprints follow the resolver
+pattern used by registry-aware modules and must not include NPC names or
+content.
+
+### Tests
+
+Give each policy one primary test owner:
+
+- shape tests own empty names, enum membership, and missing evidence;
+- registry tests own required binding, comparison-key recognition, unknown
+  names, operation-time generated overrides, empty registries, and content-safe
+  failures;
+- source-reference tests own wrong source IDs, missing units, reversed ranges,
+  and valid multi-range records;
+- relatedness tests own current-transcript-only warnings and bounded
+  diagnostics; and
+- invariant tests own exact canonical names, final ordering, canonical evidence
+  sequences, and duplicate rejection.
+
+Also verify that validators do not mutate artifacts, source documents, or
+registries and that their specs register only for
+`dnd.NPCInteractionListKind`.
+
+### Completion Criteria
+
+- Valid typed candidates are approved and malformed or ungrounded candidates
+  are rejected at the intended boundary.
+- No validator silently canonicalizes, drops, or merges records.
+- References can explain identity but cannot satisfy source evidence checks.
+
+## Stage 4 — Add Deterministic Merge And Normalization
+
+### Goal
+
+Preserve chunk-scoped occurrences through merge, then produce the canonical
+ordered artifact without semantic inference.
+
+### Changes
+
+1. Add `internal/modules/dnd/normalize/npcinteractions` with strict empty
+   options and the fixed module key and artifact kind.
+2. Declare the same required `npcs` reference slot as the extractor. Construct
+   and resolve `npcregistry.Resolver` at the same preparation/operation
+   boundaries, and require a bound registry.
+3. Implement the normalization algorithm in Fixed Decisions:
+   - clone all nested values;
+   - normalize display whitespace only as needed for lookup;
+   - replace every recognized name with the registry's exact canonical display
+     name;
+   - canonicalize each reference list without merging overlapping ranges;
+   - compute order from source-document positions rather than assuming numeric
+     unit IDs are contiguous;
+   - apply every specified tie-breaker; and
+   - collapse only exact records after canonicalization.
+4. Preserve nil versus present-empty list behavior consistently with the other
+   D&D normalizers.
+5. Emit bounded warnings for visible canonical-name changes, reference
+   canonicalization, record reordering, and exact duplicate removal. Use stable
+   reason codes and the shared D&D diagnostics helpers. Never include reference
+   content in a warning.
+6. Add an append-only merger function for `dnd.NPCInteractionList` beside the
+   other D&D typed append functions in
+   `internal/modules/dnd/register/merge.go`. It must preserve accepted
+   chunk/lane order, preserve present-empty semantics, and deep-clone source
+   reference slices so outputs do not alias inputs. Do not register it until
+   Stage 5.
+7. Add normalization-policy and NPC projection identities to checkpoint
+   fingerprints. Manifest metadata may include the policy identity and bounded
+   external-registry digest/count, but not names or source content.
+
+### Tests
+
+Cover:
+
+- canonical registry name replacement, including Unicode/case/spacing lookup;
+- required and operation-time generated registry behavior;
+- source-reference ordering and exact range deduplication;
+- ordering by real document position with all deterministic tie-breakers;
+- separation of category transitions, alignment transitions, nearby records,
+  overlapping-but-nonidentical evidence, and nonidentical records from
+  different chunks;
+- collapse of exact duplicates only;
+- stable, bounded warnings;
+- nil/present-empty behavior;
+- input ownership and nested-slice cloning; and
+- append-order merger behavior independently of normalization.
+
+Use table-driven pure normalization cases for the dense ordering and duplicate
+rules, and package-level normalizer tests for resolver and warning behavior.
+
+### Completion Criteria
+
+- Normalization is deterministic and idempotent.
+- Running normalization twice does not change the value or emit new
+  transformation warnings on the second pass.
+- Semantically distinct occurrences remain distinct.
+- The module and merger compile but are not yet selectable through production
+  composition.
+
+## Stage 5 — Compose The Complete Production Lane
+
+### Goal
+
+Register the complete typed lane atomically so configuration cannot select a
+partial implementation.
+
+### Changes
+
+Update the D&D registrar:
+
+1. In `internal/modules/dnd/register/modules.go`, register:
+   - the NPC-interaction codec;
+   - extractor and prompt assets;
+   - the append-order merger variant for
+     `dnd.NPCInteractionListKind`;
+   - the `dnd/npc-interactions` normalizer; and
+   - the generic no-op normalizer variant for the new exact Go type.
+2. In `internal/modules/dnd/register/validators.go`, register all five domain
+   validators plus generic always-accept and always-reject typed variants for
+   the new artifact type.
+3. In `internal/modules/dnd/register/chains.go`, register:
+   - extraction chain:
+     `generic/valid_json`,
+     `extract/dnd/npc-interactions/shape`,
+     `extract/dnd/npc-interactions/registry`,
+     `extract/dnd/npc-interactions/source_refs`,
+     `generic/valid_json_schema`,
+     `extract/dnd/npc-interactions/source_relatedness`;
+   - normalization chain:
+     `generic/valid_json`,
+     `extract/dnd/npc-interactions/shape`,
+     `extract/dnd/npc-interactions/registry`,
+     `normalize/dnd/npc-interactions/invariants`,
+     `extract/dnd/npc-interactions/source_refs`,
+     `generic/valid_json_schema`,
+     `extract/dnd/npc-interactions/source_relatedness`.
+4. Do not add a default merge validator chain. The append merger performs no
+   semantic mutation, and extraction plus normalized boundaries own the
+   consequential policies.
+5. Extend registrar contract tests to verify exact typed coverage, keys,
+   artifact kinds, prompt/schema assets, reference-slot agreement, and default
+   validator order.
+
+### Tests
+
+- Extend the existing D&D family registration test rather than creating
+  parallel catalog snapshots.
+- Verify that the extractor and normalizer both expose required compatible
+  `npcs` slots and that generated `dnd/npc-list` bindings resolve only from an
+  earlier step.
+- Verify production resolution rejects a missing required binding, a later or
+  same-step producer, a wrong artifact kind, and incompatible media/schema
+  metadata through existing resolver behavior.
+- Avoid retesting the framework's general ordered-step failure matrix; add only
+  interaction-specific composition cases not already covered generically.
+
+### Completion Criteria
+
+- Production registration exposes one complete type-consistent lane.
+- A valid two-step NPC-to-interactions pipeline resolves and prepares.
+- Invalid or missing NPC dependencies fail before source parsing or LLM work
+  whenever statically discoverable.
+
+## Stage 6 — Prove The Workflow And Publish Current Contracts
+
+### Goal
+
+Exercise the assembled generated-reference workflow and move all implemented
+behavior into its canonical current documentation.
+
+### Changes
+
+1. Add one representative integration test under
+   `internal/modules/integration` with focused testdata:
+   - step 1 extracts and normalizes NPCs;
+   - step 2 consumes the generated NPC artifact in an interaction lane;
+   - the fake structured LLM returns multiple interaction kinds and at least
+     one name variant;
+   - the assertion proves the names-only handoff, current-transcript evidence,
+     canonical name, stable chronology, and final durable artifact.
+2. Include one failure assertion showing that a rejected or absent normalized
+   NPC producer prevents interaction extraction. Reuse the existing runner
+   dependency behavior; do not duplicate every framework failure case.
+3. Add `examples/dnd-npc-interactions.config.yml` as a complete copyable
+   two-step profile. The first step produces `npcs`; the second step binds that
+   exact step/lane artifact to its `npcs` reference and selects
+   `dnd/npc-interactions` for extract and normalize.
+4. Add the example to the existing CLI production configuration contract test
+   so it is parsed and resolved offline.
+5. Add the durable external contract at
+   `docs/integrations/dnd-npc-interaction-artifacts.md`. It owns schema
+   identity, JSON shape, categories, evidence semantics, ordering,
+   normalization, validator chains, generated-reference behavior, and manifest
+   metadata.
+6. Update current canonical documentation:
+   - `docs/config.md` for module/validator catalogs, default chains, required
+     reference slot, and the maintained example;
+   - `docs/internal/modules.md` for implementation ownership, prompt inputs,
+     registry resolution, validation, merge, and normalization;
+   - `docs/internal/overview.md` for the implemented component inventory;
+   - `docs/internal/llm.md` only as needed to include the new lane in the
+     existing D&D prompt-order/cache policy without duplicating the manifest;
+   - `docs/integrations/dnd-npc-artifacts.md` to identify NPC interactions as a
+     consumer while retaining the rule that registry evidence is not event
+     evidence; and
+   - any directly affected configuration or integration links.
+7. Remove the NPC-interactions entry from `docs/roadmap/future.md` once the
+   production behavior and current documentation are complete. Change
+   `docs/roadmap/dnd-npc-interactions.md` to `Status: Implemented` pending its
+   later retirement; do not leave it as the canonical current contract.
+8. Create a small human-review worksheet or fixture set only if the repository
+   already has an appropriate non-test evaluation home. Otherwise record the
+   manual evaluation results in the implementation handoff rather than
+   inventing a new framework. Exercise all six categories, transitions,
+   repeated occurrences, mentions followed by presence, combat alignment
+   changes, cross-chunk repetition, and empty output on at least one intended
+   smaller model. Do not make model agreement a deterministic CI gate.
+
+### Validation
+
+Run focused tests while implementing, then run:
+
+```sh
+go test ./...
+go vet ./...
+go build ./cmd/notarius
+```
+
+Also run `git diff --check`, inspect the complete documentation diff for
+current-versus-future claims, and verify every new relative documentation link.
+
+### Completion Criteria
+
+- All repository checks pass offline.
+- The maintained example resolves with an explicit earlier NPC producer.
+- The integration test proves the accepted generated artifact, not registry
+  source references, grounds the later model request.
+- Durable output contains only `name`, `kind`, and current-transcript
+  `source_refs` per occurrence.
+- Current documentation owns implemented contracts, while the feature roadmap
+  is marked implemented and `future.md` no longer advertises the work as
+  pending.
+
+## Open Questions
+
+None. The feature roadmap and fixed decisions above define the product and
+architectural choices required for implementation.