Plan LLM-assisted NPC normalization

docs/roadmap/dnd-npc-semantic-normalization.md

@@ -0,0 +1,328 @@
+# D&D NPC Semantic Normalization
+
+## Status
+
+Accepted scope; not implemented.
+
+## Purpose
+
+Reconcile NPC records that extraction produced under different display names
+when the complete transcript establishes that they represent the same
+individual. This work addresses identities split across scene or chunk
+boundaries while preserving the existing rule that extraction records narrow
+source evidence and normalization owns document-wide reconciliation.
+
+The model should make only the semantic identity judgment. Deterministic code
+must continue to own identity derivation, proposal validation, artifact
+mutation, evidence preservation, ordering, diagnostics, and final validation.
+
+## Target Behavior
+
+The target `dnd/npcs` normalization contract combines:
+
+- the current deterministic display-name normalization, stable-ID derivation,
+  source-reference canonicalization, and equal-comparison-key consolidation;
+- one LLM determination of whether remaining, distinctly named NPC records
+  represent the same individual and which existing display name is canonical;
+  and
+- deterministic proposal validation and application followed by the configured
+  normalize validator chain.
+
+Each configured normalization attempt should make at most one semantic pass
+over the merged document-level NPC list, not one pass per extraction chunk or
+candidate pair. The pass should be skipped when fewer than two distinct
+candidate identities remain after deterministic preprocessing.
+
+False consolidation is more damaging than a missed consolidation. Prompt
+policy should therefore require affirmative contextual evidence that names
+identify the same individual and should prefer no group when identity remains
+ambiguous.
+
+Canonical selection should favor the most complete stable proper name supported
+by the transcript. A complete proper name is preferable to an abbreviation,
+while an unadorned proper name is preferable to the same name plus a contextual
+class, role, title, or relationship descriptor unless the transcript establishes
+that descriptor as part of the character's name. The model must still select
+one supplied display name rather than synthesize a better one.
+
+## Model Proposal Contract
+
+The private structured response should contain only proposed duplicate groups:
+
+```json
+{
+  "duplicate_groups": [
+    {
+      "members": [
+        "Billy",
+        "Billy the druid"
+      ],
+      "canonical_name": "Billy"
+    }
+  ]
+}
+```
+
+`members` identifies candidates by their supplied display names.
+`canonical_name` selects one existing member as the retained display name. An
+empty `duplicate_groups` array is a valid determination that no sufficiently
+supported duplicates exist.
+
+The model must not receive or return deterministic `npc:sha256:` IDs. Those IDs
+are long, non-semantic implementation identities and remain exclusively owned
+by deterministic code. The model must not invent a replacement name, rewrite
+an NPC record, propose new source references, or return a complete replacement
+artifact.
+
+Before the call, deterministic preprocessing ensures that each supplied display
+name has a unique NPC identity comparison key. Returned names may resolve using
+the existing comparison-key equivalences for case, whitespace, Unicode
+compatibility, and supported apostrophes. Resolution must not use fuzzy,
+substring, edit-distance, embedding, or other approximate matching.
+
+An applicable group must:
+
+- contain at least two distinct, known members;
+- resolve every member uniquely against the supplied candidate set;
+- select a known `canonical_name` that belongs to the group;
+- contain no repeated member; and
+- share no resolved member with any other proposed group.
+
+Unknown or ambiguous names, singleton groups, invalid canonical selections, and
+other unsafe groups must not mutate the artifact. Groups are otherwise
+independent: a locally valid group may be applied when none of its resolved
+members appears in any other proposed group.
+
+Overlap is a proposal conflict even when one participating group is already
+locally invalid. Deterministic code must discard every group in the connected
+conflict set rather than selecting a winner by response order. Locally valid,
+non-conflicting groups remain safe to apply. This permits useful partial
+reconciliation without allowing an unsafe group to influence an NPC identity
+that another group would mutate.
+
+## Retry And Safe Fallback
+
+An invalid or partially unsafe private proposal is a retryable normalization
+attempt. This includes:
+
+- structurally invalid model output classified by the LLM boundary as an
+  invalid structured completion; and
+- a structurally valid response containing any locally invalid or conflicting
+  group.
+
+The framework owns retry counting and attempt diagnostics. The normalizer
+returns a safe candidate together with a bounded, content-safe retry
+diagnostic. For structurally invalid output, that candidate is the
+deterministic pre-LLM result. For a decoded proposal with unsafe groups, it
+also includes every independently valid, non-conflicting group from that
+attempt. When normalize retries remain, the framework invokes the normalizer
+again from the same merged input; safe groups are not accumulated across
+attempts. A later completely safe proposal is applied normally.
+
+When the configured attempt budget is exhausted, the framework validates and
+accepts the final attempt's safe fallback instead of failing or rejecting the
+NPC lane, provided that fallback passes the configured normalize validators.
+It promotes one durable warning stating that one or more proposed groups were
+omitted. The fallback may therefore contain a safe partial reconciliation, or
+only the deterministic pre-LLM result when no group could be applied. Earlier
+retry diagnostics, candidate values, and warnings remain attempt-local in
+debug artifacts.
+
+The normal module-binding default remains `retries: 0`, meaning one total
+normalization attempt and immediate fallback after its invalid proposal.
+Retries occur only when the user configures a positive normalize retry count.
+For example, `retries: 2` permits the initial proposal plus two additional
+attempts before fallback.
+
+Transport, authentication, prompt-preparation, cancellation, input-encoding,
+and other operational errors are not safe-proposal failures. They retain the
+existing module/framework error behavior rather than being converted into an
+accepted fallback.
+
+## Transcript Context Policy
+
+The model should receive every candidate's display name and source references,
+together with transcript windows derived from those references. It should not
+receive the complete transcript by default.
+
+Each window must contain:
+
+- the complete inclusive source range cited by the NPC record;
+- up to two source units immediately before the cited range; and
+- up to two source units immediately after the cited range.
+
+The surrounding-unit count is defined once as a named module policy constant
+with value `2`; it is not a user-accessible configuration field. Window
+construction consumes that value through one clear boundary so a later
+configuration option can replace the fixed value without changing prompt or
+reconciliation contracts. The context-radius policy participates in module
+metadata and checkpoint identity so changing it invalidates incompatible
+normalize checkpoints.
+
+The value was selected by reviewing the July 19 evaluation transcript. Its NPC
+citations are generally self-contained; where additional context is useful,
+two surrounding units capture the relevant question-and-answer exchange,
+speaker transition, or short anaphora chain. A third unit frequently begins a
+separate joke or table exchange and adds distraction without improving the
+identity evidence.
+
+Window expansion must use source-document positions rather than arithmetic on
+unit IDs. It must clamp at document boundaries, preserve document order, and
+coalesce overlapping or adjacent expanded windows without duplicating units.
+The prepared model input must distinguish originally cited units from
+surrounding context.
+
+Only records with a non-empty NPC identity comparison key and a non-empty
+source-reference collection that is wholly valid against the current source
+document are eligible for semantic reconciliation. Ineligible records remain
+unchanged so the configured deterministic validators retain ownership of their
+rejection. They are not supplied to the model and cannot participate in a
+proposed group.
+
+Surrounding units inform the semantic decision but do not automatically become
+durable NPC evidence. An applied group unions only the source references
+already present on its member records. The LLM cannot add references from the
+context window.
+
+Full-transcript mode and a configurable context radius may be evaluated later.
+They are not part of this scope. Provider prompt-cache behavior should be
+measured rather than assumed before expanding context solely for cache
+economics.
+
+## Deterministic Application
+
+For each approved group, deterministic code should:
+
+- retain the model-selected existing display name;
+- union source references from every group member;
+- canonicalize and exact-deduplicate those references in source-document order;
+- derive the resulting stable NPC ID from the retained display name under the
+  existing NPC identity policy; and
+- emit one bounded duplicate-collapse warning describing the applied group.
+
+The consolidated record should occupy the earliest original member position so
+model response ordering cannot reorder the artifact. Unrelated records must
+remain present and retain their relative order. Every input record must be
+represented by exactly one output record, either unchanged or through one
+approved consolidation.
+
+The existing NPC shape, identity, source-reference, schema, and relatedness
+validators remain the final artifact boundary. No LLM-backed validator is
+needed: model judgment occurs in the normalizer, while deterministic validators
+continue to enforce the durable artifact contract.
+
+## Prompt, Provenance, And Diagnostics
+
+NPC semantic-normalization instructions and the private response schema should
+be module-owned prompt assets. Stable D&D-wide identity or transcript guidance
+may be reused through the existing shared prompt-asset mechanism where its
+meaning is genuinely common.
+
+The normalizer should use the configured normalize-stage LLM profile and the
+application-wide scheduled LLM client. Invalid structured output and unsafe
+semantic proposals use the framework-owned retryable-fallback contract;
+operational failures retain existing error semantics.
+
+Manifest metadata and component checkpoint fingerprints should identify the
+semantic-normalization policy, prompt identity, private response-schema
+identity, deterministic NPC identity policy, and evidence-context policy.
+They must not contain transcript text, NPC names, source paths, raw model
+responses, or other source content.
+
+Warnings and preparation or runtime errors must follow the established bounded
+diagnostic and content-safety policies. Debug artifacts may retain the normal
+attempt-local model request, response, proposal decisions, and warnings under
+the existing debug sensitivity contract.
+
+## Configuration And Documentation
+
+The maintained complete D&D configuration should demonstrate a normalize-stage
+LLM profile and an explicit positive retry count for the NPC lane. The
+application-wide default remains zero additional retries. No new configuration
+field is introduced for the context radius in this scope.
+
+Canonical documentation ownership is:
+
+- Configuration owns the LLM-backed `dnd/npcs` selection and its use of the
+  existing normalize binding's profile and retry fields.
+- Operations owns the document-level semantic reconciliation call and its
+  checkpoint behavior.
+- Pipeline internals own the provider-neutral invalid-structured-output
+  classification and normalize retryable-fallback contract.
+- The NPC integration contract owns externally observable consolidation,
+  evidence, ordering, and warning behavior.
+- Internal LLM and module documentation own the proposal boundary,
+  context-window construction, deterministic application, metadata, and
+  fingerprints.
+- The broader generic LLM-assisted deduplication item in
+  [future.md](future.md) remains future work until another artifact demonstrates
+  that extracting a shared generic facility is worthwhile.
+
+## Quality Expectations
+
+Tests should protect the semantic and safety boundaries through deterministic
+LLM fakes rather than live-provider calls. Coverage should demonstrate:
+
+- distinct display variants can be consolidated when the model proposes a
+  valid group;
+- model-facing requests and responses use display names rather than stable
+  hash IDs;
+- the selected existing canonical name controls ID derivation while all member
+  evidence is preserved;
+- window construction uses source-document position, handles document edges,
+  coalesces overlap, and distinguishes cited evidence from context;
+- no surrounding context is promoted into durable source references;
+- empty proposals and fewer-than-two-candidate inputs preserve deterministic
+  normalization behavior;
+- unknown, ambiguous, repeated, overlapping, and otherwise malformed semantic
+  groups cannot corrupt or reorder the artifact;
+- locally valid groups are applied independently, while every group that
+  participates in a resolved-member conflict is discarded;
+- an invalid proposal consumes only configured retry budget, a later valid
+  proposal can succeed, and exhaustion accepts the final attempt's safe
+  fallback with one durable warning;
+- the default zero-retry binding makes exactly one proposal attempt before
+  fallback;
+- warnings and failures remain bounded and do not expose transcript content;
+- prompt, schema, policy, or context-policy changes invalidate relevant
+  checkpoint reuse; and
+- an assembled ordered D&D pipeline supplies the reconciled NPC registry to
+  downstream consumers.
+
+Prompt tests should assert prepared message structure, supplied materials, and
+cache-boundary behavior. They must not act as change detectors for particular
+words or phrases in natural-language prompt text.
+
+Human review of representative transcripts should compare missed and false
+consolidations, latency, input-token cost, and provider cache use. Probabilistic
+model quality is an evaluation activity, not a deterministic CI assertion.
+
+## Non-Goals
+
+This scope does not:
+
+- change the durable NPC artifact schema;
+- add a durable alias collection or identity history;
+- expose the context radius or full-transcript selection as configuration;
+- ask an LLM to read, reproduce, or derive stable NPC IDs;
+- permit model-authored source references or arbitrary replacement records;
+- add fuzzy deterministic name matching;
+- add an LLM-backed normalize validator;
+- change the global default retry count;
+- reconcile NPCs concurrently across lanes or runs;
+- add canonical NPC IDs to downstream artifact schemas;
+- implement a general DAG or implicit cross-lane dependency; or
+- implement the generic cross-artifact deduplication normalizer described in
+  `future.md`.
+
+## Completion Criteria
+
+The scope is complete when `dnd/npcs` can use one document-level LLM proposal
+to reconcile differently named records conservatively, deterministic code
+validates and independently applies only safe non-conflicting name-based
+groups, the resulting NPC preserves all member evidence under a newly derived
+stable ID, the two-unit context policy is centralized and fingerprinted,
+invalid or partially unsafe proposals use configured framework retries and
+then an accepted safe fallback, downstream ordered steps receive the
+reconciled registry, and the canonical current-behavior documentation reflects
+the implemented contract.