notarius/docs/roadmap/semantic-reconciliation.md

# Semantic Reconciliation Roadmap

## Purpose

This roadmap defines a reusable, LLM-assisted semantic-reconciliation facility
for source-backed entity registries. The facility will centralize the common
candidate preparation, prompt execution, proposal validation, safety, retry,
and consolidation mechanics currently implemented by the D&D NPC, item, and
location registry normalizers while preserving typed, domain-owned output.

The model will remain a constrained proposal source. Notarius will retain
authority over candidate identity, proposal validation, deterministic mutation,
provenance, durable identifiers, warnings, and final artifact construction.

## Motivation

NPC, item, and location registry normalization now demonstrate the same useful
pattern: deterministic preprocessing produces candidate records, an LLM judges
whether some candidates describe the same underlying entity, and deterministic
code applies only safe proposed groups. The current D&D-shared implementation
proves the approach, but it still duplicates orchestration and application
logic across normalizers and requires the model to reproduce complete
contextual selectors containing names and evidence ranges.

The target design should provide one efficient and thoroughly tested
reconciliation core that can support additional artifact families without
moving domain semantics into generic code. It should also simplify the model's
task by replacing selector reproduction with small request-local integer
handles.

## Goals

- Establish one domain-neutral semantic-reconciliation core for source-backed
  entity candidates.
- Move all demonstrated common mechanics into that core, including structured
  LLM execution and deterministic proposal assessment.
- Preserve exact typed artifact ownership from merged input through normalized
  output.
- Give the model semantic evidence while asking it to return only small,
  request-local candidate identifiers.
- Provide a conservative generic semantic prompt policy that a typed artifact
  family may use by default.
- Allow an artifact family to supply narrower domain semantic policy without
  replacing the mandatory shared protocol and safety instructions.
- Have the NPC, item, and location registry normalizers use the shared core
  without changing their durable artifact contracts, configured module keys,
  domain identity rules, or fallback guarantees.
- Make addition of another eligible registry normalizer primarily an adapter
  and policy exercise rather than a copy of reconciliation machinery.

## Terminology And Ownership

A **stage module** remains one configured implementation of one pipeline stage.
An extractor and a normalizer are separate stage modules even when they
collaborate on the same artifact kind.

An **artifact family** is the cohesive domain feature that owns an artifact's
types, codec, extractor, merge choice, normalizer, validators, prompt policy,
schemas, identity helpers, and reference projections. For example, D&D spells
and the D&D NPC registry are artifact families whose implementations span
multiple explicit pipeline stages.

This terminology clarifies existing ownership without changing Notarius's
fixed pipeline or combining stages. It is consistent with the accepted
domain-first organization and typed generic-strategy boundary in
[ADR-0004](../adr/0004-package-modules-by-domain.md). The D&D registrar remains
responsible for composing D&D-owned stage modules and their policies; the
generic core remains unaware of D&D types or semantics.

## Target Architecture

### Shared Core

A domain-neutral framework package should own semantic reconciliation. It may
depend on generic source and structured-completion contracts and may consume
its scoped assets under `assets/generic/`, but it must not import a production
domain or encode D&D identity rules.

The shared core owns:

- validation and defensive copying of its inputs;
- candidate eligibility and stable input ordering;
- assignment of model-visible request-local candidate IDs;
- construction of bounded candidate and source-context materials;
- invocation of the configured structured LLM prompt;
- the private duplicate-group response schema;
- exact resolution of response IDs to the candidates visible in that request;
- rejection of malformed, unknown, repeated, ambiguous, or overlapping
  proposals;
- canonical ordering of accepted groups and group members;
- construction of an immutable reconciliation plan;
- shared retry, fallback, cancellation, and invalid-output classification;
- generic consolidation and provenance mechanics that are demonstrably common
  across typed consumers; and
- metadata and checkpoint fingerprints for shared prompts, schemas, and policy
  versions.

The core should expose stable behavior rather than a collection of unrelated
helpers. Its API should make the safe path direct: a caller supplies typed
candidates and a narrow domain policy, and receives either a deterministic
typed result or a validated reconciliation plan that can only be applied
through the typed policy boundary.

### Typed Artifact-Family Adapter

Each consuming artifact family owns a typed adapter or policy that supplies the
irreducibly domain-specific behavior:

- projection of merged typed records into eligible reconciliation candidates;
- the semantic identity scope being reconciled;
- an optional domain semantic-policy prompt asset;
- canonical-field selection beyond choosing the supplied canonical member;
- domain field and evidence consolidation rules;
- durable ID derivation;
- domain warning scopes, reason codes, and messages; and
- domain postconditions and fallback behavior not covered by the shared core.

The adapter must not parse untyped durable JSON or weaken the artifact codec's
exact Go type. The core must not use reflection to infer domain fields. Shared
typed consolidation support may use Go generics and a deliberately small policy
interface where that removes demonstrated duplication without hiding domain
rules.

### Pipeline Boundary

Semantic reconciliation remains an implementation of the normalize stage. It
does not add a pipeline stage, hide normalization inside extraction, or create
an arbitrary workflow edge. Existing typed normalizer registrations remain the
public configuration boundary.

The initial implementation does not register one universal
`generic/deduplication` module key. A module key cannot safely accept arbitrary
artifact kinds under the current exact typed-registration contract. Domain
registrars instead instantiate the shared strategy for their own artifact
types.

## Request-Local Candidate Protocol

### Candidate Presentation

After deterministic preprocessing and eligibility filtering, the core assigns
the candidates visible to one completion request contiguous integer
`candidate_id` values beginning with `1`. The mapping is owned by that request
and retains the original typed candidate and input position internally.

Each model-facing candidate includes its `candidate_id`, contextual display
name or label, and the evidence and context needed for semantic judgment. In
the initial source-backed implementation, that context consists of validated
source-reference ranges and bounded, source-ordered transcript windows.
Application-owned durable entity IDs are never included.

Candidate IDs:

- are identifiers for prompt selections, not entity identities;
- have meaning only within one structured completion request;
- restart for each request or future batch;
- never enter a durable artifact or public schema;
- are not used to derive durable IDs; and
- must be resolved through the core's retained request-local mapping.

Only model-visible eligible candidates receive IDs. Filtering must not produce
gaps that increase model burden or reveal unrelated internal ordering.

### Proposal Response

The private structured response contains an ordered `duplicate_groups` array.
Each group contains:

- `candidate_ids`: at least two distinct supplied integer IDs; and
- `canonical_candidate_id`: one supplied ID that is also a member of that
  group.

The model does not return candidate names, source ranges, replacement records,
durable IDs, or synthesized canonical values. Choosing a canonical candidate
means selecting one supplied member; typed deterministic code constructs the
resulting record.

The response schema should reject unknown fields and require every defined
field. Application validation remains authoritative and must additionally
reject out-of-range IDs, repeated members, a canonical ID outside its group,
and any candidate appearing in more than one group. Schema validation is not a
substitute for these semantic checks.

### Deterministic Plan

The core resolves accepted IDs to internal candidates, orders members by their
original deterministic positions, and orders groups by their earliest member.
Candidates omitted from the response remain distinct. Invalid groups are never
partially applied.

The reconciliation plan retains enough internal provenance for typed
application, warning generation, debugging, and validation without exposing
request-local IDs as durable identity.

## Prompt Policy

### Mandatory Shared Protocol

The core owns a shared prompt fragment that defines the response protocol and
non-negotiable safety behavior. It instructs the model to:

- identify only well-supported groups that denote the same underlying entity;
- preserve candidates that are merely similar or uncertain;
- return only supplied candidate IDs;
- select one supplied group member as canonical;
- omit uncertain groups; and
- invent no candidates, evidence, attributes, identities, or replacements.

This protocol fragment and the private schema are not replaceable by a domain
adapter. Keeping them shared ensures identical mechanics across consumers and
provides one prompt prefix for review, testing, and provider caching.

### Generic Semantic Default

The core provides a conservative generic semantic-policy fragment suitable for
an artifact family whose notion of entity identity is adequately conveyed by
its candidate labels and evidence. It asks whether candidates refer to the same
underlying entity and treats uncertainty as a reason not to collapse them.

Use of the generic default is explicit in the typed adapter. It is not an
implicit fallback for an adapter that failed to declare its policy.

### Domain Semantic Policy

An artifact family may select a domain-owned semantic-policy fragment in place
of the generic semantic fragment. The replacement defines only domain judgment
and canonical-member preferences; it does not replace the shared protocol,
response schema, or deterministic safety rules.

The existing NPC, item, and location policies remain domain-owned because they
encode meaningful distinctions among people, item types or unique
designations, currency denominations, parent and child places, and same-name
physical locations. Their prompt manifests should select the shared protocol
and their local semantic fragment while using the same generic response
schema.

Prompt selection is initially an implementation-time artifact-family choice.
This work does not add arbitrary operator-supplied prompt paths or configuration
that can replace reconciliation safety policy.

## Context, Bounds, And Model Invocation

The first shared core is intentionally scoped to candidates grounded in a
Notarius source document. It validates candidate source references, creates
bounded windows around their evidence, preserves source order, and supplies
the model with the relationship between each candidate ID and its evidence.
Source ranges are model input but are never model output.

The core skips the LLM call when fewer than two candidates remain eligible. It
must also impose explicit candidate-count and rendered-context bounds. When an
input cannot be reconciled safely within those bounds, normalization preserves
the deterministic preprocessed result and produces bounded diagnostics under
the consuming module's established fallback policy. It must not silently
process arbitrary fixed-size slices that could separate duplicates.

Every completion uses the injected scheduled structured-LLM client, propagated
profile and session ID, cancellation, and the normalizer retry contract.
Invalid structured output or an unusable proposal follows the existing
retry-then-deterministic-fallback model. Transport and provider failures remain
execution errors rather than being silently converted into semantic absence.

## Deterministic Typed Application

The model never mutates the artifact. The shared core and typed adapter apply
only fully validated groups.

Application must preserve these invariants:

- no ungrouped candidate is inserted, removed, or changed by semantic
  reconciliation;
- every accepted group produces exactly one typed output record;
- canonical display fields come from a supplied group member unless a domain
  policy explicitly performs a deterministic transformation;
- all required provenance from group members is retained and canonicalized;
- output order follows the earliest contributing deterministic input position;
- durable IDs are recomputed by the domain identity policy after
  consolidation;
- caller-owned input and request material are never mutated;
- warnings identify every collapsed group using domain-owned scopes and reason
  codes; and
- warning volume remains bounded.

Any shared typed application helper must make these invariants structural while
leaving domain field merging and ID derivation explicit.

## Target D&D Consumers

### NPC Registry

NPC normalization uses the shared core while preserving its individual-person
identity semantics, proper-name canonicalization policy, evidence union,
deterministic NPC ID derivation, warning behavior, retry, and fallback result.
Its model response uses integer candidate selection rather than copied
contextual selectors.

### Item Registry

Item normalization uses the shared core while preserving the distinction
between item types and unique designations, distinct currency denominations,
non-inference of item properties or uniqueness, item identity derivation,
warnings, and fallback behavior.

### Location Registry

Location normalization uses the shared core while preserving its
physical-place identity semantics, treatment of parent and child places and
same-name places, source-reference-dependent durable IDs, warnings, and
fallback behavior.

The migrations must retain the existing durable artifact schemas and public
module keys. Prompt, schema, policy, and implementation fingerprint changes
must invalidate only the affected normalization checkpoints through the normal
checkpoint identity mechanism.

The target tree contains no superseded D&D-specific selector-copy response
schema, shared reconciliation implementation, or obsolete prompt assets. It
does not retain compatibility shims for the private pre-release LLM response
contract.

## Verification Strategy

Tests should protect the reconciliation contract and realistic failure modes,
not private helper structure or exact prompt length.

The shared core warrants focused behavioral coverage for:

- contiguous ID assignment after eligibility filtering;
- absence of durable IDs from model inputs and absence of evidence ranges from
  model outputs;
- exact ID resolution and rejection of zero, negative, unknown, repeated, and
  overlapping IDs;
- canonical membership and minimum group size;
- deterministic group and output ordering regardless of response order;
- preservation of ungrouped candidates and complete provenance;
- ownership and defensive-copy guarantees;
- cancellation, invalid structured output, retry, and fallback behavior;
- candidate and context bounds; and
- offline prompt/schema registration and structured response decoding.

Use table-driven or property-oriented tests where they efficiently protect
group-partition and preservation invariants. Fuzzing is appropriate for the
pure proposal assessor if it remains fast and deterministic. Do not add tests
that merely freeze prompt text, message counts, asset hashes, private constant
values, or implementation call choreography.

Each migrated D&D normalizer retains focused tests for its domain identity,
consolidation, warnings, durable IDs, and fallback behavior. A small number of
integration tests should prove that production registration supplies the
shared schema and selected prompt policy. Live-model evaluation remains a
human review tool and is not part of the default offline test suite.

## Architectural Decisions And Documentation

The target documentation set includes an accepted ADR applying
[ADR-0012](../adr/0012-resolve-opaque-entity-identifiers-deterministically.md)
to semantic reconciliation. That decision records:

- request-local ordinal candidate handles as the standard reconciliation
  selection mechanism;
- why requiring the model to reproduce contextual selectors is unnecessary
  and error-prone;
- the mandatory shared protocol plus generic or domain semantic-policy
  composition;
- LLM proposal versus deterministic application ownership;
- the typed adapter boundary; and
- rejected alternatives, including durable IDs, name-only selection,
  model-synthesized replacement records, and arbitrary untyped normalization.

ADR-0012's accepted decision text remains unchanged. The new ADR cites it and
provides the concrete justification it requires for request-local labels.

In the target state, architecture and internal documentation:

- distinguish artifact families from configured stage modules;
- identify the shared reconciliation core and its dependency direction;
- document the generic prompt protocol and domain semantic-policy ownership;
- document the integer candidate protocol and deterministic safety boundary;
- describe the three migrated registry normalizers accurately; and
- remove descriptions of contextual-selector response copying.

Until then, this roadmap remains the canonical description of the proposed
behavior; current-behavior documents must not describe it as implemented.

## Non-Goals

This work does not include:

- a new pipeline stage or a compound module that combines extraction and
  normalization;
- a universal configured normalizer for arbitrary artifact kinds or untyped
  JSON;
- a physical reorganization of every D&D package around artifact-family
  directories;
- operator-configurable arbitrary prompt assets or replacement of core safety
  instructions;
- naive batching, cross-batch clustering, or unbounded reconciliation inputs;
- semantic deduplication of event artifacts whose identity dimensions are
  already handled deterministically;
- alternate non-source context providers;
- changes to durable D&D artifact schemas or public module keys; or
- live provider calls in the default test suite.

## Completion Criteria

The feature is complete when:

- one domain-neutral core owns candidate IDs, source context, prompt execution,
  response assessment, retry classification, and common safe application
  mechanics;
- the private response schema uses only integer candidate handles for member
  and canonical selection;
- the generic safety protocol and conservative semantic default exist as
  shared assets;
- domain adapters can explicitly select the generic semantic default or a
  domain-owned semantic fragment without replacing core safety behavior;
- NPC, item, and location registry normalizers use the shared core and retain
  their typed domain behavior and durable contracts;
- the model is not required to reproduce names, evidence ranges, durable IDs,
  or replacement records in reconciliation output;
- invalid proposals cannot partially mutate or partially collapse an artifact;
- oversized inputs preserve deterministic results rather than being naively
  divided;
- the superseded D&D-specific response and reconciliation path is removed;
- the new ADR and current-behavior documentation accurately reflect the
  implemented boundary; and
- focused package tests and the repository-wide Go test and vet suites pass.