Plan semantic reconciliation improvements

docs/roadmap/semantic-reconciliation.md

@@ -0,0 +1,426 @@
+# Semantic Reconciliation Roadmap
+
+## Purpose
+
+This roadmap defines a reusable, LLM-assisted semantic-reconciliation facility
+for source-backed entity registries. The facility will centralize the common
+candidate preparation, prompt execution, proposal validation, safety, retry,
+and consolidation mechanics currently implemented by the D&D NPC, item, and
+location registry normalizers while preserving typed, domain-owned output.
+
+The model will remain a constrained proposal source. Notarius will retain
+authority over candidate identity, proposal validation, deterministic mutation,
+provenance, durable identifiers, warnings, and final artifact construction.
+
+## Motivation
+
+NPC, item, and location registry normalization now demonstrate the same useful
+pattern: deterministic preprocessing produces candidate records, an LLM judges
+whether some candidates describe the same underlying entity, and deterministic
+code applies only safe proposed groups. The current D&D-shared implementation
+proves the approach, but it still duplicates orchestration and application
+logic across normalizers and requires the model to reproduce complete
+contextual selectors containing names and evidence ranges.
+
+The target design should provide one efficient and thoroughly tested
+reconciliation core that can support additional artifact families without
+moving domain semantics into generic code. It should also simplify the model's
+task by replacing selector reproduction with small request-local integer
+handles.
+
+## Goals
+
+- Establish one domain-neutral semantic-reconciliation core for source-backed
+  entity candidates.
+- Move all demonstrated common mechanics into that core, including structured
+  LLM execution and deterministic proposal assessment.
+- Preserve exact typed artifact ownership from merged input through normalized
+  output.
+- Give the model semantic evidence while asking it to return only small,
+  request-local candidate identifiers.
+- Provide a conservative generic semantic prompt policy that a typed artifact
+  family may use by default.
+- Allow an artifact family to supply narrower domain semantic policy without
+  replacing the mandatory shared protocol and safety instructions.
+- Have the NPC, item, and location registry normalizers use the shared core
+  without changing their durable artifact contracts, configured module keys,
+  domain identity rules, or fallback guarantees.
+- Make addition of another eligible registry normalizer primarily an adapter
+  and policy exercise rather than a copy of reconciliation machinery.
+
+## Terminology And Ownership
+
+A **stage module** remains one configured implementation of one pipeline stage.
+An extractor and a normalizer are separate stage modules even when they
+collaborate on the same artifact kind.
+
+An **artifact family** is the cohesive domain feature that owns an artifact's
+types, codec, extractor, merge choice, normalizer, validators, prompt policy,
+schemas, identity helpers, and reference projections. For example, D&D spells
+and the D&D NPC registry are artifact families whose implementations span
+multiple explicit pipeline stages.
+
+This terminology clarifies existing ownership without changing Notarius's
+fixed pipeline or combining stages. It is consistent with the accepted
+domain-first organization and typed generic-strategy boundary in
+[ADR-0004](../adr/0004-package-modules-by-domain.md). The D&D registrar remains
+responsible for composing D&D-owned stage modules and their policies; the
+generic core remains unaware of D&D types or semantics.
+
+## Target Architecture
+
+### Shared Core
+
+A domain-neutral framework package should own semantic reconciliation. It may
+depend on generic source and structured-completion contracts and may consume
+its scoped assets under `assets/generic/`, but it must not import a production
+domain or encode D&D identity rules.
+
+The shared core owns:
+
+- validation and defensive copying of its inputs;
+- candidate eligibility and stable input ordering;
+- assignment of model-visible request-local candidate IDs;
+- construction of bounded candidate and source-context materials;
+- invocation of the configured structured LLM prompt;
+- the private duplicate-group response schema;
+- exact resolution of response IDs to the candidates visible in that request;
+- rejection of malformed, unknown, repeated, ambiguous, or overlapping
+  proposals;
+- canonical ordering of accepted groups and group members;
+- construction of an immutable reconciliation plan;
+- shared retry, fallback, cancellation, and invalid-output classification;
+- generic consolidation and provenance mechanics that are demonstrably common
+  across typed consumers; and
+- metadata and checkpoint fingerprints for shared prompts, schemas, and policy
+  versions.
+
+The core should expose stable behavior rather than a collection of unrelated
+helpers. Its API should make the safe path direct: a caller supplies typed
+candidates and a narrow domain policy, and receives either a deterministic
+typed result or a validated reconciliation plan that can only be applied
+through the typed policy boundary.
+
+### Typed Artifact-Family Adapter
+
+Each consuming artifact family owns a typed adapter or policy that supplies the
+irreducibly domain-specific behavior:
+
+- projection of merged typed records into eligible reconciliation candidates;
+- the semantic identity scope being reconciled;
+- an optional domain semantic-policy prompt asset;
+- canonical-field selection beyond choosing the supplied canonical member;
+- domain field and evidence consolidation rules;
+- durable ID derivation;
+- domain warning scopes, reason codes, and messages; and
+- domain postconditions and fallback behavior not covered by the shared core.
+
+The adapter must not parse untyped durable JSON or weaken the artifact codec's
+exact Go type. The core must not use reflection to infer domain fields. Shared
+typed consolidation support may use Go generics and a deliberately small policy
+interface where that removes demonstrated duplication without hiding domain
+rules.
+
+### Pipeline Boundary
+
+Semantic reconciliation remains an implementation of the normalize stage. It
+does not add a pipeline stage, hide normalization inside extraction, or create
+an arbitrary workflow edge. Existing typed normalizer registrations remain the
+public configuration boundary.
+
+The initial implementation does not register one universal
+`generic/deduplication` module key. A module key cannot safely accept arbitrary
+artifact kinds under the current exact typed-registration contract. Domain
+registrars instead instantiate the shared strategy for their own artifact
+types.
+
+## Request-Local Candidate Protocol
+
+### Candidate Presentation
+
+After deterministic preprocessing and eligibility filtering, the core assigns
+the candidates visible to one completion request contiguous integer
+`candidate_id` values beginning with `1`. The mapping is owned by that request
+and retains the original typed candidate and input position internally.
+
+Each model-facing candidate includes its `candidate_id`, contextual display
+name or label, and the evidence and context needed for semantic judgment. In
+the initial source-backed implementation, that context consists of validated
+source-reference ranges and bounded, source-ordered transcript windows.
+Application-owned durable entity IDs are never included.
+
+Candidate IDs:
+
+- are identifiers for prompt selections, not entity identities;
+- have meaning only within one structured completion request;
+- restart for each request or future batch;
+- never enter a durable artifact or public schema;
+- are not used to derive durable IDs; and
+- must be resolved through the core's retained request-local mapping.
+
+Only model-visible eligible candidates receive IDs. Filtering must not produce
+gaps that increase model burden or reveal unrelated internal ordering.
+
+### Proposal Response
+
+The private structured response contains an ordered `duplicate_groups` array.
+Each group contains:
+
+- `candidate_ids`: at least two distinct supplied integer IDs; and
+- `canonical_candidate_id`: one supplied ID that is also a member of that
+  group.
+
+The model does not return candidate names, source ranges, replacement records,
+durable IDs, or synthesized canonical values. Choosing a canonical candidate
+means selecting one supplied member; typed deterministic code constructs the
+resulting record.
+
+The response schema should reject unknown fields and require every defined
+field. Application validation remains authoritative and must additionally
+reject out-of-range IDs, repeated members, a canonical ID outside its group,
+and any candidate appearing in more than one group. Schema validation is not a
+substitute for these semantic checks.
+
+### Deterministic Plan
+
+The core resolves accepted IDs to internal candidates, orders members by their
+original deterministic positions, and orders groups by their earliest member.
+Candidates omitted from the response remain distinct. Invalid groups are never
+partially applied.
+
+The reconciliation plan retains enough internal provenance for typed
+application, warning generation, debugging, and validation without exposing
+request-local IDs as durable identity.
+
+## Prompt Policy
+
+### Mandatory Shared Protocol
+
+The core owns a shared prompt fragment that defines the response protocol and
+non-negotiable safety behavior. It instructs the model to:
+
+- identify only well-supported groups that denote the same underlying entity;
+- preserve candidates that are merely similar or uncertain;
+- return only supplied candidate IDs;
+- select one supplied group member as canonical;
+- omit uncertain groups; and
+- invent no candidates, evidence, attributes, identities, or replacements.
+
+This protocol fragment and the private schema are not replaceable by a domain
+adapter. Keeping them shared ensures identical mechanics across consumers and
+provides one prompt prefix for review, testing, and provider caching.
+
+### Generic Semantic Default
+
+The core provides a conservative generic semantic-policy fragment suitable for
+an artifact family whose notion of entity identity is adequately conveyed by
+its candidate labels and evidence. It asks whether candidates refer to the same
+underlying entity and treats uncertainty as a reason not to collapse them.
+
+Use of the generic default is explicit in the typed adapter. It is not an
+implicit fallback for an adapter that failed to declare its policy.
+
+### Domain Semantic Policy
+
+An artifact family may select a domain-owned semantic-policy fragment in place
+of the generic semantic fragment. The replacement defines only domain judgment
+and canonical-member preferences; it does not replace the shared protocol,
+response schema, or deterministic safety rules.
+
+The existing NPC, item, and location policies remain domain-owned because they
+encode meaningful distinctions among people, item types or unique
+designations, currency denominations, parent and child places, and same-name
+physical locations. Their prompt manifests should select the shared protocol
+and their local semantic fragment while using the same generic response
+schema.
+
+Prompt selection is initially an implementation-time artifact-family choice.
+This work does not add arbitrary operator-supplied prompt paths or configuration
+that can replace reconciliation safety policy.
+
+## Context, Bounds, And Model Invocation
+
+The first shared core is intentionally scoped to candidates grounded in a
+Notarius source document. It validates candidate source references, creates
+bounded windows around their evidence, preserves source order, and supplies
+the model with the relationship between each candidate ID and its evidence.
+Source ranges are model input but are never model output.
+
+The core skips the LLM call when fewer than two candidates remain eligible. It
+must also impose explicit candidate-count and rendered-context bounds. When an
+input cannot be reconciled safely within those bounds, normalization preserves
+the deterministic preprocessed result and produces bounded diagnostics under
+the consuming module's established fallback policy. It must not silently
+process arbitrary fixed-size slices that could separate duplicates.
+
+Every completion uses the injected scheduled structured-LLM client, propagated
+profile and session ID, cancellation, and the normalizer retry contract.
+Invalid structured output or an unusable proposal follows the existing
+retry-then-deterministic-fallback model. Transport and provider failures remain
+execution errors rather than being silently converted into semantic absence.
+
+## Deterministic Typed Application
+
+The model never mutates the artifact. The shared core and typed adapter apply
+only fully validated groups.
+
+Application must preserve these invariants:
+
+- no ungrouped candidate is inserted, removed, or changed by semantic
+  reconciliation;
+- every accepted group produces exactly one typed output record;
+- canonical display fields come from a supplied group member unless a domain
+  policy explicitly performs a deterministic transformation;
+- all required provenance from group members is retained and canonicalized;
+- output order follows the earliest contributing deterministic input position;
+- durable IDs are recomputed by the domain identity policy after
+  consolidation;
+- caller-owned input and request material are never mutated;
+- warnings identify every collapsed group using domain-owned scopes and reason
+  codes; and
+- warning volume remains bounded.
+
+Any shared typed application helper must make these invariants structural while
+leaving domain field merging and ID derivation explicit.
+
+## Target D&D Consumers
+
+### NPC Registry
+
+NPC normalization uses the shared core while preserving its individual-person
+identity semantics, proper-name canonicalization policy, evidence union,
+deterministic NPC ID derivation, warning behavior, retry, and fallback result.
+Its model response uses integer candidate selection rather than copied
+contextual selectors.
+
+### Item Registry
+
+Item normalization uses the shared core while preserving the distinction
+between item types and unique designations, distinct currency denominations,
+non-inference of item properties or uniqueness, item identity derivation,
+warnings, and fallback behavior.
+
+### Location Registry
+
+Location normalization uses the shared core while preserving its
+physical-place identity semantics, treatment of parent and child places and
+same-name places, source-reference-dependent durable IDs, warnings, and
+fallback behavior.
+
+The migrations must retain the existing durable artifact schemas and public
+module keys. Prompt, schema, policy, and implementation fingerprint changes
+must invalidate only the affected normalization checkpoints through the normal
+checkpoint identity mechanism.
+
+The target tree contains no superseded D&D-specific selector-copy response
+schema, shared reconciliation implementation, or obsolete prompt assets. It
+does not retain compatibility shims for the private pre-release LLM response
+contract.
+
+## Verification Strategy
+
+Tests should protect the reconciliation contract and realistic failure modes,
+not private helper structure or exact prompt length.
+
+The shared core warrants focused behavioral coverage for:
+
+- contiguous ID assignment after eligibility filtering;
+- absence of durable IDs from model inputs and absence of evidence ranges from
+  model outputs;
+- exact ID resolution and rejection of zero, negative, unknown, repeated, and
+  overlapping IDs;
+- canonical membership and minimum group size;
+- deterministic group and output ordering regardless of response order;
+- preservation of ungrouped candidates and complete provenance;
+- ownership and defensive-copy guarantees;
+- cancellation, invalid structured output, retry, and fallback behavior;
+- candidate and context bounds; and
+- offline prompt/schema registration and structured response decoding.
+
+Use table-driven or property-oriented tests where they efficiently protect
+group-partition and preservation invariants. Fuzzing is appropriate for the
+pure proposal assessor if it remains fast and deterministic. Do not add tests
+that merely freeze prompt text, message counts, asset hashes, private constant
+values, or implementation call choreography.
+
+Each migrated D&D normalizer retains focused tests for its domain identity,
+consolidation, warnings, durable IDs, and fallback behavior. A small number of
+integration tests should prove that production registration supplies the
+shared schema and selected prompt policy. Live-model evaluation remains a
+human review tool and is not part of the default offline test suite.
+
+## Architectural Decisions And Documentation
+
+The target documentation set includes an accepted ADR applying
+[ADR-0012](../adr/0012-resolve-opaque-entity-identifiers-deterministically.md)
+to semantic reconciliation. That decision records:
+
+- request-local ordinal candidate handles as the standard reconciliation
+  selection mechanism;
+- why requiring the model to reproduce contextual selectors is unnecessary
+  and error-prone;
+- the mandatory shared protocol plus generic or domain semantic-policy
+  composition;
+- LLM proposal versus deterministic application ownership;
+- the typed adapter boundary; and
+- rejected alternatives, including durable IDs, name-only selection,
+  model-synthesized replacement records, and arbitrary untyped normalization.
+
+ADR-0012's accepted decision text remains unchanged. The new ADR cites it and
+provides the concrete justification it requires for request-local labels.
+
+In the target state, architecture and internal documentation:
+
+- distinguish artifact families from configured stage modules;
+- identify the shared reconciliation core and its dependency direction;
+- document the generic prompt protocol and domain semantic-policy ownership;
+- document the integer candidate protocol and deterministic safety boundary;
+- describe the three migrated registry normalizers accurately; and
+- remove descriptions of contextual-selector response copying.
+
+Until then, this roadmap remains the canonical description of the proposed
+behavior; current-behavior documents must not describe it as implemented.
+
+## Non-Goals
+
+This work does not include:
+
+- a new pipeline stage or a compound module that combines extraction and
+  normalization;
+- a universal configured normalizer for arbitrary artifact kinds or untyped
+  JSON;
+- a physical reorganization of every D&D package around artifact-family
+  directories;
+- operator-configurable arbitrary prompt assets or replacement of core safety
+  instructions;
+- naive batching, cross-batch clustering, or unbounded reconciliation inputs;
+- semantic deduplication of event artifacts whose identity dimensions are
+  already handled deterministically;
+- alternate non-source context providers;
+- changes to durable D&D artifact schemas or public module keys; or
+- live provider calls in the default test suite.
+
+## Completion Criteria
+
+The feature is complete when:
+
+- one domain-neutral core owns candidate IDs, source context, prompt execution,
+  response assessment, retry classification, and common safe application
+  mechanics;
+- the private response schema uses only integer candidate handles for member
+  and canonical selection;
+- the generic safety protocol and conservative semantic default exist as
+  shared assets;
+- domain adapters can explicitly select the generic semantic default or a
+  domain-owned semantic fragment without replacing core safety behavior;
+- NPC, item, and location registry normalizers use the shared core and retain
+  their typed domain behavior and durable contracts;
+- the model is not required to reproduce names, evidence ranges, durable IDs,
+  or replacement records in reconciliation output;
+- invalid proposals cannot partially mutate or partially collapse an artifact;
+- oversized inputs preserve deterministic results rather than being naively
+  divided;
+- the superseded D&D-specific response and reconciliation path is removed;
+- the new ADR and current-behavior documentation accurately reflect the
+  implemented boundary; and
+- focused package tests and the repository-wide Go test and vet suites pass.