notarius/docs/roadmap/semantic-reconciliation-candidate-numbers.md

# Semantic Reconciliation Candidate Numbers

## Purpose

Make the private semantic-reconciliation protocol easier for smaller models to
follow by clearly separating request-local candidate selection from transcript
unit addressing. Replace the ambiguous `candidate_id` vocabulary with explicit
one-based candidate numbers, declare the valid number range for every request,
and ensure both initial and corrective instructions describe the exact response
the model can produce.

## Motivation And Current State

The domain-neutral semantic-reconciliation core supplies each eligible registry
candidate with a contiguous, one-based integer handle. The model returns groups
of those handles, deterministic code resolves them to retained typed records,
and unsafe groups are discarded without merging their candidates. This is the
architecture required by ADR-0013 and correctly prevents model-generated values
from becoming durable entity identity.

The current private JSON vocabulary calls these handles `candidate_id`,
`candidate_ids`, and `canonical_candidate_id`. Candidate material and transcript
context also contain `start_unit_id`, `end_unit_id`, and transcript unit `id`
values. Although the shared protocol says to use only supplied candidate IDs,
all of these values are nearby integers and the term “ID” does not clearly
communicate which namespace is request-local.

An observed small-model run exhausted semantic-proposal retries after returning
a duplicate group with an unknown candidate member and fewer than two remaining
distinct valid members. The safe fallback behaved correctly: the group was
omitted and all affected items were preserved separately. The issue pattern is
consistent with confusing a transcript or evidence unit ID for a candidate
handle, though an ordinary warning intentionally does not retain the exact
model response needed to prove that inference.

The item-specific semantic instructions contain a related mismatch. They tell
the model to choose a supplied candidate name as canonical even though the
private response can return only the handle of the candidate whose label should
be retained.

## Decisions

### Use candidate-number vocabulary throughout the private protocol

The model-visible candidate material will use `candidate_number`. Duplicate
groups will use `candidate_numbers` and `canonical_candidate_number`. Matching
Go names, internal comments, diagnostics, tests, and current-behavior internal
documentation will use “candidate number” where they refer to this private
request-local value.

The rename applies coherently to the generic semantic-reconciliation core and
all current consumers: NPC, item, and location registry normalization. It does
not change the abstract architectural term “request-local candidate handle” in
ADR-0013; a candidate number remains the concrete representation of that
handle.

The old JSON property names will not be accepted as aliases. No compatibility
adapter, dual-field schema, migration reader, or legacy prompt will remain.
This is a private model contract, not a durable artifact contract.

### Declare the valid range in each request

Prepared candidate material will contain an explicit request-level range:

```json
{
  "candidate_number_range": {
    "first": 1,
    "last": 12
  },
  "candidates": [
    {
      "candidate_number": 1,
      "label": "Example",
      "source_refs": []
    }
  ]
}
```

`first` and `last` are required integers. For every model invocation, `first`
is 1 and `last` is the number of eligible candidates in the prepared request.
The candidate list remains contiguous and one-based. Preparation owns this
derived declaration and must keep it consistent with the emitted candidate
objects and retained mappings.

The initial shared protocol will state that valid candidate numbers are exactly
the declared inclusive range and that the model must use the explicit
`candidate_number` attached to each candidate. Transcript unit `id` values and
evidence `start_unit_id` and `end_unit_id` values identify source positions;
they are never candidate numbers and must never appear in the response merely
because they are integers.

### Make retry guidance request-aware

When proposal assessment finds a non-positive or unknown member or canonical
number, the model-facing correction will include the request’s exact valid
inclusive candidate-number range. It will retain response-local duplicate-group
ordinals so the model can locate the defective group in the appended prior
response. It will explicitly distinguish candidate numbers from transcript and
evidence unit IDs.

Correction guidance must remain bounded and semantically meaningful. It must
not expose internal issue categories, reason codes, Go paths, durable entity
IDs, source identities, or raw operator errors. The framework will continue to
append the exact latest defective response separately under the existing
replacement-request protocol; correction prose must not duplicate it.

### Describe canonical selection in terms of the actual response

The item-registry normalization instructions will say to set
`canonical_candidate_number` to the supplied candidate number whose label is
the clearest transcript-supported designation. Equivalent NPC and location
instructions must be reviewed for the same mismatch and corrected if present.
Domain instructions continue to decide semantic equivalence and preferred
canonical labels; the shared protocol continues to own selection mechanics.

## Target End State

- Every semantic-reconciliation prompt presents one unambiguous, request-local
  candidate-number namespace.
- Candidate material declares the exact valid inclusive range and assigns every
  eligible candidate one explicit number in that range.
- Transcript and evidence unit identifiers are clearly identified as a
  separate namespace that is never valid in proposal selection fields.
- Structured responses contain only `duplicate_groups`,
  `candidate_numbers`, and `canonical_candidate_number` at their existing
  logical levels; the former `candidate_id` property family is rejected.
- Proposal assessment remains deterministic, validates exact membership in the
  prepared candidate set, discards unsafe groups, applies independent safe
  groups, and preserves affected candidates separately on fallback.
- A retry for an invalid selection tells the model which response-local groups
  need correction and the exact valid candidate-number range without exposing
  internal diagnostic vocabulary.
- NPC, item, and location registry reconciliation use the same generic private
  protocol, while their typed consolidation and identity rules remain
  domain-owned.
- Durable registry schemas, artifact IDs, evidence ranges, warning policy,
  retry budgets, and PromptKit structural-repair budgets do not change.

## Required Work

### Generic protocol and model

- Rename the model-facing fields in the generic candidate material, proposal
  response types, private response schema, shared protocol prompt, and
  presentation prompt.
- Rename internal semantic-reconciliation fields and accessors where they
  represent request-local candidate numbers, so implementation terminology
  does not immediately drift back toward ambiguous IDs.
- Add the required `candidate_number_range` object to prepared material and
  derive it from the filtered eligible candidate set.
- Preserve the current contiguous one-based assignment, immutable retained
  mapping, bounded material limits, stable plan ordering, and exact typed
  resolution behavior.
- Keep exact candidate membership as deterministic semantic validation. The
  shared static response schema may enforce positive integers and structural
  shape, but it must not pretend to encode a request-specific maximum that it
  cannot know.

### Request-aware corrective feedback

- Make the valid candidate-number range available to the generic correction
  renderer through owned preparation or reconciliation result state.
- Add the exact range and the candidate-number/source-unit distinction to
  corrections for unknown or non-positive proposal members and canonical
  selections. Retain existing guidance for repeated members, undersized groups,
  canonical membership, and overlapping groups.
- Preserve stable grouping, de-duplication, response-local group ordinals, byte
  bounds, and separation between operator diagnostics and model guidance.
- Ensure NPC, item, and location normalizers use the generic renderer rather
  than duplicating range or namespace language. Item-specific currency
  guidance remains an additive domain rule.

### D&D semantic instructions

- Rewrite the item-registry canonical-selection sentence around
  `canonical_candidate_number` and the selected candidate’s label.
- Review the NPC- and location-registry normalization instructions and apply
  the same correction wherever they describe returning a name rather than
  selecting its candidate number.
- Do not duplicate the generic number-range or namespace rules in D&D-specific
  files; those rules belong in the mandatory shared protocol.

### Fingerprints and checkpoints

- Revise the existing private v1 prompt and schema assets in place; do not add
  legacy schema variants or compatibility aliases.
- Ensure changed protocol, candidate-material, response-schema, correction,
  and typed-policy behavior participates in the established prompt, asset, and
  checkpoint fingerprint mechanisms. Explicitly bump a core or local policy
  fingerprint wherever the effective behavior is not already covered by an
  asset digest.
- A checkpoint produced under the former candidate-ID contract must not be
  reused as compatible with the candidate-number contract.

## Testing

Add lean offline coverage at the narrowest stable boundaries:

- preparation tests should prove contiguous candidate numbers, an exact
  matching range declaration, filtering behavior, immutable mappings, and the
  absence of former model-facing field names;
- private-schema tests should accept the candidate-number proposal shape and
  reject the former candidate-ID fields, missing required properties, unknown
  properties, non-integer numbers, non-positive numbers, and undersized arrays;
- proposal-assessment tests should retain the existing safety matrix using the
  renamed vocabulary and prove that exact prepared-set membership still owns
  acceptance;
- correction tests should prove that an unknown candidate number produces the
  exact valid range and namespace distinction without internal issue labels,
  reason codes, or operator paths;
- prompt tests should prove selection of the shared protocol and the corrected
  domain instruction, without snapshotting complete prose or exact message
  lengths; and
- representative NPC, item, and location normalizer tests should prove safe
  application, corrective retry, and exhaustion fallback through the renamed
  private contract.

Do not add live-provider tests to the default suite. After implementation, a
maintainer may repeat the previously problematic small-model workload as
observational validation, preferably with an explicitly requested debug bundle
if the warning persists.

## Documentation

Update `docs/internal/modules.md` as the canonical description of the generic
semantic-reconciliation mechanism and `docs/internal/dnd.md` for its three D&D
registry consumers. Describe candidate numbers as request-local handles,
including the declared range and namespace separation, without repeating the
private JSON schema in both documents.

No new ADR is required. This work reinforces ADR-0012’s prohibition on opaque
model-reproduced identity and ADR-0013’s accepted request-local-handle design;
it changes only the concrete private vocabulary and reliability presentation.
The architecture policy already owns the durable invariant and need not change
unless implementation reveals a genuinely new architectural decision.

## Non-Goals

- Changing durable NPC, item, or location artifact schemas or identifiers.
- Asking the model to return candidate labels, evidence ranges, transcript
  text, source identities, or replacement records.
- Accepting both candidate-ID and candidate-number response shapes.
- Guessing that an unknown number denotes a transcript unit, list position, or
  different candidate and silently repairing the proposal.
- Weakening deterministic proposal validation or typed domain guards.
- Increasing stage retries or PromptKit structural-repair attempts to mask an
  ambiguous protocol.
- Introducing a request-specific dynamic response schema in this work set.

## Acceptance Criteria

- No model-facing semantic-reconciliation asset or private JSON shape uses the
  former `candidate_id`, `candidate_ids`, or `canonical_candidate_id` fields.
- Candidate material and retry guidance expose the exact valid inclusive
  candidate-number range for the current request.
- Shared instructions explicitly prohibit using transcript unit IDs or evidence
  range endpoints as candidate numbers.
- Item, NPC, and location canonical-selection instructions refer to the actual
  candidate-number response contract.
- Invalid or ambiguous proposals remain safely rejected without false merges;
  retry exhaustion preserves affected candidates separately and emits the
  existing process warning.
- Durable outputs and public schemas are unchanged, while incompatible private
  checkpoints are invalidated.
- Canonical internal documentation records the implemented terminology and
  ownership boundaries.
- Focused tests, `go test ./...`, `go vet ./...`, and
  `go build ./cmd/notarius` pass in a supported development environment.