810 lines
37 KiB
Markdown
810 lines
37 KiB
Markdown
# Semantic Reconciliation Implementation Plan
|
|
|
|
## Purpose
|
|
|
|
This document is the ordered implementation plan for the target state defined
|
|
in the [Semantic Reconciliation Roadmap](semantic-reconciliation.md). Each
|
|
numbered stage is sized for one gpt-5.6-terra implementation prompt and must be
|
|
completed in order.
|
|
|
|
The feature roadmap owns product intent and durable policy. This document owns
|
|
implementation sequence, package placement, private contracts, migration
|
|
mechanics, and verification. When a detail below conflicts with a current
|
|
implementation detail, preserve the feature-roadmap policy and deliberately
|
|
replace the superseded implementation.
|
|
|
|
## Execution Rules
|
|
|
|
For every stage:
|
|
|
|
1. Read `docs/development.md`, all files under `docs/policy/`, the feature
|
|
roadmap, the files named by the stage, and focused tests before editing.
|
|
2. Implement only the assigned stage and prerequisites left incomplete by an
|
|
earlier stage. Do not begin a later migration opportunistically.
|
|
3. Preserve unrelated user changes and public module keys. Do not weaken typed
|
|
artifact registration or introduce untyped JSON mutation.
|
|
4. Keep all default tests deterministic, offline, and credential-free. Use a
|
|
small fake structured-LLM client only at the external completion boundary.
|
|
5. Test package-level behavior and consequential invariants. Do not add prompt
|
|
length, exact asset hash, private constant, or collaborator-choreography
|
|
change detectors.
|
|
6. Run `gofmt` on changed Go files, the focused commands listed for the stage,
|
|
and `git diff --check`. Resolve failures before handing off the stage.
|
|
7. Do not update current-behavior documentation before Stage 11. ADR-0013 is
|
|
the documentation-policy exception and may record the accepted decision
|
|
before the complete implementation exists.
|
|
|
|
## Fixed Implementation Decisions
|
|
|
|
The following decisions are not left to individual stages:
|
|
|
|
- The shared package is `internal/framework/semanticreconcile`. It is
|
|
domain-neutral framework support, not a configured pipeline module and not a
|
|
D&D package.
|
|
- The initial core supports source-backed entity candidates only. It accepts
|
|
generic source references and source documents but no D&D types.
|
|
- The model-facing candidate object uses `candidate_id`, `label`, and
|
|
source-free `source_refs`. Eligible visible candidates receive contiguous
|
|
integer IDs beginning with `1`.
|
|
- The private response uses `duplicate_groups`, `candidate_ids`, and
|
|
`canonical_candidate_id`. It contains no names, labels, evidence ranges,
|
|
durable IDs, or replacement records.
|
|
- The generic response contract is version `v1`, with schema key
|
|
`semantic_reconciliation_llm`, schema ID
|
|
`notarius.generic.semantic_reconciliation.llm`, schema name
|
|
`notarius_semantic_reconciliation_llm_v1`, and registered filename
|
|
`semantic_reconciliation_llm.v1.json`.
|
|
- The complete generic default prompt ID is
|
|
`generic.semantic_reconciliation`, version `v1`. It has no default LLM
|
|
profile; callers must pass the resolved normalization profile.
|
|
- Core prompt protocol and schema are mandatory. A domain prompt may replace
|
|
only the generic semantic-policy message and must continue to select the
|
|
core-owned protocol and input presentation assets.
|
|
- Default source-context limits are radius `2`, at most `128` eligible
|
|
candidates, and at most `262144` total serialized bytes across candidate and
|
|
transcript input materials. Define these through one core-owned default
|
|
limits value. Test custom limits relationally rather than duplicating the
|
|
production literals throughout tests.
|
|
- Fewer than two eligible candidates skips the LLM without warning. Exceeding
|
|
a deterministic limit skips the LLM, preserves the deterministic
|
|
preprocessed artifact, and emits a bounded domain fallback warning without a
|
|
futile retry.
|
|
- Invalid structured output requests the normalizer's existing retry behavior.
|
|
A structurally valid proposal may retain independent safe groups while
|
|
discarding invalid groups; any discarded group requests a retry, and retry
|
|
exhaustion uses the existing deterministic fallback-warning contract.
|
|
- Transport, provider, cancellation, and context-material construction errors
|
|
remain execution errors. They are not converted into empty semantic results.
|
|
- The shared application helper never emits domain warnings or derives durable
|
|
IDs. It owns partition application, stable ordering, preservation of
|
|
ungrouped records, and provenance bookkeeping; typed domain policy owns value
|
|
consolidation, group guards, IDs, and warning presentation.
|
|
- Production registration of the generic prompt and schema belongs to the
|
|
generic registrar, which already runs before the D&D registrar. D&D prompt
|
|
definitions may mount core-owned shared assets but remain owned and hashed by
|
|
their D&D normalizer packages.
|
|
- Existing D&D prompt IDs and prompt versions remain unchanged. The new generic
|
|
schema has its own `v1` identity, so no private selector-schema compatibility
|
|
layer is required.
|
|
- Bump normalization policies to `dnd.npc_registry.normalize.v5`,
|
|
`dnd.item_registry.normalize.v3`, and
|
|
`dnd.location_registry.normalize.v3` during their respective migrations.
|
|
- No public configuration, durable D&D schema, integration contract, module
|
|
key, or validator-chain change is part of this work.
|
|
|
|
## Stage 1: Record The Semantic Reconciliation Decision
|
|
|
|
### Goal
|
|
|
|
Record the accepted architectural decision before introducing the reusable
|
|
mechanism, without describing unimplemented behavior in current architecture
|
|
or internal documentation.
|
|
|
|
### Work
|
|
|
|
- Create
|
|
`docs/adr/0013-use-request-local-candidate-handles-for-semantic-reconciliation.md`
|
|
using the repository's Nygard ADR format and `Status: Accepted`.
|
|
- Cite ADR-0003, ADR-0004, ADR-0009, and ADR-0012 where their typed-artifact,
|
|
package-boundary, evidence, and opaque-ID decisions apply.
|
|
- Record these decisions:
|
|
- semantic reconciliation is a framework mechanism used by typed normalize
|
|
stage modules;
|
|
- a model receives contextual candidate data but returns request-local
|
|
one-based integer handles only;
|
|
- the LLM proposes duplicate groups and a supplied canonical member;
|
|
- deterministic code validates and applies the proposal;
|
|
- a mandatory shared protocol is combined with an explicit generic or
|
|
domain-owned semantic policy; and
|
|
- durable IDs, copied contextual selectors, synthesized replacements,
|
|
reflection-based arbitrary JSON, and hidden cross-stage behavior are
|
|
rejected alternatives.
|
|
- State explicitly that acceptance does not imply implementation completion
|
|
and link to the feature and implementation roadmaps for status.
|
|
- Do not modify the accepted decision text of ADR-0012.
|
|
|
|
### Acceptance Criteria
|
|
|
|
- The ADR contains context, decision, alternatives, and consequences and does
|
|
not claim that the new core already exists.
|
|
- No current-behavior document outside `docs/adr/` or `docs/roadmap/` changes.
|
|
- All ADR and roadmap links resolve.
|
|
|
|
### Validation
|
|
|
|
```sh
|
|
git diff --check
|
|
```
|
|
|
|
## Stage 2: Build Bounded Candidate And Source-Context Preparation
|
|
|
|
### Goal
|
|
|
|
Create the domain-neutral candidate preparation boundary with contiguous
|
|
request-local IDs and bounded prompt materials, without changing any D&D
|
|
normalizer yet.
|
|
|
|
### Work
|
|
|
|
- Create `internal/framework/semanticreconcile` with package documentation and
|
|
source-context preparation code adapted from
|
|
`internal/modules/dnd/shared/entityreconcile/context.go`. Leave the old D&D
|
|
package in place until all migrations finish.
|
|
- Define a domain-neutral `Candidate` containing a contextual `Label` and owned
|
|
generic `[]source.SourceRef`. Do not add a durable ID field.
|
|
- Define `Limits` and one `DefaultLimits()` value with the fixed radius,
|
|
candidate, and serialized-material bounds above. Validation rejects negative
|
|
radius and non-positive maximums before doing work.
|
|
- Prepare candidates in caller order:
|
|
- validate every source reference against `source.DocumentIndex`;
|
|
- exclude a candidate with no references or any invalid reference;
|
|
- canonicalize its source-free ranges in source order and remove exact
|
|
duplicate ranges;
|
|
- assign IDs only to eligible candidates and make them contiguous from `1`;
|
|
and
|
|
- retain an internal ID-to-original-candidate-position mapping.
|
|
- Do not exclude two candidates merely because their labels and ranges are
|
|
identical. Their local integer IDs now disambiguate them.
|
|
- Serialize candidate input as an object containing `candidates`, where each
|
|
entry has `candidate_id`, `label`, and `source_refs` with only
|
|
`start_unit_id` and `end_unit_id`.
|
|
- Adapt the existing coalesced transcript-window algorithm. Preserve source
|
|
order, cloned unit metadata, and the `cited` marker; do not mutate the source
|
|
document or candidate references.
|
|
- Return an explicit preparation disposition for:
|
|
- ready materials;
|
|
- fewer than two eligible candidates; and
|
|
- candidate or combined-material limit exceeded.
|
|
- Enforce the candidate limit before rendering large materials and the byte
|
|
limit over the sum of serialized candidate and transcript material. Do not
|
|
split the request.
|
|
- Construct `contracts.LLMInputMaterial` values with stable content type,
|
|
content digest, and the existing `candidates` and `transcript` input names.
|
|
- Expose owned accessors for visible candidate mappings and materials. Do not
|
|
expose mutable retained slices or maps.
|
|
|
|
### Tests
|
|
|
|
- Port and revise the valuable behavior tests from `entityreconcile`, replacing
|
|
selector-collision expectations with contiguous-ID behavior.
|
|
- Cover invalid and missing references, non-sequential source unit IDs,
|
|
coalesced windows, candidate filtering, identical descriptors, nil source,
|
|
input ownership, and deterministic serialization.
|
|
- Use custom small limits to prove exactly-at-limit acceptance and one-over
|
|
skipping for candidate count and material bytes. Do not assert the production
|
|
default literals merely as change detectors.
|
|
- Prove model material contains no source document ID or application entity ID.
|
|
|
|
### Acceptance Criteria
|
|
|
|
- The package imports no D&D or production-module package.
|
|
- Same-label and same-range candidates remain independently selectable.
|
|
- No LLM client or response schema is introduced in this stage.
|
|
- Existing D&D behavior remains unchanged and all prior tests still pass.
|
|
|
|
### Validation
|
|
|
|
```sh
|
|
go test ./internal/framework/semanticreconcile
|
|
go test ./internal/modules/dnd/shared/entityreconcile
|
|
git diff --check
|
|
```
|
|
|
|
## Stage 3: Implement The Integer Proposal Contract And Assessor
|
|
|
|
### Goal
|
|
|
|
Add the private integer response schema and a pure deterministic assessor that
|
|
turns model proposals into immutable, safely ordered reconciliation plans.
|
|
|
|
### Work
|
|
|
|
- Add core response types corresponding exactly to:
|
|
- `duplicate_groups`;
|
|
- each group's `candidate_ids`; and
|
|
- `canonical_candidate_id`.
|
|
- Add
|
|
`assets/generic/normalize/deduplication/schemas/semantic_reconciliation_llm.v1.json`
|
|
with Draft 2020-12 metadata and the fixed schema identity above.
|
|
- Require all fields, reject unknown fields, require at least two group members,
|
|
require positive integers, and use `uniqueItems: true` for member IDs. Keep
|
|
application semantic validation authoritative.
|
|
- Add schema loading metadata and helpers to the core package, following the
|
|
existing `internal/framework/llm` response-schema pattern.
|
|
- Implement assessment against the prepared request mapping:
|
|
- reject zero, negative, or unknown member and canonical IDs;
|
|
- reject repeated members even if schema validation was bypassed in a unit
|
|
test;
|
|
- reject groups with fewer than two distinct valid members;
|
|
- reject a canonical ID that is not a valid member;
|
|
- mark every otherwise-local-valid group sharing a member with another such
|
|
group as conflicting and discard all conflicting groups;
|
|
- retain independent safe groups even when another group is discarded; and
|
|
- report stable issue categories and original response group indexes for
|
|
retry diagnostics.
|
|
- Produce an immutable plan expressed in original candidate positions rather
|
|
than model-visible IDs. Sort members by original position and safe groups by
|
|
earliest member so response ordering cannot alter deterministic output.
|
|
- Provide owned accessors for plan groups, issues, discarded-group count, and
|
|
whether retry is required.
|
|
|
|
### Tests
|
|
|
|
- Cover empty proposals, valid multi-member groups, unknown IDs, repeated IDs,
|
|
canonical-not-member, too-small groups, overlapping groups, independent safe
|
|
plus invalid groups, and response-order independence.
|
|
- Compile and validate representative accepted and rejected JSON against the
|
|
actual schema offline.
|
|
- Prove returned plans and issues cannot mutate retained assessment state.
|
|
- Add a focused fuzz test only if it remains small and protects the partition
|
|
and no-panic invariants more efficiently than table cases.
|
|
|
|
### Acceptance Criteria
|
|
|
|
- The model response types contain no contextual selectors or source ranges.
|
|
- Invalid groups cannot enter the plan, and overlapping groups cannot be
|
|
partially accepted.
|
|
- Plan ordering depends only on deterministic candidate order.
|
|
- The old D&D response schema remains temporarily available for unmigrated
|
|
normalizers.
|
|
|
|
### Validation
|
|
|
|
```sh
|
|
go test ./internal/framework/semanticreconcile
|
|
git diff --check
|
|
```
|
|
|
|
## Stage 4: Add Generic Prompt Assets And Production Asset Registration
|
|
|
|
### Goal
|
|
|
|
Provide one complete generic default prompt, reusable mandatory protocol/input
|
|
assets for domain prompts, and one production registration owner for the new
|
|
prompt and schema.
|
|
|
|
### Work
|
|
|
|
- Replace the two draft files currently under
|
|
`assets/generic/normalize/deduplication/` with a coherent asset set:
|
|
- `prompts/prompt.yaml` for `generic.semantic_reconciliation` `v1`;
|
|
- `prompts/system.md` with provider-neutral structured reconciliation role;
|
|
- `prompts/protocol.md` with the mandatory integer selection and safety
|
|
contract;
|
|
- `prompts/instructions.md` with the conservative generic semantic default;
|
|
- `prompts/candidates.md`; and
|
|
- `prompts/transcript-windows.md`.
|
|
- The default prompt declares required `candidates` and `transcript`
|
|
`application/json` inputs, omits `default_profile`, selects the new schema,
|
|
uses zero PromptKit repair attempts, and orders messages from stable to
|
|
variable:
|
|
1. generic system;
|
|
2. mandatory protocol;
|
|
3. generic semantic instructions, carrying the ephemeral cache marker for
|
|
the stable prefix;
|
|
4. candidate material; and
|
|
5. transcript windows.
|
|
- Keep protocol, semantic policy, and presentation as separate assets even
|
|
though the rendered prompt is compact. Domain prompts must be able to reuse
|
|
protocol and presentation while substituting only semantic instructions.
|
|
- In `internal/framework/semanticreconcile`, add scoped asset loading,
|
|
deterministic prompt/schema hashes, `RegisterAssets`, and a narrow allowlist
|
|
helper that returns core-owned shared prompt files for a consuming manifest.
|
|
The root `assets` package remains unchanged and contains no logic.
|
|
- Extend `internal/modules/dnd/shared.PromptAssetManifest` to accept explicitly
|
|
supplied external shared files in addition to named D&D shared files. Include
|
|
those external files in both the mounted prompt filesystem and prompt hash.
|
|
Reuse `promptfs.SharedPromptFile`; do not teach D&D shared code paths about a
|
|
hard-coded generic directory.
|
|
- Update `internal/modules/generic/register.Register` to require a non-nil asset
|
|
registry and register the semantic-reconciliation assets before its existing
|
|
modules and validators. Preserve contextual duplicate-registration errors.
|
|
- Do not yet change the three D&D normalization prompts or remove their old
|
|
schema registration.
|
|
|
|
### Tests
|
|
|
|
- Test the complete generic prompt and schema can be registered and prepared
|
|
offline with an explicit test profile and representative integer candidate
|
|
material.
|
|
- Verify the prepared output contract selects the generic schema and that the
|
|
rendered prompt asks for integer IDs rather than names or source ranges in
|
|
the response. Do not snapshot all prose or its exact length.
|
|
- Extend D&D manifest tests for external shared-file mounting, hashing, missing
|
|
files, and duplicate destinations at the stable manifest boundary.
|
|
- Update generic registrar tests for nil assets, successful asset registration,
|
|
and duplicate registration without duplicating PromptKit internals.
|
|
|
|
### Acceptance Criteria
|
|
|
|
- Generic production registration makes the complete default prompt and schema
|
|
available before D&D registration.
|
|
- Domain prompt manifests can mount the exact core protocol bytes without
|
|
copying them.
|
|
- The generic prompt has a stable prefix followed only by variable candidates
|
|
and transcript material.
|
|
- Existing D&D prompt behavior remains operational during the migration.
|
|
|
|
### Validation
|
|
|
|
```sh
|
|
go test ./internal/framework/semanticreconcile ./internal/framework/promptfs
|
|
go test ./internal/modules/generic/register ./internal/modules/dnd/shared
|
|
go test ./internal/cli -run 'Production|Catalog|Prompt'
|
|
git diff --check
|
|
```
|
|
|
|
## Stage 5: Add The Shared Structured-LLM Reconciliation Engine
|
|
|
|
### Goal
|
|
|
|
Centralize prompt execution and outcome classification behind one
|
|
provider-neutral engine while leaving typed artifact mutation to later stages.
|
|
|
|
### Work
|
|
|
|
- Add an `Engine` in `internal/framework/semanticreconcile` constructed with a
|
|
non-nil `contracts.StructuredLLMClient`, validated limits, and a prompt spec.
|
|
A prompt spec contains a non-empty prompt ID, version, and SHA-256 digest; it
|
|
may identify the generic default or a domain-owned prompt that obeys the same
|
|
schema. Validate the digest at construction so manifest metadata and
|
|
checkpoint fingerprints never silently omit the selected prompt bytes.
|
|
- Define a request containing stage name, source document, candidates, resolved
|
|
profile ID, and session ID. Preserve profile and session values exactly in
|
|
`contracts.StructuredCompletionRequest`.
|
|
- The engine must:
|
|
- reject nil engine, client, context, and invalid construction state with
|
|
contextual errors;
|
|
- honor context cancellation before material preparation and completion;
|
|
- use Stage 2 preparation and skip without calling the LLM for insufficient
|
|
or over-limit inputs;
|
|
- call `CompleteStructured` exactly once for a ready request, using only the
|
|
`candidates` and `transcript` materials;
|
|
- classify `contracts.ErrInvalidStructuredOutput` as a retryable semantic
|
|
outcome rather than a transport error;
|
|
- return other completion failures as errors with stage context;
|
|
- assess a decoded response through Stage 3; and
|
|
- return a safe plan, stable issues, discarded count, and a disposition that
|
|
distinguishes complete, retryable invalid structured output, retryable
|
|
discarded proposal groups, insufficient candidates, and deterministic
|
|
limit skip.
|
|
- Keep normalize-stage reason codes, warning text, and `NormalizeRetry`
|
|
construction out of the engine. The engine exposes neutral classifications
|
|
that typed normalizers translate through their existing policies.
|
|
- Expose core manifest metadata and checkpoint fingerprints for prompt ID,
|
|
version and hash, schema identity and hash, core reconciliation policy, and
|
|
complete limits. Domain normalizers append their identity and normalization
|
|
policy fingerprints rather than rebuilding core metadata independently.
|
|
- Fingerprint the limit policy as one canonical core-owned value so any limit
|
|
change invalidates relevant normalization checkpoints without test suites
|
|
duplicating every literal.
|
|
|
|
### Tests
|
|
|
|
- Use a compact recording fake client to cover request propagation, successful
|
|
assessment, empty groups, invalid structured output, transport failure,
|
|
cancellation, insufficient candidates, candidate limit, and material limit.
|
|
- Prove no LLM call occurs for deterministic skip outcomes.
|
|
- Prove repeated calls do not retain plans, issues, candidate mappings, or
|
|
warnings from earlier calls.
|
|
- Test metadata and fingerprints relationally: required categories must be
|
|
present and change when supplied prompt/schema/limit policy changes; do not
|
|
freeze current digest values.
|
|
|
|
### Acceptance Criteria
|
|
|
|
- Consuming modules no longer need to construct the common structured
|
|
completion request once migrated.
|
|
- The engine has no dependency on pipeline retry orchestration or a D&D type.
|
|
- Provider and transport behavior remain behind `StructuredLLMClient`.
|
|
- The old normalizers remain unchanged and compiling.
|
|
|
|
### Validation
|
|
|
|
```sh
|
|
go test ./internal/framework/semanticreconcile
|
|
go test ./internal/framework/llm ./internal/framework/contracts
|
|
git diff --check
|
|
```
|
|
|
|
## Stage 6: Add Generic Typed Plan Application
|
|
|
|
### Goal
|
|
|
|
Centralize safe partition application, ordering, and provenance bookkeeping for
|
|
typed records without moving domain consolidation rules into the core.
|
|
|
|
### Work
|
|
|
|
- Add a generic record envelope that owns a typed value, sorted unique original
|
|
input indexes, and earliest deterministic input position. Constructors and
|
|
accessors must preserve caller ownership.
|
|
- Add a deliberately small typed application policy with these responsibilities:
|
|
- clone one typed value;
|
|
- optionally reject an otherwise safe semantic group through a stable neutral
|
|
category; and
|
|
- consolidate owned member values using one supplied canonical member.
|
|
- Implement plan application that:
|
|
- validates plan positions against the provided record slice;
|
|
- never mutates records or policy inputs;
|
|
- preserves every ungrouped record as an owned clone;
|
|
- emits exactly one record for an accepted group;
|
|
- preserves all members unchanged when the typed guard rejects a group;
|
|
- unions and sorts original input indexes in core bookkeeping;
|
|
- sets the resulting earliest position to the earliest member;
|
|
- places output by earliest contributing position; and
|
|
- returns neutral applied-group and rejected-group events containing the
|
|
provenance needed for domain warning and retry construction.
|
|
- Treat a policy consolidation error as an execution error unless it is the
|
|
explicit safe-group rejection result. Do not silently preserve records after
|
|
an unexpected application failure.
|
|
- Do not make the helper aware of source references, names, durable IDs,
|
|
currencies, people, or locations. Domain consolidators continue to union and
|
|
canonicalize evidence inside their typed value and derive the final ID.
|
|
|
|
### Tests
|
|
|
|
- Use a minimal synthetic typed record to cover no groups, one and multiple
|
|
groups, ungrouped preservation, canonical member selection, output ordering,
|
|
provenance union, guard rejection, malformed plan positions, consolidation
|
|
failure, nil versus empty ownership, and non-mutation.
|
|
- Prefer table-driven invariant tests over tests for private loops or callback
|
|
invocation order.
|
|
|
|
### Acceptance Criteria
|
|
|
|
- A future domain adapter can apply a plan without reimplementing partition
|
|
traversal or provenance ordering.
|
|
- Rejected groups preserve all members exactly once.
|
|
- The helper cannot derive a domain ID or emit a domain warning.
|
|
|
|
### Validation
|
|
|
|
```sh
|
|
go test ./internal/framework/semanticreconcile
|
|
git diff --check
|
|
```
|
|
|
|
## Stage 7: Migrate NPC Registry Normalization
|
|
|
|
### Goal
|
|
|
|
Use the shared engine and typed application path for NPC reconciliation while
|
|
preserving the existing NPC artifact and failure semantics.
|
|
|
|
### Work
|
|
|
|
- Update the NPC normalization prompt manifest and assets:
|
|
- keep prompt ID `dnd.npc_registry.normalize` and version `v1`;
|
|
- retain the D&D system prompt and NPC-owned semantic instructions;
|
|
- mount the generic protocol, candidate presentation, and transcript-window
|
|
assets from `semanticreconcile`;
|
|
- order messages as D&D system, generic protocol, NPC semantic instructions
|
|
with the stable-prefix cache marker, candidates, then transcript windows;
|
|
and
|
|
- select `semantic_reconciliation_llm.v1.json`.
|
|
- Remove requirements for the model to copy names or ranges. Adjust NPC policy
|
|
prose to describe integer candidate selection while retaining all current
|
|
individual-person and canonical-name distinctions.
|
|
- Construct a shared engine in `npcregistry.New`, using the NPC prompt spec and
|
|
default limits. Replace direct `BuildContext`, `CompleteStructured`, and
|
|
`Assessment` orchestration with one engine call.
|
|
- Convert deterministic preprocessed NPC records into core typed envelopes and
|
|
candidates. Keep exact deterministic name/evidence normalization before the
|
|
semantic pass.
|
|
- Implement the narrow NPC typed policy:
|
|
- no additional group guard;
|
|
- choose the canonical member's normalized NPC name;
|
|
- union and source-order all member references;
|
|
- derive the final NPC ID through `npcs/identity`; and
|
|
- return an owned value.
|
|
- Translate engine and application outcomes to existing NPC warnings and retry
|
|
behavior. Safe independent groups may appear in the retry candidate result;
|
|
invalid structured output retries from the deterministic value; limit skip
|
|
returns the deterministic value with one bounded reconciliation-exhausted
|
|
warning and no retry.
|
|
- Preserve existing NPC reason-code strings and warning cap. Bump only the
|
|
normalization policy to `v5`.
|
|
- Replace locally assembled common prompt/schema/context metadata and
|
|
fingerprints with the core-provided values plus NPC identity and
|
|
normalization policy. Retain user-useful manifest field names where practical
|
|
but ensure the new generic schema identity and complete limits are recorded.
|
|
- Delete NPC-only key translation, safe-group traversal, and duplicated generic
|
|
application helpers made obsolete by the core. Retain NPC-specific
|
|
preprocessing, consolidation policy, and diagnostics.
|
|
|
|
### Tests
|
|
|
|
- Rewrite fake LLM responses to return integer IDs and inspect candidate input
|
|
only where needed to prove contiguous IDs and no durable identity leakage.
|
|
- Preserve behavior coverage for exact deterministic consolidation, semantic
|
|
aliases, canonical name choice, evidence union, ID recomputation, warnings,
|
|
invalid proposal retry, exhaustion fallback, cancellation, ownership, and
|
|
no-call cases.
|
|
- Add focused coverage for identical contextual NPC descriptors remaining
|
|
independently addressable and for a limit skip producing no LLM call.
|
|
- Update prompt preparation tests for the generic schema and shared protocol
|
|
without snapshotting full prompt prose or exact message length.
|
|
|
|
### Acceptance Criteria
|
|
|
|
- NPC normalization imports `semanticreconcile` and no longer imports the old
|
|
D&D `entityreconcile` package.
|
|
- Its model response requires only integer candidate handles.
|
|
- Durable NPC output and module key remain unchanged.
|
|
- Focused tests demonstrate behavior parity plus the safer selection protocol.
|
|
|
|
### Validation
|
|
|
|
```sh
|
|
go test ./internal/modules/dnd/normalize/npcregistry
|
|
go test ./internal/framework/semanticreconcile
|
|
go test ./internal/modules/dnd/register -run 'NPC|Prompt|Schema'
|
|
git diff --check
|
|
```
|
|
|
|
## Stage 8: Migrate Item Registry Normalization
|
|
|
|
### Goal
|
|
|
|
Move item registry reconciliation onto the shared path while preserving its
|
|
item-type and currency safety rules.
|
|
|
|
### Work
|
|
|
|
- Update the item normalization prompt exactly as in Stage 7, retaining prompt
|
|
ID `dnd.item_registry.normalize`, version `v1`, and item-owned semantic policy
|
|
while selecting the generic protocol, presentation assets, and schema.
|
|
- Construct and use the shared engine with default limits. Convert deterministic
|
|
item records into core candidates and typed envelopes.
|
|
- Implement the item typed policy:
|
|
- retain the existing currency-denomination group guard;
|
|
- reject a group mixing currency with non-currency or different
|
|
denominations through the core's explicit typed-group rejection path;
|
|
- choose the canonical member's normalized item name;
|
|
- union and source-order evidence;
|
|
- derive the item ID through `items/identity`; and
|
|
- preserve all current item-type and unique-designation semantics.
|
|
- Count a typed currency guard rejection with discarded model groups for retry
|
|
and exhaustion diagnostics. Preserve rejected group members unchanged and
|
|
exactly once.
|
|
- Translate insufficient, limit, invalid structured, invalid proposal, valid,
|
|
and transport outcomes consistently with NPC behavior while retaining item
|
|
reason codes and warning cap.
|
|
- Bump the item normalization policy to `v3`, adopt core metadata and
|
|
fingerprints, and remove item copies of generic key translation, group
|
|
traversal, and application logic.
|
|
|
|
### Tests
|
|
|
|
- Convert semantic response fixtures to integer IDs.
|
|
- Retain coverage for item aliases, canonical names, evidence, IDs, warnings,
|
|
retries, fallback, ownership, and no-call cases.
|
|
- Specifically protect same-denomination alias acceptance, cross-denomination
|
|
rejection, currency/non-currency rejection, preservation of every rejected
|
|
member, and combination of an accepted independent group with a rejected
|
|
group.
|
|
- Update prompt/schema preparation tests without exact prompt snapshots.
|
|
|
|
### Acceptance Criteria
|
|
|
|
- Item normalization no longer imports old `entityreconcile` code.
|
|
- Its domain guard is expressed only through the typed policy and cannot be
|
|
bypassed by a schema-valid proposal.
|
|
- Durable item output and public module behavior remain unchanged.
|
|
|
|
### Validation
|
|
|
|
```sh
|
|
go test ./internal/modules/dnd/normalize/itemregistry
|
|
go test ./internal/framework/semanticreconcile
|
|
go test ./internal/modules/dnd/register -run 'Item|Prompt|Schema'
|
|
git diff --check
|
|
```
|
|
|
|
## Stage 9: Migrate Location Registry Normalization
|
|
|
|
### Goal
|
|
|
|
Complete adoption by moving location registry reconciliation onto the shared
|
|
engine and typed application path.
|
|
|
|
### Work
|
|
|
|
- Update the location normalization prompt as in Stages 7 and 8, retaining
|
|
prompt ID `dnd.location_registry.normalize`, version `v1`, and the
|
|
location-owned physical-place semantic policy.
|
|
- Construct and use the shared engine with default limits. Convert deterministic
|
|
location records into core candidates and typed envelopes.
|
|
- Implement the location typed policy:
|
|
- no additional group guard beyond existing location semantic policy;
|
|
- choose the supplied canonical member's normalized location name;
|
|
- union and source-order all member references;
|
|
- derive the location ID from the final name and final references through
|
|
`locations/identity`; and
|
|
- preserve same-name, parent/child, and physical-place distinctions.
|
|
- Translate outcomes through existing location retry, fallback, reason-code,
|
|
and warning-cap behavior. Bump the normalization policy to `v3`.
|
|
- Adopt core metadata and fingerprints and delete location copies of generic
|
|
reconciliation and application mechanics.
|
|
|
|
### Tests
|
|
|
|
- Convert semantic response fixtures to integer IDs while retaining physical
|
|
place, same-name, canonical-name, evidence-dependent ID, warning, retry,
|
|
fallback, ownership, and no-call coverage.
|
|
- Prove same-name records with distinct evidence can both be addressed by IDs
|
|
without copied selectors.
|
|
- Update prompt/schema preparation tests without freezing prompt prose.
|
|
|
|
### Acceptance Criteria
|
|
|
|
- All three semantic D&D registry normalizers use the shared core.
|
|
- Location durable IDs are still derived only after final evidence union.
|
|
- No location code imports old `entityreconcile`.
|
|
|
|
### Validation
|
|
|
|
```sh
|
|
go test ./internal/modules/dnd/normalize/locationregistry
|
|
go test ./internal/framework/semanticreconcile
|
|
go test ./internal/modules/dnd/register -run 'Location|Prompt|Schema'
|
|
git diff --check
|
|
```
|
|
|
|
## Stage 10: Remove The Legacy Reconciliation Path And Consolidate Registration
|
|
|
|
### Goal
|
|
|
|
Delete superseded selector-copying code and assets, leave one schema and core,
|
|
and verify production composition as a whole.
|
|
|
|
### Work
|
|
|
|
- Delete `internal/modules/dnd/shared/entityreconcile` after confirming no
|
|
imports remain.
|
|
- Delete `assets/dnd/entity-reconciliation/` and
|
|
`assets/dnd/shared/prompts/common-dnd-entity-reconciliation.md`.
|
|
- Remove the obsolete shared prompt-name entry and the D&D registrar's old
|
|
entity-reconciliation schema registration.
|
|
- Remove any obsolete local candidate or transcript template files from the
|
|
three normalization asset trees after their manifests use the shared generic
|
|
presentation assets. Keep only their `prompt.yaml` and domain semantic
|
|
`instructions.md` unless another file remains genuinely domain-specific.
|
|
- Ensure the generic registrar is the sole production owner of generic
|
|
reconciliation prompt/schema registration and that the existing production
|
|
registrar order remains generic, Seriatim, D&D.
|
|
- Update registrar and CLI composition tests so they prove assembled production
|
|
assets are complete. A D&D registrar unit test may pre-register required
|
|
generic assets or limit itself to D&D-owned assets; do not duplicate generic
|
|
asset ownership inside D&D merely to preserve an old isolated-test setup.
|
|
- Search all Go, asset, and roadmap-adjacent current documentation for old
|
|
schema keys, selector response types, `common-dnd-entity-reconciliation.md`,
|
|
and model-output source ranges. Remove only obsolete uses; source ranges
|
|
remain valid model input.
|
|
- Review the three normalizers side by side. Consolidate any remaining
|
|
demonstrated neutral retry/adaptation helper into the core if it can be done
|
|
without domain warning semantics; otherwise keep the difference explicit.
|
|
|
|
### Tests
|
|
|
|
- Run the complete framework, generic module, D&D module, D&D registrar, and
|
|
production composition suites.
|
|
- Add or retain one assembled prompt/schema test per meaningful owner rather
|
|
than reproducing every core schema case in all three domains.
|
|
- Confirm duplicate production registration still fails contextually and no
|
|
prompt or schema path collides.
|
|
|
|
### Acceptance Criteria
|
|
|
|
- Exactly one semantic reconciliation response schema and assessor exist.
|
|
- No model-facing reconciliation response requires a name or evidence range.
|
|
- Production composition registers every selected D&D normalization prompt and
|
|
the generic schema exactly once.
|
|
- No compatibility shim or dead legacy asset remains.
|
|
|
|
### Validation
|
|
|
|
```sh
|
|
go test ./internal/framework/...
|
|
go test ./internal/modules/generic/...
|
|
go test ./internal/modules/dnd/...
|
|
go test ./internal/cli -run 'Production|Catalog|Prompt|Schema'
|
|
rg -n 'dnd_entity_reconcile_llm|common-dnd-entity-reconciliation|entityreconcile' internal assets docs/internal docs/policy docs/config.md
|
|
git diff --check
|
|
```
|
|
|
|
The `rg` command should return no live legacy references. Separately review
|
|
roadmap and ADR historical references rather than deleting accurate decision
|
|
history.
|
|
|
|
## Stage 11: Finalize Current Documentation And Repository Verification
|
|
|
|
### Goal
|
|
|
|
Make current documentation accurately describe the implemented architecture,
|
|
then perform the complete offline verification pass.
|
|
|
|
### Work
|
|
|
|
- Update `docs/policy/architecture.md` to:
|
|
- distinguish an artifact family from one configured stage module without
|
|
changing the fixed pipeline;
|
|
- identify semantic reconciliation as a domain-neutral framework mechanism;
|
|
- state that request-local candidate handles are an approved application of
|
|
ADR-0012 and link ADR-0013; and
|
|
- retain typed domain mutation, evidence, and dependency-direction
|
|
invariants.
|
|
- Update `docs/internal/overview.md` with the implemented
|
|
`internal/framework/semanticreconcile` responsibility, linking to the focused
|
|
internal owners rather than duplicating mechanics.
|
|
- Update `docs/internal/modules.md` to document artifact-family ownership,
|
|
stage-module registration, and how a typed normalizer instantiates the shared
|
|
strategy.
|
|
- Update `docs/internal/dnd.md` to replace selector-copying descriptions with
|
|
the integer candidate protocol, generic protocol/domain policy composition,
|
|
limit behavior, typed application boundary, and the three registry-specific
|
|
rules. Keep exact public keys and validator chains in `docs/config.md` rather
|
|
than duplicating them.
|
|
- Update `docs/internal/llm.md` only as needed to identify ownership and
|
|
registration of the generic prompt/schema assets and checkpoint metadata.
|
|
- Do not change integration contracts unless repository inspection finds an
|
|
incorrect statement: the durable artifact shapes are intentionally
|
|
unchanged. Do not add user configuration documentation because this feature
|
|
adds no configuration.
|
|
- Confirm ADR-0013 accurately matches the final implementation. Do not rewrite
|
|
its accepted decision to accommodate accidental implementation drift; fix
|
|
the implementation or create a superseding decision if a genuine conflict
|
|
was discovered.
|
|
- Review `docs/roadmap/future.md` only for links and scope boundaries. Keep
|
|
batching, operator-selected policy, broader inputs, a universal module key,
|
|
and physical package reorganization deferred.
|
|
- Run formatting, tests, vet, build, whitespace, link, and stale-reference
|
|
checks. No command may contact a live LLM provider.
|
|
|
|
### Acceptance Criteria
|
|
|
|
- Current docs describe only implemented behavior and assign each fact to its
|
|
canonical owner.
|
|
- The architecture clearly separates artifact families, stage modules, shared
|
|
model judgment, and typed deterministic application.
|
|
- All durable D&D schemas, module keys, and examples remain valid.
|
|
- No stale selector-copying documentation or asset path remains.
|
|
- The repository-wide test, vet, and build checks pass offline.
|
|
|
|
### Validation
|
|
|
|
```sh
|
|
go test ./...
|
|
go vet ./...
|
|
go build ./cmd/notarius
|
|
git diff --check
|
|
```
|
|
|
|
Run `gofmt` on every changed Go file before these commands; do not pass package
|
|
directories to `gofmt`.
|
|
|
|
Also verify every changed Markdown link target manually or with the repository's
|
|
available link checker. Review `git status --short` and the complete diff to
|
|
confirm that no unrelated file or generated secret-bearing material was added.
|