notarius/docs/roadmap/implementation.md

D&D Location Tracking Implementation Plan

Objective

Implement the target state in D&D Location Tracking: an evidence-grounded dnd/locations registry lane and a dependent dnd/location-occurrences lane, including conservative location identity, shared D&D entity-reconciliation infrastructure, production validation, generated-reference wiring, maintained examples, and current documentation.

This is an ordered implementation plan for a gpt-5.6-terra coding agent. Implement one stage per prompt, in order. Finish each stage's tests and leave the repository coherent before proceeding. Do not implement later-stage production registrations early merely to make an incomplete feature selectable.

All stages must follow:

• Architecture Policy
• Testing Policy
• Documentation Policy
• the D&D conventions in D&D Module Internals
• the durable policy decisions in the feature roadmap

Use behavior-level tests. Do not add tests that merely freeze source layout, exact prompt wording, message counts, shared-prefix length, or other incidental implementation details. Keep tests deterministic, offline, and owned by the component whose behavior they exercise.

Stage 1: Define Location Domain Types And Identity

Goal

Establish the in-process contracts and deterministic identity policy on which both lanes depend.

Work

• Extend internal/modules/dnd/types.go with:
  • LocationListKind = dnd/location-list;
  • LocationOccurrenceListKind = dnd/location-occurrence-list;
  • LocationList, Location, LocationOccurrenceList, LocationOccurrence, and LocationOccurrenceKind;
  • exact JSON members and the four occurrence constants specified in location.md.
• Add internal/modules/dnd/locations/identity.
• Implement display normalization and comparison normalization consistently with the existing NPC identity policy. Share a lower-level comparison helper only if doing so preserves NPC behavior exactly; otherwise keep the small policy-specific function explicit.
• Implement the versioned compact-JSON-array ID derivation contract from location.md, including ID syntax checks and immutable list validation.
• Identity validation must require correctly derived, unique IDs while allowing two records to have the same comparison name when their evidence anchors differ.
• Add focused tests for Unicode normalization, whitespace, apostrophes, deterministic encoding, evidence ordering, same-name/different-anchor IDs, malformed IDs, missing evidence, and non-mutation.

Acceptance Criteria

• The types compile without production registration.
• ID derivation exactly follows the documented five-element compact JSON input.
• Same normalized name plus different earliest evidence yields different IDs.
• Validation does not reject same-name records solely because their names match, and it reports duplicate or mismatched IDs deterministically.
• go test ./internal/modules/dnd/locations/... ./internal/modules/dnd/... passes for the packages available at this stage.

Prompt Size

Small enough for one implementation prompt.

Stage 2: Add Durable Codecs And Schemas

Goal

Create strict durable JSON ownership for both artifact kinds without exposing either lane as a selectable pipeline yet.

Work

• Add internal/modules/dnd/codec/locations and internal/modules/dnd/codec/locationoccurrences, following the existing D&D candidate/approved codec pattern.
• Add embedded Draft 2020-12 schemas with the IDs, names, root members, required fields, enums, source-reference shape, and additionalProperties: false contracts in location.md.
• Keep both schemas at v1.
• Support strict candidate decoding before semantic approval and strict durable encoding/decoding after approval.
• Add representative valid fixtures and tests for schema metadata, defensive schema bytes, empty arrays, unknown fields, missing fields, invalid types, invalid enum values, malformed source references, invalid ID syntax, and round trips.

Acceptance Criteria

• Each codec advertises the correct artifact kind and metadata count.
• Candidate decoding preserves semantic mistakes for validators while rejecting structurally invalid JSON.
• Approved encoding and decoding enforce the durable shape.
• go test ./internal/modules/dnd/codec/locations/... ./internal/modules/dnd/codec/locationoccurrences/... passes offline.

Prompt Size

Small enough for one implementation prompt.

Stage 3: Extract Shared Entity-Reconciliation Infrastructure

Goal

Create the D&D-shared, domain-safe proposal machinery needed by both NPC and location normalization, without changing NPC production behavior yet.

Work

• Add internal/modules/dnd/shared/entityreconcile.
• Move or generalize the reusable behavior currently owned by internal/modules/dnd/normalize/npcs/context_material.go and proposal.go:
  • assign deterministic opaque candidate keys such as candidate-000001 in input order;
  • clone candidate names and source references;
  • build bounded transcript windows in source-document order;
  • omit candidates whose references cannot safely produce context;
  • coalesce overlapping or adjacent windows without mutating the source;
  • define the private duplicate_groups proposal with members and canonical candidate keys;
  • reject blank, unknown, repeated, ineligible, overlapping, too-small, or canonical-not-a-member groups; and
  • return defensive, immutable assessment data identifying only safe groups.
• Keep LLM calls, retry decisions, artifact mutation, canonical-name policy, durable ID derivation, and warning wording out of this package.
• Add a shared prompt instruction asset that states the key-copying and proposal-safety contract without NPC- or location-specific identity rules.
• Add a shared private structured-response schema and loader/registration support with a stable v1 key, ID, name, and fingerprint. Registering the schema more than once must not be required.
• Add table-driven tests for context bounds, ordering, invalid references, coalescing, every unsafe proposal category, non-overlapping safe groups, deterministic keys, defensive copies, and non-mutation.

Acceptance Criteria

• The package has no dependency on dnd.NPC, dnd.Location, either normalizer, or a concrete LLM client.
• Proposal values can identify duplicate candidates even when display names are equal.
• The shared response contract cannot directly supply replacement records or evidence.
• Existing NPC packages still compile before their migration.
• go test ./internal/modules/dnd/shared/... passes offline.

Prompt Size

Medium, but coherent and suitable for one implementation prompt. Do not combine it with the NPC migration.

Stage 4: Migrate NPC Normalization To The Shared Helper

Goal

Make the existing NPC normalizer the first production consumer of the shared entity-reconciliation contract while retaining its durable behavior.

Work

• Refactor internal/modules/dnd/normalize/npcs to use opaque candidate keys, shared context construction, shared proposal assessment, the shared response schema, and the shared generic reconciliation instruction asset.
• Retain NPC-owned responsibilities:
  • comparison-name preparation and deterministic duplicate handling;
  • the NPC-specific task and canonical display-name rules;
  • LLM invocation, bounded retry, fallback, warnings, and diagnostics;
  • application of safe groups, evidence union, NPC ID derivation, and output ordering.
• Remove superseded NPC-private context/proposal code and private schema assets once no longer referenced.
• Update NPC prompt metadata and checkpoint fingerprints for the intentional prompt/private-schema contract change.
• Preserve public module keys, durable NPC schema, identity policy, validator chains, warning bounds, and fallback semantics.
• Test equal display names as distinct keyed candidates, alias consolidation, rejected unsafe groups, retry exhaustion, private input ownership, redacted errors, deterministic fallback, and non-mutation.

Acceptance Criteria

• No durable NPC artifact field or module key changes.
• NPC normalization cannot confuse two candidates merely because their display names match.
• Unsafe proposals leave a valid deterministic result and follow existing retry/fallback policy.
• Obsolete NPC-only reconciliation helpers and schema are removed.
• go test ./internal/modules/dnd/normalize/npcs/... ./internal/modules/dnd/shared/... passes offline.

Prompt Size

Medium-to-large but bounded to one existing module. Suitable for one implementation prompt; do not add location normalization in this stage.

Stage 5: Implement Location Extraction

Goal

Add the LLM-backed extractor that produces evidence-grounded location candidates.

Work

• Add internal/modules/dnd/extract/locations following current D&D extractor conventions: strict empty options, typed builder and registration function, llm_backed execution metadata, immutable inputs, redacted errors, prompt and response-schema fingerprints, and bounded diagnostics.
• Add a private response schema containing only name and source ranges; the model must not produce durable IDs or prose.
• Compose the prompt from existing shared D&D system, identity, campaign reference, transcript, and evidence assets plus module-owned task and instructions. Preserve the documented extraction-message ordering and cache controls.
• Define physical-place inclusion and conservative omission exactly as in location.md, including generic labels, aliases, and nested places.
• Map source ranges to the current source ID, canonicalize exact duplicate ranges, derive candidate location IDs in code, preserve semantically invalid candidates for validators where safe, and return deterministic ordering.
• Add prompt-asset tests that verify shared asset reuse and rendered inputs by behavior, without asserting exact shared-prefix length or prompt wording.
• Add extractor tests for empty output, mapping, evidence ownership, generic same-name locations with different anchors, invalid candidate preservation, client failures, registration metadata, fingerprints, and non-mutation.

Acceptance Criteria

• The extractor cannot manufacture source identities or accept campaign references as evidence.
• The private model response does not contain a durable ID.
• Same-name candidates with different evidence survive extraction as distinct candidates.
• The package is testable through its local registration but is not yet added to the production D&D registrar.
• go test ./internal/modules/dnd/extract/locations/... passes offline.

Prompt Size

Medium and suitable for one implementation prompt.

Stage 6: Implement Semantic Location Normalization

Goal

Add conservative alias and repeated-place reconciliation without collapsing same-named or nested locations by default.

Work

• Add internal/modules/dnd/normalize/locations as an llm_backed normalizer using the shared entity-reconciliation package and private response schema.
• Deterministically clone and prepare the merged candidates first:
  • normalize display whitespace;
  • canonicalize and deduplicate source references;
  • remove only exact duplicates with the same comparison name and exact canonical evidence;
  • assign opaque reconciliation keys; and
  • retain same-name records with different evidence.
• Use bounded transcript windows and a module-owned location task that permits grouping only when evidence clearly identifies one physical place. Explicitly prohibit grouping solely by equal names, proximity, nesting, or generic labels.
• Validate proposals through the shared package. Apply only safe groups in deterministic code, choose the canonical name from the selected existing candidate, union evidence, and derive the final evidence-anchored ID.
• Retain the deterministic candidate set on unusable proposals and follow the existing NPC retry/fallback and bounded-warning conventions.
• Publish prompt, response-schema, identity-policy, normalization-policy, and semantic-context fingerprints.
• Test aliases, repeated appearances, same-name distinct places, parent/child locations, invalid and overlapping proposals, proposal retries, fallback, ordering, ID recomputation, warning bounds, idempotent deterministic application, and non-mutation.

Acceptance Criteria

• The model proposes groups but cannot directly replace durable locations.
• A failed or ambiguous proposal cannot lose a valid candidate.
• Same-name locations remain distinct unless an approved evidence-backed group joins them.
• Final IDs are derived only after group evidence is unioned.
• go test ./internal/modules/dnd/normalize/locations/... passes offline.

Prompt Size

Medium-to-large but scoped to one normalizer and suitable for one implementation prompt.

Stage 7: Add The Immutable Location Registry

Goal

Provide safe generated-reference resolution and an unambiguous prompt projection for downstream occurrence extraction.

Work

• Add internal/modules/dnd/locations/registry, modeled on the immutable NPC registry and its operation-time resolver.
• Define ReferenceSlot = "locations", a 1,048,576-byte limit, and exactly one accepted application/json location-list item when bound.
• Validate durable decoding and location identity before constructing a registry.
• Store canonical durable bytes and semantic digests without retaining mutable caller-owned content. Return defensive copies from all accessors.
• Produce a compact, source-free prompt projection containing ordered {id, name} pairs. Do not include source references or generated-reference provenance.
• Support exact lookup by ID and verify the matching canonical name; do not provide an ambiguous name-only lookup as the occurrence linkage mechanism.
• Preserve the established seeded/operation resolver behavior and concurrency- safe semantic caching.
• Test absent, empty, malformed, oversized, wrong-media-type, invalid-identity, and valid registries; projections; ID lookup; defensive copies; raw and semantic cache reuse; and concurrent resolution.

Acceptance Criteria

• Distinct same-name records are both representable and addressable by ID.
• Registry evidence cannot appear in the prompt projection.
• Malformed static references fail during construction and malformed generated references fail at operation resolution through existing boundaries.
• go test ./internal/modules/dnd/locations/registry/... passes offline.

Prompt Size

Medium and suitable for one implementation prompt.

Stage 8: Implement Location-Occurrence Extraction

Goal

Add the dependent LLM-backed lane that classifies source-grounded location occurrences.

Work

• Add internal/modules/dnd/extract/locationoccurrences with module key dnd/location-occurrences, llm_backed execution metadata, strict empty options, typed construction, and a required locations reference slot.
• Resolve the immutable registry at construction and for each operation using the established generated-reference pattern.
• Add a private response schema requiring location_id, name, kind, and source ranges. Restrict kinds to visited, planned, recalled, and mentioned.
• Reuse the shared D&D extraction prompt assets and ordering. Place the compact location registry after the shared transcript/evidence material and before module task/instructions, consistent with current generated grounding.
• Encode the exact classification rules, precedence, multi-fact behavior, and conservative omission policy from location.md.
• Map evidence only to the current source. Copy candidate IDs and names without silently repairing unknown or mismatched values so deterministic validators retain ownership of those diagnostics.
• Canonically order output and exact duplicates without dropping distinct kinds or independent evidence.
• Test every kind, precedence, multiple supported facts, no-location and no- occurrence outputs, required registry failures, same-name ID selection, source-free prompt projection, current-transcript evidence, prompt/profile metadata, client failures, non-mutation, and local registration.

Acceptance Criteria

• Construction and operation specs declare locations as required and accept only dnd/location-list JSON.
• The model sees IDs and names but no registry evidence.
• Registry context never becomes occurrence evidence.
• The extractor remains locally testable but is not production-selectable yet.
• go test ./internal/modules/dnd/extract/locationoccurrences/... passes offline.

Prompt Size

Medium and suitable for one implementation prompt.

Stage 9: Implement Deterministic Occurrence Normalization

Goal

Canonicalize occurrence records against the exact location registry without a second LLM call.

Work

• Add internal/modules/dnd/normalize/locationoccurrences as a deterministic normalizer with the same required locations slot.
• Clone all inputs. Normalize source ranges, names, ordering, and exact duplicates.
• For a known location_id, replace display-name variation with the registry's exact canonical name. Do not perform a name-only guess.
• Preserve an unknown ID or otherwise invalid record for validator diagnostics and emit bounded warnings where current D&D normalizer conventions require them.
• Sort using the complete order defined in location.md.
• Publish normalization and registry-projection fingerprints consistent with the other registry-backed normalizers.
• Test all kind values, canonical name replacement, same-name distinct IDs, exact-duplicate removal, distinct evidence retention, stable ordering, unknown IDs, malformed registry resolution, warnings, idempotence, and non-mutation.

Acceptance Criteria

• No LLM client or prompt assets are required.
• Canonicalization is exclusively ID-based.
• Invalid records are not silently redirected to a different location.
• go test ./internal/modules/dnd/normalize/locationoccurrences/... passes offline.

Prompt Size

Small enough for one implementation prompt.

Stage 10: Add Location Registry Validators

Goal

Give dnd/location-list the complete validator ownership expected of a production D&D artifact.

Work

• Add location validator packages under internal/modules/dnd/validate/locations for:
  • extraction shape;
  • source-reference bounds/current-chunk ownership;
  • advisory source relatedness; and
  • normalized identity derivation and ID uniqueness.
• Use shared D&D citation, unit-reference, diagnostic, and matching helpers where their contracts apply.
• The identity validator must allow repeated comparison names and validate the evidence-anchored derivation policy instead of importing NPC uniqueness assumptions.
• Keep diagnostics indexed, aggregated, bounded, stable, and free of raw prompt or reference content.
• Add tests for accepted values, every owned failure, same-name distinct locations, malformed/unreadable citations, advisory relatedness, bounds, registration metadata, fingerprints, nil safety where applicable, and non-mutation.

Acceptance Criteria

• Validator responsibilities do not overlap merely to increase test coverage.
• Relatedness remains advisory and uses only cited current-transcript text.
• Validators do not repair or mutate artifacts.
• go test ./internal/modules/dnd/validate/locations/... passes offline.

Prompt Size

Medium and suitable for one implementation prompt.

Stage 11: Add Location-Occurrence Validators

Goal

Give dnd/location-occurrence-list complete structural, registry, ordering, evidence, and advisory validation.

Work

• Add validator packages under internal/modules/dnd/validate/locationoccurrences for:
  • extraction shape and supported kinds;
  • required registry membership and exact location_id/name pairing;
  • normalized ordering and exact-duplicate invariants;
  • source-reference bounds/current-chunk ownership; and
  • advisory source relatedness.
• Reuse the immutable location resolver rather than decoding caller-owned references independently in each validator.
• Ensure same-name registry records remain distinguishable by ID.
• Keep registry context out of evidence checks.
• Add focused tests for each kind, unknown IDs, mismatched names, same-name locations, ordering, duplicates, malformed required references, invalid evidence, advisory diagnostics, registration metadata, fingerprints, diagnostic bounds, and non-mutation.

Acceptance Criteria

• An ID/name mismatch is rejected even when another registry record has the supplied name.
• Missing or malformed required registry references fail at the established boundary.
• Validators remain deterministic and do not alter occurrence records or the registry.
• go test ./internal/modules/dnd/validate/locationoccurrences/... passes offline.

Prompt Size

Medium and suitable for one implementation prompt.

Stage 12: Compose The Production D&D Family

Goal

Make both lanes selectable as one coherent production addition after all component contracts are present.

Work

• Extend internal/modules/dnd/register to register, in dependency-safe order:
  • both codecs;
  • both extractors;
  • typed append-order mergers;
  • the LLM-backed location normalizer;
  • the deterministic occurrence normalizer;
  • all validators;
  • shared reconciliation schema assets and both new prompt manifests;
  • evidence projectors; and
  • extract and normalize default validator chains.
• Ensure both LLM-backed modules select the maintained dnd-extraction fallback profile and inherit the existing profile policy.
• Define chain order consistently with existing D&D artifacts: generic JSON, shape, registry/identity or normalized invariants at the appropriate stage, source references, durable JSON Schema, then advisory relatedness.
• Update registrar tests for artifact kinds, keys, execution classes, reference slot requirements, builder construction, assets, profile use, evidence projection, chain contents/order, duplicate registration, and failure propagation.
• Update any integration-level artifact-kind allowlists or typed registries required by the framework; do not add module-specific orchestration logic.

Acceptance Criteria

• One D&D registration call exposes both complete lanes and no partial registration succeeds silently.
• Catalog inspection reports correct artifact kinds, stages, execution classes, reference slots, profiles, and fingerprints.
• Both artifact kinds support evidence projection without registry evidence leakage.
• go test ./internal/modules/dnd/register/... ./internal/modules/dnd/... passes offline.

Prompt Size

Medium-to-large but limited to composition and suitable for one implementation prompt.

Stage 13: Add Maintained Pipeline And Handoff Coverage

Goal

Exercise the feature through real configuration, ordered generated references, chunk operations, acceptance gates, and durable output.

Work

• Update examples/dnd-complete.config.yml:
  • add locations to the first descriptive step with extract, append-order merge, and LLM-backed normalize bindings;
  • add a generated locations reference to the next step;
  • add location-occurrences to that step with extract, append-order merge, and deterministic normalize bindings; and
  • add both lanes to evidence-context output where appropriate.
• Keep the minimal example minimal unless its stated purpose requires a location lane; do not turn it into a second complete example.
• Extend maintained example-loading/config-validation tests.
• Add integration tests that prove:
  • the normalized accepted registry is handed off in memory;
  • the occurrence lane cannot run before its producer;
  • missing, cyclic, wrong-kind, wrong-media-type, rejected, or unaccepted producers are rejected at the existing boundaries;
  • same-name locations remain distinguishable by ID through the handoff;
  • registry evidence never becomes occurrence evidence;
  • retries and checkpoints honor prompt, schema, identity, and generated- reference fingerprints; and
  • output contains both durable artifact envelopes and evidence context.
• Use recording/fake structured clients only; no network-dependent tests.

Acceptance Criteria

• The complete example loads through the real config path and exercises all registered D&D lanes.
• Ordered handoff failure semantics match the framework's existing fail-whole- run policy.
• Integration tests cover behavior rather than duplicating package internals.
• go test ./internal/modules/integration/... ./internal/config/... and any example-specific test targets pass offline.

Prompt Size

Medium-to-large but coherent as one end-to-end integration prompt.

Stage 14: Publish Current Documentation And Perform Final Verification

Goal

Make the implemented feature discoverable and retire fulfilled future-work language without leaving development-history documentation behind.

Work

• Create canonical integration contracts:
  • docs/integrations/dnd-location-artifacts.md;
  • docs/integrations/dnd-location-occurrence-artifacts.md.
• Update docs/config.md with both selectable keys, the locations reference slot and compatibility, execution classes, validators, default chains, and the complete-example link.
• Update docs/internal/dnd.md with nine-lane composition, shared entity reconciliation, evidence-anchored identity, occurrence grounding, prompt asset reuse, and intentional lane differences. Link to integration contracts instead of duplicating their JSON shapes.
• Update docs/integrations/json-output.md, README.md, and other current canonical inventories only where repository inspection shows that the new artifact kinds or maintained example must be listed.
• Remove the fulfilled Location Extraction section from docs/roadmap/future.md. Keep the generic LLM-assisted deduplication item and clarify only if needed that the new D&D helper does not fulfill that broader feature.
• Verify all relative links and search current documentation for stale seven-lane counts, missing keys, obsolete location-planning claims, and accidental claims that references are evidence.
• Run formatting, focused tests, the full Go test suite, static analysis, and repository-provided config/example checks. Inspect git diff --check and confirm no unrelated files changed.
• After implementation and verification are complete, leave location.md and this plan in place for the user's separate roadmap-retirement step; do not delete them unless explicitly asked.

Acceptance Criteria

• Current documentation describes the implemented contracts and only implemented behavior outside docs/roadmap/.
• The roadmap no longer presents completed location tracking as future work.
• Links, examples, module inventories, and default-chain tables agree with production registration.
• go test ./internal/modules/dnd/... passes.
• go test ./internal/modules/integration/... passes.
• go test ./... passes.
• go vet ./... passes.
• Repository-provided configuration/example validation passes.
• git diff --check reports no errors.

Prompt Size

Medium and suitable for one implementation prompt.

Open Questions

None. The feature roadmap fixes the artifact shapes, identity scope and derivation, occurrence categories, classification precedence, reference dependency, reconciliation safety boundary, pipeline placement, and non-goals needed to implement every stage without an additional product decision.