notarius/docs/roadmap/implementation.md

Minimal D&D Extraction Contracts Implementation Plan

Status: Implemented

Objective

Implement the durable contract redesign defined by Minimal D&D Extraction Contracts and ADR-0009. The result is a coordinated in-place redesign of NPC, spell-cast, and combat-turn extraction. Each lane must emit only its narrow, evidence-grounded facts; no removed rich-schema field may survive as an optional field, placeholder, compatibility shim, or deterministic synthesis.

This plan deliberately starts with NPCs because the NPC registry is a generated reference consumed by spell and combat extraction. Spell and combat then change independently, followed by one repository-wide contract and documentation pass.

Fixed Decisions And Guardrails

The implementing agent must treat these as decisions, not open design choices:

• Keep the artifact kinds dnd/npc-list, dnd/spell-list, and dnd/combat-turn-list, their module keys and capabilities, their media types, and their prompt IDs stable.
• Retain exactly the existing v1 schema keys, IDs, names, versions, filenames, prompt IDs, and prompt versions listed in the feature roadmap. Change their unpublished shapes and content in place; do not harmonize the private spell identity as part of this work.
• Do not add v2 assets, runtime version negotiation, migration code, dual-write behavior, a second module registration, or compatibility fixtures for the superseded pre-release shapes. Existing fixtures should be rewritten or deleted according to whether they still protect current behavior.
• Require the top-level list field and every record field. A list may be empty, but every returned record must have at least one source reference. Use strict JSON objects with unknown fields rejected and no nullable or optional legacy fields.
• Keep private source references limited to start_unit_id and end_unit_id. Mapping assigns the current source document ID; campaign references and an NPC registry never become event evidence.
• Keep the shared prompt ordering policy: stable shared instructions first, stable campaign and generated references next, module task material after those references, and the chunk-variable transcript last.
• Preserve the existing production validator-chain order and the rule that a configured chain is authoritative. Remove or simplify validators; do not silently reorder chains.
• Preserve the existing NPC ID derivation algorithm and comparison/display normalization for canonical names. Retain existing semantic-policy identifiers as well as schema and prompt versions; pre-redesign development state is disposable. Unchanged canonical names must not receive new IDs.
• Use deterministic code only for display normalization, known catalog or NPC canonicalization, application-owned IDs, evidence canonicalization, ordering, and exact duplicate collapse. Do not introduce fuzzy matching or inferred enrichment.
• Follow the testing policy: protect schemas and behavior at their owning boundaries, delete obsolete tests, and avoid tests that snapshot prompt prose or detect exact shared-prefix length. No live or paid model call is a stage completion requirement.
• Each stage must leave the repository compiling and go test ./... passing. Update all affected fixtures and callers within the stage that changes a public Go type; do not leave an intentionally broken intermediate commit.

Stage 1: Cut Over The NPC Contract And Name Projection

This stage establishes the redesigned generated reference on which later stages depend.

1.1 Replace the durable and private data shapes

• Reduce dnd.NPC in internal/modules/dnd/types.go to exactly ID, Name, and SourceRefs. Delete NPCRelationship and every alias, description, and relationship field or helper that becomes unused.
• Rewrite the existing strict durable schema dnd_npcs.v1.json with required top-level npcs, and records containing only required id, name, and source_refs. Keep the NPC codec at schema version v1, name notarius_dnd_npcs_v1, and its existing schema ID and media type.
• Rewrite the strict private schema dnd_npcs_llm.v1.json. Its records contain only required name and model-facing source ranges. Keep the extractor schema identity and prompt ID/version unchanged.
• Reduce the private response DTO and mapping accordingly. Preserve candidate data at the mapping boundary so deterministic validators, rather than mapping defaults, reject blank names or invalid ranges. Assign the current source ID and derive the NPC ID in application code.
• Rewrite the NPC task and instruction assets to ask only for individually identifiable NPC names and supporting transcript ranges. Explicitly exclude anonymous groups, generic roles, invented labels, descriptions, aliases, and relationships. Keep transcript material last in the manifest.

1.2 Simplify NPC identity, normalization, and validation

• Keep the identity policy at dnd.npcs.identity.v1. Retain display normalization, comparison keys, ID syntax, and deterministic ID derivation. Validate nonblank canonical names, exact ID/name agreement, duplicate canonical names, and duplicate IDs. Delete alias validation and alias-specific issue locations/codes.
• Keep NPC normalization at dnd.npcs.normalize.v1. Normalize the retained display name, derive its ID, canonicalize and deduplicate source references, consolidate records only by the canonical-name comparison key, preserve the first stable record, and union exact evidence. Remove alias promotion, relationship merge/rewrites, and their warning codes.
• Keep the existing NPC shape and source-relatedness policy identifiers. The shape validator checks only list presence, nonblank id and name, and nonempty source references; source-relatedness grounds a record only through its retained name. The existing source-ref validator remains the owner of range validity.
• Keep the default validation order domain-first, then JSON Schema, then the advisory relatedness validator. Simplify only the validator implementations and selections whose owned behavior changed.

1.3 Separate durable registry provenance from model input

• Continue to canonicalize and retain the complete redesigned NPC artifact for registry validation, cache identity, manifests, and provenance. Registry.Digest() remains the digest of that complete canonical artifact when bound and remains empty when no registry was supplied.
• Build a second structural JSON projection for model grounding with the exact shape {"npcs":[{"name":"Mira Thorn"}]} in normalized registry order. It contains names only: no IDs, source references, origin URI, aliases, or other provenance. Generate it with typed values and json.Marshal, not string concatenation. The empty projection is exactly {"npcs":[]}.
• Add an explicit projection digest accessor. The digest is SHA-256 over the exact projected bytes, including for an absent or empty registry, and PromptInput().Digest must equal it. Keep full-artifact and projection digests distinct even when their content happens to coincide.
• Index Registry.Lookup by canonical name only. Retain immutable return values and defensive copies. Remove alias indexing and alias-aware comments.
• In spell extraction, combat extraction, and combat normalization, use the NPC name-projection digest for the component-local checkpoint fingerprint because it exactly describes the names that affect those operations. Keep the full registry digest and count in manifest metadata for provenance. Framework-owned generated-reference fingerprints may still invalidate a run when any upstream artifact byte changes; do not broaden this stage into a framework cache redesign.

1.4 Update owners, consumers, and tests

• Update NPC codec, schema, extractor, identity, validator, normalizer, registry, registration, pipeline-integration, and CLI fixtures to the new shape. Adapt spell and combat tests that construct dnd.NPC values so the repository remains buildable, but do not change their own artifact contracts yet.
• Add or rewrite focused tests for strict schema acceptance/rejection, codec round trips, unchanged ID derivation for known names, name-only consolidation, evidence union, registry immutability, and canonical-name-only lookup. Rewrite the existing fixture for the minimal current contract; do not retain the superseded rich fixture solely to test backwards incompatibility.
• Test the projection as a data contract: it contains only ordered names, equivalent normalized registries produce identical bytes and digest, evidence/ID-only changes do not change the projection digest, and name/order changes do. This is not authorization to snapshot assembled prompt prose or prefix lengths.
• Update docs/integrations/dnd-npc-artifacts.md to own the redesigned v1 durable schema. Update docs/internal/modules.md and docs/internal/llm.md only for current NPC behavior that lands in this stage.

Stage 1 completion criteria

• NPC durable and private schemas expose no removed enrichment fields.
• The generated NPC prompt input contains names only while manifests retain full registry provenance.
• Spell and combat consumers accept generated or external registries in the redesigned v1 shape; rich pre-redesign registry JSON fails strict decoding.
• Focused NPC, spell-wiring, combat-wiring, integration, and CLI tests pass, and go test ./... passes.

Stage 2: Cut Over Spell-Cast Extraction

2.1 Replace the spell contract and prompt

• Reduce dnd.SpellCast to exactly Caster, Spell, and SourceRefs.
• Rewrite dnd_spells.v1.json in place. Keep the durable codec at version v1, name notarius_dnd_spells_v1, and its existing schema ID. Its strict record contains only required caster, spell, and source_refs.
• Rewrite dnd_spells_llm.v1.json with the same logical fields and model-facing ranges. Keep the existing private key, ID, name, prompt version, and schema path unchanged.
• Reduce the private DTO, canonicalization, and mapping to the retained fields. Continue assigning current source IDs, stable-ordering candidates by evidence, and preserving semantically invalid candidates for deterministic validation.
• Rewrite spell task and instruction assets around the narrow casting-event boundary. Retain the spell catalog and name-only NPC projection as disambiguation inputs, never evidence. Remove effect, outcome, and narrative duties and remove the common-dnd-immediate-resolution.md message from the spell manifest and spell asset registration. Do not delete the shared file in this stage because combat still uses it.

2.2 Simplify spell policies

• Keep the spell shape policy identifier unchanged and validate only list presence, nonblank caster/spell, and nonempty source references.
• Keep catalog validation, source-reference validation, and source-relatedness behavior and policy identities unchanged where their actual semantics are already limited to spell name, caster, and evidence.
• Simplify normalization and duplicate comparison to caster, catalog-canonical spell name, and the complete valid evidence set. Remove all prose selection, copying, fixture fields, and assertions. Preserve catalog fingerprints and metadata.
• Keep the exact inclusion rule from the feature roadmap: an actual casting or unambiguous declared attempt is included; mentions, plans, rules discussion, and catalog matches are not.

2.3 Update owners and tests

• Update spell codec, private schema, extractor, validators, normalizer, merge and registration tests, pipeline integration, CLI output fixtures, and any maintained examples to the minimal shape. Rewrite or delete rich-schema fixtures rather than retaining them as compatibility cases.
• Replace tests of effects and narrative descriptions with focused tests of the retained contract: strict unknown-field rejection, codec round trips, current source-ID assignment, catalog canonicalization, evidence ordering, duplicate collapse, and NPC name-projection wiring.
• Update docs/integrations/dnd-spell-artifacts.md as the canonical redesigned v1 contract and update current internal module/LLM documentation for the smaller prompt and response. Do not duplicate the spell catalog contract owned by its existing integration document.

Stage 2 completion criteria

• No production spell type, schema, prompt, validator, normalizer, fixture, or documentation contract refers to effect or narrative description.
• Spell prompt/schema identities remain exactly their existing v1 values.
• Focused spell and pipeline tests pass, and go test ./... passes.

Stage 3: Cut Over Combat-Turn Extraction

3.1 Replace the combat contract and prompt

• Reduce dnd.CombatTurn to exactly Actor, TurnKind, and SourceRefs. Delete CombatAction, CombatActionCategory, their constants, and all now-unused helpers. Retain the existing five CombatTurnKind values.
• Rewrite dnd_combat_turns.v1.json in place. Keep the durable codec at version v1, name notarius_dnd_combat_turns_v1, and its existing schema ID. Use a strict required record with actor, turn_kind, and source_refs only.
• Simplify the durable codec to direct strict encoding/decoding if its custom wire representation exists only to distinguish removed nullable fields. Preserve presence semantics for the top-level list and strict unknown-field rejection.
• Rewrite dnd_combat_turns_llm.v1.json while retaining its private schema and prompt identities, and reduce the DTO/mapping to actor, turn kind, and model-facing ranges. Keep the semantic mapping policy identifier unchanged.
• Rewrite combat task/instruction assets around detecting ordered turns and discrete interrupting events. Remove round, action, target, declaration, resolution, outcome, and summary duties. Remove the shared immediate- resolution message from the combat manifest.
• Once both spell and combat manifests no longer use it, delete common-dnd-immediate-resolution.md and its shared and module asset registrations. Retain all other shared evidence, identity, reference, NPC, and transcript assets in their cache-friendly order.

3.2 Simplify combat normalization and validation

• Keep combat normalization at dnd.combat_turns.normalize.v1. Continue to display-normalize and registry-canonicalize actors, canonicalize evidence, order records chronologically, and collapse exact duplicates by actor, turn kind, and complete valid evidence. Delete action/target/prose normalization and warning codes.
• Keep the invariant policy identifier unchanged. It checks canonical actor display, canonical and chronological evidence, stable event ordering, and absence of duplicate event identities; it performs no nested-action checks.
• Keep the shape policy identifier unchanged and validate only list presence, nonblank actor, allowed turn kind, and nonempty source references.
• Keep the source-relatedness policy identifier unchanged and compare only the actor against cited transcript material. Remove declaration-token heuristics and their now-unused helpers. Keep source-reference validation unchanged if its semantics did not change.
• Simplify merge/clone behavior to copy only retained values and source refs. Preserve stable ordering and defensive ownership.

3.3 Update owners and tests

• Update codec, private schema, extractor, normalizer, validators, merge, registration, pipeline integration, CLI fixtures, and examples to the minimal shape. Rewrite or delete rich-schema fixtures rather than retaining them as compatibility cases.
• Delete tests whose sole policy was round/action/summary handling. Add or rewrite focused tests for strict schemas, enum validation, mapping and source-ID assignment, actor canonicalization through the redesigned NPC registry, chronology, exact duplicate collapse, and invariant validation.
• Update docs/integrations/dnd-combat-turn-artifacts.md as the canonical redesigned v1 contract and update current internal module/LLM documentation for the implemented behavior.

Stage 3 completion criteria

• No production combat type, prompt, schema, policy, normalizer, fixture, or current documentation refers to rounds, actions, summaries, declarations, targets, resolutions, or action categories.
• The immediate-resolution shared asset has no remaining registration or file.
• Focused combat and pipeline tests pass, and go test ./... passes.

Stage 4: Complete The Repository-Wide Cutover

4.1 Audit contract identity and stale surface area

• Search code, embedded assets, tests, examples, and current documentation for all removed field names and unintended v2 schema/prompt names. Removed fields may remain only in historical ADR or roadmap context; active schema, prompt, fixture, and current-behavior surfaces must describe the minimal v1 contract.
• Verify the three artifact registrations still bind their original kinds and exact redesigned Go types through extract, merge, normalize, codec, and validators.
• Verify all prompt manifests still point at their v1 private schemas, stable reference material precedes module-variable material, and transcript content remains last. Do not add a change-detector test for prompt message count, prose, shared-prefix content, or prefix length.
• Verify pre-redesign development checkpoints naturally miss through changed prompt/schema content and generated-dependency fingerprints. Do not add migration or compatibility handling for disposable pre-release state.

4.2 Exercise representative assembled behavior

• Keep unit case matrices at their owning schema, validator, normalizer, and registry boundaries. At the assembled-pipeline boundary, retain only representative tests proving each redesigned lane is registered, strict JSON/schema failures are attributed to the correct validator, semantic failures are attributed to the correct domain validator, and generated NPC output is accepted by later spell/combat stages.
• Cover an ordered multi-step run in which NPC extraction produces the minimal artifact and spell/combat consume its name projection. Assert that downstream event source references point only to the current transcript and not to NPC registry evidence.
• Verify CLI logical output and manifests retain the v1 schema/prompt identities, full NPC registry provenance, and the relevant component fingerprints without exposing prompt bodies or reference payloads.

4.3 Finish documentation and lifecycle state

• Reconcile the three integration documents, docs/internal/modules.md, and docs/internal/llm.md with the final code. Correct any stale validator-order descriptions while doing so; current docs must describe the registered order, not preserve an older generic ordering example.
• Mark ADR-0009 Accepted and the feature roadmap Implemented when the code, tests, and current-behavior documentation all land. Remove the completed item from docs/roadmap/future.md; retain the feature roadmap and ADR as design rationale unless the repository's normal roadmap-retirement practice calls for moving the completed roadmap later.
• Record any human-reviewed rich/minimal model evaluation separately from CI results. Evaluation may motivate later prompt tuning but does not reopen the approved durable minimal field set within this implementation.

4.4 Final verification

Run, in order:

gofmt -w <changed Go files>
go test ./...
go vet ./...
go build ./cmd/notarius
go test -race ./internal/modules/dnd/...
git diff --check

If the repository-wide race command exposes an unrelated, pre-existing failure, document it with the narrower affected package result; do not weaken or skip ordinary tests for the changed D&D packages.

Stage 4 completion criteria

• All three lanes use only their minimal v1 contracts from model response through durable output.
• No compatibility shim, removed-field policy, unintended v2 asset, or stale current-behavior documentation remains.
• Generated NPC references are name-only for LLM input and remain full-fidelity for durable provenance.
• Repository tests, vet, build, race checks for the changed domain, and diff hygiene checks pass.

Open Questions

None. The feature roadmap, ADR, and fixed decisions above define the cutover, pre-release schema, evidence, projection, validation, testing, and documentation policies needed to implement each stage without further product decisions.