notarius/docs/roadmap/implementation.md

D&D Combat-Turn Extraction Implementation Plan

Status: Ready for implementation.

Implement this plan in order. The feature policy and target state are defined in D&D Combat-Turn Extraction; this document owns implementation sequencing and concrete technical decisions.

Do not introduce a DAG, prior-run discovery, encounter state engine, scene-based provider-call suppression, LLM-backed validator, or semantic reconciliation. Preserve the fixed input -> chunk -> extract -> merge -> normalize -> output architecture and the existing framework retry, warning, rejection, checkpoint, debug, and output contracts.

Cross-Stage Decisions

Production Identities

Use these exact identities:

• artifact kind: dnd/combat-turn-list;
• extractor key and normalizer key: dnd/combat-turns;
• extractor capability: dnd.combat_turns;
• prompt ID: dnd.combat_turns, version v1;
• private response-schema key: dnd_combat_turns_llm;
• private schema ID: notarius.dnd.combat_turns.llm;
• private schema name: notarius_dnd_combat_turns_llm_v1;
• durable schema ID: notarius.dnd.combat_turns;
• durable schema name: notarius_dnd_combat_turns_v1;
• durable schema version: v1; and
• durable media type: application/json.

The private response schema omits source_id; mapping code assigns the current source identity. No combat-turn ID is added in v1.

Canonical Go And JSON Shape

Add these types to the canonical D&D model:

type CombatTurnList struct {
    CombatTurns []CombatTurn `json:"combat_turns"`
}

type CombatTurn struct {
    Actor      string             `json:"actor"`
    TurnKind   CombatTurnKind     `json:"turn_kind"`
    Round      *int               `json:"round"`
    Actions    []CombatAction     `json:"actions"`
    Summary    string             `json:"summary"`
    SourceRefs []source.SourceRef `json:"source_refs"`
}

type CombatAction struct {
    Category    CombatActionCategory `json:"category"`
    Declaration string               `json:"declaration"`
    Targets     []string             `json:"targets"`
    Resolution  *string              `json:"resolution"`
}

Define string types and only these lowercase durable values:

• CombatTurnKind: turn, reaction, legendary_action, lair_action, other;
• CombatActionCategory: attack, spell, movement, item, ability_check, saving_throw, condition, other.

All object fields and the top-level combat_turns array are required. combat_turns may be empty. Every turn requires a non-empty actor, supported turn kind, at least one action, non-empty summary, and at least one source reference. round is either null or a positive integer. Every action requires a supported category, non-empty declaration, and present targets array; targets may be empty but may not contain empty strings. resolution is either null or non-empty after trimming. Unknown fields are rejected at every object level.

Strict raw decoding must distinguish a missing required round or resolution key from an explicitly present JSON null, even though the accepted Go value represents null with a nil pointer. Use presence-aware wire decoding or raw key validation at codec and private-response boundaries; do not silently turn a missing nullable field into an accepted null.

One turn-level source-reference collection collectively supports the actor, kind, round, actions, summary, targets, and resolutions. Do not add per-action references in v1.

Ordering And Duplicate Identity

Source-document slice position is chronology. Numeric unit IDs are identifiers only. An item's earliest evidence position is the minimum valid start-unit index across all its ranges. Stable ordering puts records with valid evidence in ascending earliest-position order, preserves encounter order for ties, and puts records without valid evidence after valid records without reordering them.

The normalizer may collapse two records only when all of these match:

• actor under npcs/identity.ComparisonKey;
• exact turn_kind;
• both round values, including nil versus a value; and
• the complete, non-empty, canonically ordered set of exact source references.

Every reference in the duplicate key must pass source.ValidateRef against the current document. Invalid or empty evidence never participates in duplicate collapse. Actions and prose do not participate in identity because two model representations of the same cited turn should collapse; retain the complete first record without merging prose or action lists.

NPC Registry Contract And Prompt Reuse

Promote NPC registry preparation from the spell extractor into internal/modules/dnd/npcs/registry. It owns:

• slot name npcs and maximum size 1,048,576 bytes;
• exact-one-item and application/json validation when bound;
• approved NPC codec decoding, whole-registry identity validation, canonical re-encoding, semantic SHA-256 digest, and content-safe bounded failures;
• the exact unbound prompt value {"npcs":[]};
• immutable accessors for bound state, a defensive NPC list, canonical bytes, digest, count, and prompt input; and
• exact canonical-name/alias lookup using the NPC identity comparison policy.

Registry source references may identify another session and are not checked against the current source. The resolver never returns content, names, aliases, paths, or raw decoder errors in failures. Identity failures use issue codes and record/alias indexes under the established 20-issue, 128-rune, and 4,096-byte limits.

Move the generic bounded diagnostic helper from dnd/npcs/diagnostics to dnd/shared/diagnostics, updating existing NPC consumers. This is an internal package move with no diagnostic-policy change.

Add common-dnd-npcs.md to the shared D&D prompt assets. Its generic policy allows exact canonical participant names and alias recognition while stating that registry content is context, not event evidence. Both spell and combat prompts use this exact fragment immediately after the existing shared campaign reference message. Remove the spell-owned NPC fragment and include the shared fragment in both prompt hashes. This intentionally changes the spell prompt fingerprint once but must not change its request inputs, metadata shape, registry semantics, or output contract.

Both combat extraction and normalization declare the same optional registry slot. A bound registry contributes manifest metadata npc_registry_digest and npc_count and a local checkpoint fingerprint npc_registry. An absent slot contributes neither metadata fields nor that fingerprint.

Validation And Diagnostics

Add deterministic validators with these exact identities:

Validator key	Result reason	Policy fingerprint
extract/dnd/combat-turns/shape	rejection invalid_combat_turn_shape	dnd.combat_turns.validator.shape.v1
extract/dnd/combat-turns/source_refs	rejection invalid_combat_turn_source_refs	dnd.combat_turns.validator.source_refs.v1
extract/dnd/combat-turns/source_relatedness	warning combat_turn_not_near_source	dnd.combat_turns.validator.source_relatedness.v1
normalize/dnd/combat-turns/invariants	rejection invalid_combat_turn_normalization	dnd.combat_turns.validator.normalized.v1

Shape owns required arrays, enum membership, nullable fields, positive rounds, and non-empty strings. Source-reference validation owns source identity, unit existence, and range order through source.ValidateRef. Later validators defer when shape is invalid; relatedness also defers when any source range is invalid.

Relatedness combines the turn's cited units once in document order, removing overlap. The actor is related when its NPC comparison key is a substring of the comparison-normalized cited text. For each action declaration, split its comparison-normalized form on runes that are not Unicode letters or digits, retain tokens of at least four Unicode code points, and require at least one retained token to occur as a complete cited-text token. No retained token means the declaration is unrelated. Emit at most one approved warning per turn, listing whether the actor and which action indexes were not related. Do not check targets deterministically.

The normalize invariant validator requires display-normalized strings, comparison-unique targets within each action, canonical exact source-reference order without duplicates, chronological record order for valid evidence, and absence of the duplicate identity defined above. It does not require every actor or target to exist in the optional NPC registry.

All rejection and warning messages use the shared diagnostic helper: at most 20 displayed issues, at most 128 Unicode code points per displayed value, valid UTF-8, at most 4,096 bytes, Go quoting for control characters, and the exact total omitted count. Validators use strict empty-options decoders and provide one local policy checkpoint fingerprint with the value in the table.

Normalization Policy

Use normalization policy dnd.combat_turns.normalize.v1 and these warning reason codes:

• combat_turn_fields_normalized;
• combat_actor_canonicalized;
• combat_target_canonicalized;
• source_references_normalized;
• combat_turns_reordered; and
• duplicate_combat_turn_collapsed.

Deep-clone all nested slices and pointers. Normalize actor, summary, declarations, targets, and non-null resolutions by collapsing Unicode whitespace through npcs/identity.NormalizeDisplay. Preserve nil resolution. Remove comparison-duplicate targets while retaining the first display value and target order. Do not deduplicate or reorder actions.

When a registry is bound, rewrite an actor or target only if its comparison key matches exactly one validated canonical name or alias. Preserve unmatched values. Registry validation makes ambiguous lookup impossible at preparation; do not guess or perform fuzzy matching.

Sort every source-reference list by exact source_id, start_unit_id, and end_unit_id, then remove exact duplicates. Stable-sort records by the chronology rule before duplicate detection. Collapse duplicates in that order and retain the first record unchanged after its per-record normalization.

Warning scopes use merged input indexes such as combat_turns[3], even after sorting or collapse. Emit one bounded warning for each affected record or collapsed group; group warnings identify the retained and removed input indexes. Only warnings from a validator-approved attempt become durable, under existing framework policy.

The normalizer exposes manifest metadata normalization_policy, identity_policy, and optional registry digest/count. Its checkpoint fingerprints are normalization_policy, identity_policy, and optional npc_registry; bump the normalization policy when any transformation, ordering, or duplicate rule changes.

Testing Rules

Follow docs/policy/testing.md. Tests are offline and deterministic. Use a fake structured LLM only at the completion boundary. Protect durable shapes, domain invariants, preparation failures, reference sensitivity, registration, and one representative assembled workflow.

Do not add prompt-prose change detectors or tests requiring particular words or phrases. Prompt tests may verify registration, message/input wiring, prompt and schema identities, stable shared-fragment use, and absence of raw content from metadata or diagnostics. Human evaluation owns semantic output quality.

Stage 1: Shared NPC Registry And Prompt Grounding

Goal

Create one reusable, content-safe NPC registry boundary before adding a second consumer, while preserving spell behavior.

Changes

• Add the domain registry resolver and immutable lookup described above; migrate spell extraction from its private resolver without changing the spell extractor's public module contract.
• Relocate bounded D&D diagnostics to the shared domain package and update all NPC imports and tests.
• Add the shared NPC prompt fragment, move the spell prompt to it, place it immediately after shared campaign references, and remove the spell-owned fragment.
• Keep raw reference provenance in framework identity independently from the semantic registry fingerprint. Preserve the exact empty prompt input and all existing content-safe error behavior.
• Update the internal overview, module, and LLM documentation in this stage so the implemented registry owner and shared prompt-fragment ownership remain accurate; do not claim combat extraction exists yet.

Tests

• Move resolver contract tests to the domain package: absent, valid, formatted- equivalent, malformed, unknown-field, invalid-ID, collision, media type, item count, byte limit, defensive copies, lookup, digest, and bounded content-free diagnostics.
• Retain spell-level tests only for spell request wiring, metadata/fingerprint behavior, and no-regression output; remove duplicated resolver cases.
• Verify the shared prompt asset registers for spells and hashes the shared fragment without asserting its prose.

Completion Check

Run go test ./internal/modules/dnd/npcs/... ./internal/modules/dnd/extract/spells ./internal/modules/dnd/validate/npcs/... and git diff --check.

Stage 2: Combat Domain Contract And Codec

Goal

Establish the typed artifact and durable serialization boundary without making the combat module selectable.

Changes

• Add the canonical combat types, enums, constants, and artifact kind using the exact shape and values above.
• Add internal/modules/dnd/codec/combatturns, following the existing candidate versus approved codec boundary:
  • strict single-value JSON with unknown-field rejection;
  • candidate encode/decode preserving validator-visible invalid values;
  • approved encode/decode enforcing structural validity only;
  • defensive schema and metadata values; and
  • metadata containing combat_turn_count only.
• Add the durable v1 JSON Schema with required nullable fields, enum values, source-reference shape, array rules, and additionalProperties: false.
• Add docs/integrations/dnd-combat-turn-artifacts.md, describing only the implemented artifact and codec at this stage.

Tests

• Test approved round trips and candidate preservation for every nullable, enum, required-array, required-string, target, and source-reference boundary.
• Test malformed, trailing, unknown-field, and invalid structural input without snapshotting complete errors.
• Test defensive schema/metadata copies, nil versus present-empty arrays, and exact codec identity.
• Keep one compact durable v1 fixture for intentional compatibility coverage.

Completion Check

Run go test ./internal/modules/dnd/codec/combatturns and git diff --check.

Stage 3: Combat Extractor And Extraction Validators

Goal

Implement package-complete LLM extraction and deterministic candidate validation without production composition.

Changes

• Add internal/modules/dnd/extract/combatturns with the exact module, prompt, schema, capability, artifact, and reference identities above. It accepts no options and requires chunks and source.transcript. The prompt manifest is dnd.combat_turns.yaml, uses default profile gemini-2-flash, JSON Schema validation, and zero provider-side repair attempts so pipeline retries remain the only extraction retry policy.
• Declare existing optional players, party, glossary, and deprecated roster slots plus the optional structured NPC registry slot. Reuse shared D&D prompt inputs and the shared NPC prompt fragment.
• Embed a private response schema matching the durable shape except that source ranges omit source_id. Require present arrays and nullable round/resolution values exactly as in the durable contract.
• Prompt for the inclusion, exclusion, chronology, identity, round restraint, and immediate-resolution policy in the feature roadmap. Instruct the model that campaign and NPC references disambiguate identities but are never combat evidence.
• Preserve malformed response values for validators. Canonicalize and exactly deduplicate source ranges, assign the current source ID, and stable-sort records by earliest valid source-document position. Do not merge records, canonicalize NPC names deterministically, or skip any chunk in the extractor.
• Expose prompt and private-schema manifest metadata plus optional registry digest/count. Fingerprint local names prompt, response_schema, mapping_policy with value dnd.combat_turns.extract_mapping.v1, and optional npc_registry.
• Implement shape, source-reference, and source-relatedness validators with the exact contracts and policies above.

Tests

• Through a fake LLM, test request identity, profile/session propagation, chunk-scoped source input, campaign and NPC inputs, response mapping, source assignment, non-monotonic-unit chronology, invalid-candidate preservation, cancellation, and contextual provider failures.
• Test prompt/schema registration, required input wiring, metadata and fingerprint sensitivity, defensive copies, and content redaction without asserting prompt prose.
• Test each validator's meaningful approval, rejection, deferral, and warning categories, including Unicode comparison, overlapping evidence, declaration tokenization, bounded diagnostics, strict options, and typed registration.
• Verify that a bound NPC registry affects grounding metadata and checkpoint identity but never supplies combat source references.

Completion Check

Run go test ./internal/modules/dnd/extract/combatturns ./internal/modules/dnd/validate/combatturns/... and git diff --check.

Stage 4: Combat Normalization And Final Invariants

Goal

Implement conservative identity canonicalization, chronological ordering, and exact-evidence duplicate collapse.

Changes

• Add internal/modules/dnd/normalize/combatturns with key dnd/combat-turns, requirements merged, capability normalized, no options, and the optional NPC registry slot.
• Resolve and retain the immutable registry during preparation. Runtime uses the prepared view; it does not parse references repeatedly.
• Apply the exact per-record, registry, source-reference, chronology, duplicate, warning, metadata, and fingerprint policies above. Never mutate or retain caller-owned slices, pointers, registry data, or source data.
• Add the normalized-invariants validator. It remains deterministic, accepts no references of its own, and defers malformed shape or invalid source evidence to their owning validators.

Tests

• Use table-driven cases for whitespace, nullable resolution, target deduplication, actor/target registry matches, unmatched names, reference normalization, non-monotonic source IDs, stable ties, and warning scopes.
• Cover duplicate collapse and non-collapse for every identity dimension, especially invalid evidence, distinct ranges, different turn kinds, and nil versus numbered rounds.
• Prove actions/prose come only from the first retained record, merged input is immutable, and all nested output storage is independent.
• Test absent and bound registries, preparation failures, metadata, fingerprints, policy sensitivity, cancellation, strict options, and module registration.
• Test normalized-invariant rejection for each owned invariant without duplicating shape and source-validator case matrices.

Completion Check

Run go test ./internal/modules/dnd/normalize/combatturns ./internal/modules/dnd/validate/combatturns/... and git diff --check.

Stage 5: Production Composition, Sequential Workflow, And Documentation

Goal

Make the complete combat lane selectable, verify the assembled workflow, and document current behavior in canonical locations.

Changes

• Extend the D&D registrar with the combat codec, extractor and prompt assets, typed append-order merger, combat normalizer, typed no-op normalizer, all four validators, and typed always-accept/always-reject variants.
• Add append behavior that preserves present-empty versus nil output and chunk record order before normalization.
• Register the extract default chain in this order: generic/valid_json, generic/valid_json_schema, combat shape, combat source references, combat source relatedness.
• Register the normalize default chain in this order: generic/valid_json, generic/valid_json_schema, combat shape, normalized invariants, combat source references, combat source relatedness.
• Do not add a merge validator chain or change framework defaults.
• Add examples/dnd-combat-turns.config.yml and examples/dnd-npc-combat-sequential.config.yml. Use version 3, safe relative state paths, explicit checkpoint enabled: false, a generic chunker, combat extraction with retries: 2, and combat normalization. Keep the dynamic NPC output unbound in the static sequential example.
• Demonstrate runtime binding to both stage-local slots with selectors combat.extract.npcs=<npc-run>/lanes/npcs.json and combat.normalize.npcs=<npc-run>/lanes/npcs.json.
• Update canonical current-behavior documentation:
  • Configuration owns module/validator catalogs, reference slots, limits, default chains, and maintained examples;
  • CLI owns the explicit two-selector invocation syntax;
  • Operations owns the independent NPC-then-combat workflow and state sensitivity;
  • the combat integration contract owns durable fields, enum values, evidence, normalization, warnings, and manifest metadata;
  • the NPC integration contract notes combat as a consumer without redefining combat fields;
  • JSON output links the new lane payload; and
  • internal overview, module, and LLM docs describe concrete packages, preparation, prompt reuse, and fingerprints.
• At completion, mark this plan and the feature roadmap complete. Remove the implemented combat proposal from future.md while retaining scene-routing, narrative, generic deduplication, and other unimplemented work.

Tests

• Extend registrar tests for production keys, typed variants, prompt assets, default chain order, nil dependencies, and duplicate registration behavior.
• Add config/example resolution coverage for stage-local NPC bindings, capabilities, codec compatibility, strict options, and invalid validator placement.
• Add one assembled pipeline integration covering extract, retry-capable validation, append merge, registry-backed normalization, final validation, JSON output, warnings, manifest metadata, and checkpoint fingerprints using a fake LLM.
• Add one sequential integration that materializes normalized NPC output as both combat references and verifies canonical actor/target output, reference provenance, and that NPC source ranges never become combat evidence.
• Add preparation-boundary coverage showing malformed or oversized NPC input fails before checkpoint construction or pipeline execution. Do not duplicate the shared resolver's complete malformed-input matrix.
• Validate maintained examples and documentation links through existing test mechanisms.

Completion Check

Run the final verification suite.

Final Verification

Run:

git diff --check
go test ./...
go vet ./...
go build ./cmd/notarius
go test -race ./internal/modules/dnd/... ./internal/framework/pipeline \
  ./internal/cli ./internal/modules/integration

Review the final diff for:

• D&D or provider behavior leaking into generic framework packages;
• accidental scene-based call suppression or automatic pipeline composition;
• mutation or aliasing of typed artifacts, schemas, references, metadata, or fingerprints;
• transcript, registry, prompt, schema, path, or decoder content leaking into errors, manifests, fingerprints, or redacted summaries;
• prompt-prose change-detector tests, redundant cross-layer cases, or exact LLM output goldens;
• current-behavior documentation claiming features before the implementing stage lands; and
• unrelated worktree changes.

Open Questions

None. Artifact fields and enums, nullable behavior, chronology, duplicate identity, reference reuse, prompt composition, validator placement, normalization, checkpoint semantics, documentation ownership, and stage boundaries are fixed by this plan.