diff --git a/docs/adr/0009-minimal-evidence-grounded-extraction-artifacts.md b/docs/adr/0009-minimal-evidence-grounded-extraction-artifacts.md new file mode 100644 index 0000000..6302bea --- /dev/null +++ b/docs/adr/0009-minimal-evidence-grounded-extraction-artifacts.md @@ -0,0 +1,98 @@ +# ADR-0009: Prefer minimal evidence-grounded extraction artifacts + +**Status:** Proposed +**Date:** 2026-07-22 + +## Context + +Notarius is intended to extract structured facts from source material. Several +early D&D artifacts grew to include descriptive prose, inferred relationships, +immediate outcomes, summaries, and other enrichment alongside the facts that +identify an event or entity. Those fields make one model call responsible for +both extraction and synthesis. + +In practice, the richer contracts have produced overlapping or weakly grounded +fields and have made structurally valid, semantically coherent output harder for +cost-effective smaller models. They also increase prompt size, validation and +normalization policy, durable schema surface, downstream coupling, and the +number of claims whose provenance must be evaluated. + +The application needs a consistent rule for deciding what belongs in an +extractor before redesigning the current D&D spell, NPC, and combat-turn +contracts or adding new artifact families. + +## Decision + +An extraction module answers one narrowly stated question and returns the +smallest durable structured artifact that usefully answers it. + +Every model-produced field in an extraction artifact must: + +- be necessary to answer the extractor's stated question or serve a known + downstream consumer; +- represent a fact or bounded classification that can be supported directly by + cited source ranges; +- remain independently meaningful without model-generated explanatory prose; + and +- justify the additional prompt, schema, validation, normalization, and + compatibility surface it creates. + +Source references are required provenance for extracted records. Auxiliary +references may disambiguate identities or canonical names, but they do not +establish source facts and are not copied into evidence. + +Extraction artifacts do not include narrative summaries, general analysis, +speculative enrichment, inferred biography or relationships, or redundant +free-text descriptions by default. When such output has a demonstrated use, it +belongs in an explicitly named extraction, classification, enrichment, or +analysis module with its own contract and evidence policy. + +Occurrence-level facts are not forced into entity-level attributes. A fact +that can change between encounters, such as an NPC's role in a scene, belongs +on an occurrence artifact rather than as one scalar property of a normalized +NPC registry entry. + +Deterministic mapping and normalization may assign application-owned +identifiers, canonicalize known catalog values, order and deduplicate evidence, +and collapse records under an explicit identity rule. They must not manufacture +removed descriptive fields or synthesize missing claims to satisfy an older +contract. + +This is a default design rule, not a prohibition on rich artifacts. A richer +field is appropriate when its consumer, evidence semantics, and ownership are +explicit. + +## Alternatives considered + +- Keep rich schemas and improve prompts or use larger models. This retains + potentially convenient prose but does not resolve overlapping field + responsibilities, weak provenance, higher cost, or unnecessary downstream + coupling. +- Make enrichment fields optional. This reduces rejection pressure but leaves + ambiguous artifact semantics and inconsistent records, and many strict + structured-output providers still require nullable placeholders. +- Keep minimal private LLM schemas while preserving rich durable artifacts. + Deterministic code would have to invent, default, or separately derive the + missing fields, hiding synthesis behind the extraction boundary. +- Use one broad session-analysis module. This reduces the number of lanes but + couples unrelated facts, schemas, retries, evaluation, and downstream + consumers into one model call. + +## Consequences + +Extraction prompts and response schemas become smaller, more focused, and more +suitable for lower-cost models. Artifacts carry fewer unsupported claims, and +their evidence and validation policies become easier to explain and evaluate. +Independent extractors can evolve, retry, and be consumed without requiring +unrelated enrichment. + +Some descriptive convenience fields will disappear from primary artifacts. +Consumers that genuinely need them may require a separate module and explicit +pipeline step. Entity registries may no longer resolve aliases or relationships +unless a dedicated, evidence-grounded capability supplies them. + +Removing durable fields is a schema compatibility change. Each affected +artifact requires an explicit version and reference policy; private prompt +changes alone are insufficient. Current-behavior integration and internal +documentation must change with implementation, while the roadmap owns the +proposed contract until then. diff --git a/docs/roadmap/future.md b/docs/roadmap/future.md index 411af44..1d94e4e 100644 --- a/docs/roadmap/future.md +++ b/docs/roadmap/future.md @@ -7,6 +7,19 @@ not as committed release dates. ## Near-Term D&D Pipeline +### Simplify D&D Extraction Contracts + +- Adopt the proposed principle in + [ADR-0009](../adr/0009-minimal-evidence-grounded-extraction-artifacts.md) + that extractors should return the smallest useful evidence-grounded facts + needed to answer one narrow question. +- Redesign the spell, NPC, and combat-turn durable contracts and prompts as + described in + [Minimal D&D Extraction Contracts](minimal-dnd-extraction-contracts.md). +- Replace the unpublished v1 schema shapes in place rather than adding optional + or synthetic compatibility fields, and evaluate the reduced outputs against + human-reviewed transcripts after cutover. + ### Evaluate Spell Extraction And Normalization - Evaluate ordinary extraction retries and the completed normalization path diff --git a/docs/roadmap/implementation.md b/docs/roadmap/implementation.md index 235e98a..d7d408e 100644 --- a/docs/roadmap/implementation.md +++ b/docs/roadmap/implementation.md @@ -1,174 +1,373 @@ -# D&D Validation-Boundary Alignment Implementation Plan +# Minimal D&D Extraction Contracts Implementation Plan -Status: Proposed. +**Status:** Ready for implementation -Implement this plan in order. It repairs the immediate combat extraction -failure, makes D&D validation diagnostics consistently domain-owned, and then -aligns the spell and NPC private response boundaries with the same policy. +## Objective -Do not change durable artifact schemas, artifact kinds, public Go types, -framework retry behavior, checkpoint formats, or the scene chunker. Existing -v1 private-schema identities may be corrected in place because Notarius has not -been run in production. Changed prompt and schema content will invalidate -development checkpoints through existing fingerprints. +Implement the durable contract redesign defined by +[Minimal D&D Extraction Contracts](minimal-dnd-extraction-contracts.md) and +[ADR-0009](../adr/0009-minimal-evidence-grounded-extraction-artifacts.md). +The result is a coordinated in-place redesign of NPC, spell-cast, and combat-turn +extraction. Each lane must emit only its narrow, evidence-grounded facts; no +removed rich-schema field may survive as an optional field, placeholder, compatibility +shim, or deterministic synthesis. -## Cross-Stage Decisions +This plan deliberately starts with NPCs because the NPC registry is a generated +reference consumed by spell and combat extraction. Spell and combat then change +independently, followed by one repository-wide contract and documentation +pass. -Private LLM schemas own the transport envelope: required fields, JSON types, -nullability, array/object shape, and unknown-field rejection. Deterministic -domain validators own semantic rules: supported enum values, nonblank values, -required non-empty collections, positive and resolvable source units, -catalog/identity policy, and normalized invariants. +## Fixed Decisions And Guardrails -For typed D&D artifacts, `generic/valid_json` remains first as a representation -sanity check. All deterministic domain validators that can reject a candidate -run next. `generic/valid_json_schema` runs after them as a durable-schema -backstop, followed by warning-only relatedness validators. This order gives -expected candidate failures bounded domain reason codes while retaining a final -check that typed encoding conforms to the durable contract. +The implementing agent must treat these as decisions, not open design choices: -Do not expose raw JSON Schema errors or candidate values through the generic -validator. Do not add tests that require particular words or phrases to remain -in prompt prose. +- Keep the artifact kinds `dnd/npc-list`, `dnd/spell-list`, and + `dnd/combat-turn-list`, their module keys and capabilities, their media types, + and their prompt IDs stable. +- Retain exactly the existing v1 schema keys, IDs, names, versions, filenames, + prompt IDs, and prompt versions listed in the feature roadmap. Change their + unpublished shapes and content in place; do not harmonize the private spell + identity as part of this work. +- Do not add v2 assets, runtime version negotiation, migration code, dual-write + behavior, a second module registration, or compatibility fixtures for the + superseded pre-release shapes. Existing fixtures should be rewritten or + deleted according to whether they still protect current behavior. +- Require the top-level list field and every record field. A list may be empty, + but every returned record must have at least one source reference. Use strict + JSON objects with unknown fields rejected and no nullable or optional legacy + fields. +- Keep private source references limited to `start_unit_id` and `end_unit_id`. + Mapping assigns the current source document ID; campaign references and an + NPC registry never become event evidence. +- Keep the shared prompt ordering policy: stable shared instructions first, + stable campaign and generated references next, module task material after + those references, and the chunk-variable transcript last. +- Preserve the existing production validator-chain order and the rule that a + configured chain is authoritative. Remove or simplify validators; do not + silently reorder chains. +- Preserve the existing NPC ID derivation algorithm and comparison/display + normalization for canonical names. Retain existing semantic-policy + identifiers as well as schema and prompt versions; pre-redesign development + state is disposable. Unchanged canonical names must not receive new IDs. +- Use deterministic code only for display normalization, known catalog or NPC + canonicalization, application-owned IDs, evidence canonicalization, ordering, + and exact duplicate collapse. Do not introduce fuzzy matching or inferred + enrichment. +- Follow the testing policy: protect schemas and behavior at their owning + boundaries, delete obsolete tests, and avoid tests that snapshot prompt prose + or detect exact shared-prefix length. No live or paid model call is a stage + completion requirement. +- Each stage must leave the repository compiling and `go test ./...` passing. + Update all affected fixtures and callers within the stage that changes a + public Go type; do not leave an intentionally broken intermediate commit. -## Stage 1: Repair Combat Extraction +## Stage 1: Cut Over The NPC Contract And Name Projection -### Changes +This stage establishes the redesigned generated reference on which later stages +depend. -- Update the combat extraction instructions to enumerate the complete allowed - values: - - `turn_kind`: `turn`, `reaction`, `legendary_action`, `lair_action`, - `other`; - - action `category`: `attack`, `spell`, `movement`, `item`, - `ability_check`, `saving_throw`, `condition`, `other`. -- Keep the private combat schema structurally permissive and the durable schema - strict. Do not restore enum, minimum, `minLength`, or `minItems` constraints - to the private schema. -- Reorder the combat extraction default chain to: - `generic/valid_json`, combat shape, combat source references, - `generic/valid_json_schema`, combat source relatedness. -- Reorder the combat normalization default chain to: - `generic/valid_json`, combat shape, normalized invariants, combat source - references, `generic/valid_json_schema`, combat source relatedness. -- Preserve configured validator overrides as authoritative; only production - default composition changes. +### 1.1 Replace the durable and private data shapes -### Tests +- Reduce `dnd.NPC` in `internal/modules/dnd/types.go` to exactly `ID`, `Name`, + and `SourceRefs`. Delete `NPCRelationship` and every alias, description, and + relationship field or helper that becomes unused. +- Rewrite the existing strict durable schema `dnd_npcs.v1.json` with required + top-level `npcs`, and records containing only required `id`, `name`, and + `source_refs`. Keep the NPC codec at schema version `v1`, name + `notarius_dnd_npcs_v1`, and its existing schema ID and media type. +- Rewrite the strict private schema `dnd_npcs_llm.v1.json`. Its records contain + only required `name` and model-facing source ranges. Keep the extractor schema + identity and prompt ID/version unchanged. +- Reduce the private response DTO and mapping accordingly. Preserve candidate + data at the mapping boundary so deterministic validators, rather than mapping + defaults, reject blank names or invalid ranges. Assign the current source ID + and derive the NPC ID in application code. +- Rewrite the NPC task and instruction assets to ask only for individually + identifiable NPC names and supporting transcript ranges. Explicitly exclude + anonymous groups, generic roles, invented labels, descriptions, aliases, and + relationships. Keep transcript material last in the manifest. -- Add an assembled combat pipeline case whose raw LLM response contains an - unsupported turn kind or action category. Exhausted retries must produce a - non-fatal `invalid_combat_turn_shape` rejection owned by the combat shape - validator, not `json_schema_invalid` or a framework error. -- Retain coverage that a later valid retry succeeds and discarded-attempt - warnings/rejections do not become durable. -- Update registrar contract tests to assert the new extraction and - normalization order. -- Rely on behavioral enum/schema tests and prompt fingerprint coverage; do not - add prompt-word change-detector tests. +### 1.2 Simplify NPC identity, normalization, and validation -### Completion Check +- Keep the identity policy at `dnd.npcs.identity.v1`. Retain display + normalization, comparison keys, ID syntax, and deterministic ID derivation. + Validate nonblank canonical names, exact ID/name agreement, duplicate + canonical names, and duplicate IDs. Delete alias validation and alias-specific + issue locations/codes. +- Keep NPC normalization at `dnd.npcs.normalize.v1`. Normalize the retained + display name, derive its ID, canonicalize and deduplicate source references, + consolidate records only by the canonical-name comparison key, preserve the + first stable record, and union exact evidence. Remove alias promotion, + relationship merge/rewrites, and their warning codes. +- Keep the existing NPC shape and source-relatedness policy identifiers. The + shape validator checks only list presence, nonblank `id` and `name`, and + nonempty source references; source-relatedness grounds a record only through + its retained name. The existing source-ref validator remains the owner of + range validity. +- Keep the default validation order domain-first, then JSON Schema, then the + advisory relatedness validator. Simplify only the validator implementations + and selections whose owned behavior changed. -Run `go test ./internal/modules/dnd/extract/combatturns -./internal/modules/dnd/validate/combatturns/... ./internal/modules/dnd/register -./internal/modules/integration` and `git diff --check`. +### 1.3 Separate durable registry provenance from model input -## Stage 2: Make D&D Validator Ordering Consistent +- Continue to canonicalize and retain the complete redesigned NPC artifact for + registry validation, cache identity, manifests, and provenance. + `Registry.Digest()` remains the digest of that complete canonical artifact + when bound and remains empty when no registry was supplied. +- Build a second structural JSON projection for model grounding with the exact + shape `{"npcs":[{"name":"Mira Thorn"}]}` in normalized registry order. It + contains names only: no IDs, source references, origin URI, aliases, or other + provenance. Generate it with typed values and `json.Marshal`, not string + concatenation. The empty projection is exactly `{"npcs":[]}`. +- Add an explicit projection digest accessor. The digest is SHA-256 over the + exact projected bytes, including for an absent or empty registry, and + `PromptInput().Digest` must equal it. Keep full-artifact and projection digests + distinct even when their content happens to coincide. +- Index `Registry.Lookup` by canonical name only. Retain immutable return values + and defensive copies. Remove alias indexing and alias-aware comments. +- In spell extraction, combat extraction, and combat normalization, use the NPC + name-projection digest for the component-local checkpoint fingerprint because + it exactly describes the names that affect those operations. Keep the full + registry digest and count in manifest metadata for provenance. Framework-owned + generated-reference fingerprints may still invalidate a run when any upstream + artifact byte changes; do not broaden this stage into a framework cache + redesign. -### Changes +### 1.4 Update owners, consumers, and tests -- Move `generic/valid_json_schema` behind all rejecting domain validators in - every spell and NPC extraction and normalization default chain: - - spell extraction/normalization: shape, catalog, source references, schema, - then source relatedness; - - NPC extraction: shape, source references, schema, then source relatedness; - - NPC normalization: shape, identity, source references, schema, then source - relatedness. -- Keep `generic/valid_json` first and warning-only source relatedness last. -- Do not change validator implementations, reason codes, warning promotion, - retry counts, or user-provided chain order. -- Document the default-chain policy in the pipeline/module internals: domain - validators diagnose expected semantic failures and the generic schema - validator is the final rejecting representation backstop. +- Update NPC codec, schema, extractor, identity, validator, normalizer, registry, + registration, pipeline-integration, and CLI fixtures to the new shape. Adapt + spell and combat tests that construct `dnd.NPC` values so the repository + remains buildable, but do not change their own artifact contracts yet. +- Add or rewrite focused tests for strict schema acceptance/rejection, + codec round trips, unchanged ID derivation for known names, name-only + consolidation, evidence union, registry immutability, and canonical-name-only + lookup. Rewrite the existing fixture for the minimal current contract; do not + retain the superseded rich fixture solely to test backwards incompatibility. +- Test the projection as a data contract: it contains only ordered names, + equivalent normalized registries produce identical bytes and digest, + evidence/ID-only changes do not change the projection digest, and name/order + changes do. This is not authorization to snapshot assembled prompt prose or + prefix lengths. +- Update `docs/integrations/dnd-npc-artifacts.md` to own the redesigned v1 + durable schema. Update `docs/internal/modules.md` and + `docs/internal/llm.md` only for current NPC behavior that lands in this stage. -### Tests +### Stage 1 completion criteria -- Update production registrar tests for every affected chain. -- Add one representative spell and NPC assembled rejection proving that a - domain-invalid but encodable candidate is attributed to the owning domain - validator rather than the generic schema validator. Do not duplicate each - validator package's existing case matrix at integration level. -- Confirm explicitly configured validator chains retain their exact configured - order. +- NPC durable and private schemas expose no removed enrichment fields. +- The generated NPC prompt input contains names only while manifests retain + full registry provenance. +- Spell and combat consumers accept generated or external registries in the + redesigned v1 shape; rich pre-redesign registry JSON fails strict decoding. +- Focused NPC, spell-wiring, combat-wiring, integration, and CLI tests pass, and + `go test ./...` passes. -### Completion Check +## Stage 2: Cut Over Spell-Cast Extraction -Run `go test ./internal/modules/dnd/register ./internal/modules/integration -./internal/framework/pipeline` and `git diff --check`. +### 2.1 Replace the spell contract and prompt -## Stage 3: Align Spell and NPC Private Response Boundaries +- Reduce `dnd.SpellCast` to exactly `Caster`, `Spell`, and `SourceRefs`. +- Rewrite `dnd_spells.v1.json` in place. Keep the durable codec at version v1, + name `notarius_dnd_spells_v1`, and its existing schema ID. Its strict record + contains only required `caster`, `spell`, and `source_refs`. +- Rewrite `dnd_spells_llm.v1.json` with the same logical fields and model-facing + ranges. Keep the existing private key, ID, name, prompt version, and schema + path unchanged. +- Reduce the private DTO, canonicalization, and mapping to the retained fields. + Continue assigning current source IDs, stable-ordering candidates by evidence, + and preserving semantically invalid candidates for deterministic validation. +- Rewrite spell task and instruction assets around the narrow casting-event + boundary. Retain the spell catalog and name-only NPC projection as + disambiguation inputs, never evidence. Remove effect, outcome, and narrative + duties and remove the `common-dnd-immediate-resolution.md` message from the + spell manifest and spell asset registration. Do not delete the shared file in + this stage because combat still uses it. -### Changes +### 2.2 Simplify spell policies -- Revise the existing v1 spell and NPC private schemas in place: - - retain required fields, JSON types, array/object structure, - `additionalProperties: false`, and omission of framework-assigned fields; - - remove `minLength`, `minItems`, and positive-number `minimum` constraints; - - leave the durable spell and NPC schemas unchanged. -- Replace `shared.UnitRef` in the private spell and NPC response DTOs with - integer candidates so zero and negative unit IDs survive decoding and mapping - into `source.SourceRef` for deterministic source validation. -- Update canonicalization and ordering helpers to operate on candidate integers - without repairing invalid values. Valid positive IDs retain current output, - ordering, and exact-deduplication behavior; invalid ranges remain available - to validators. -- Keep malformed JSON, missing/unknown fields, wrong JSON types, and - non-integer source IDs as LLM-boundary errors. -- Update LLM/module internals and the spell/NPC integration contracts to state - the structural-private/semantic-validator ownership boundary. +- Keep the spell shape policy identifier unchanged and validate only list + presence, nonblank caster/spell, and nonempty source references. +- Keep catalog validation, source-reference validation, and source-relatedness + behavior and policy identities unchanged where their actual semantics are + already limited to spell name, caster, and evidence. +- Simplify normalization and duplicate comparison to caster, catalog-canonical + spell name, and the complete valid evidence set. Remove all prose selection, + copying, fixture fields, and assertions. Preserve catalog fingerprints and + metadata. +- Keep the exact inclusion rule from the feature roadmap: an actual casting or + unambiguous declared attempt is included; mentions, plans, rules discussion, + and catalog matches are not. -### Tests +### 2.3 Update owners and tests -- For each private schema, prove structurally valid candidates with blank - strings, empty required collections, and nonpositive unit IDs pass the - private schema, while missing fields, unknown fields, and wrong JSON types do - not. -- Through raw-JSON LLM fakes, prove semantic values survive decoding and mapping - without repair. -- Add representative assembled cases showing: - - blank or empty spell/NPC fields are rejected by the appropriate shape - validator; - - nonpositive or nonexistent unit IDs are rejected by the appropriate source - validator; and - - exhausted validation retries remain non-fatal rejected outputs. -- Preserve existing valid mapping, source-position ordering, deduplication, - catalog, identity, checkpoint-fingerprint, and durable codec tests. +- Update spell codec, private schema, extractor, validators, normalizer, merge + and registration tests, pipeline integration, CLI output fixtures, and any + maintained examples to the minimal shape. Rewrite or delete rich-schema + fixtures rather than retaining them as compatibility cases. +- Replace tests of effects and narrative descriptions with focused tests of the + retained contract: strict unknown-field rejection, codec round trips, current + source-ID assignment, catalog canonicalization, evidence ordering, duplicate + collapse, and NPC name-projection wiring. +- Update `docs/integrations/dnd-spell-artifacts.md` as the canonical redesigned + v1 contract and update current internal module/LLM documentation for the + smaller prompt and response. Do not duplicate the spell catalog contract + owned by its existing integration document. -### Completion Check +### Stage 2 completion criteria -Run `go test ./internal/modules/dnd/extract/spells -./internal/modules/dnd/extract/npcs ./internal/modules/dnd/validate/spells/... -./internal/modules/dnd/validate/npcs/... ./internal/modules/integration` and -`git diff --check`. +- No production spell type, schema, prompt, validator, normalizer, fixture, or + documentation contract refers to effect or narrative description. +- Spell prompt/schema identities remain exactly their existing v1 values. +- Focused spell and pipeline tests pass, and `go test ./...` passes. -## Final Verification +## Stage 3: Cut Over Combat-Turn Extraction -Run: +### 3.1 Replace the combat contract and prompt -```text -git diff --check +- Reduce `dnd.CombatTurn` to exactly `Actor`, `TurnKind`, and `SourceRefs`. + Delete `CombatAction`, `CombatActionCategory`, their constants, and all + now-unused helpers. Retain the existing five `CombatTurnKind` values. +- Rewrite `dnd_combat_turns.v1.json` in place. Keep the durable codec at version + v1, name `notarius_dnd_combat_turns_v1`, and its existing schema ID. Use a + strict required record with `actor`, `turn_kind`, and `source_refs` only. +- Simplify the durable codec to direct strict encoding/decoding if its custom + wire representation exists only to distinguish removed nullable fields. + Preserve presence semantics for the top-level list and strict unknown-field + rejection. +- Rewrite `dnd_combat_turns_llm.v1.json` while retaining its private schema and + prompt identities, and reduce the DTO/mapping to actor, turn kind, and + model-facing ranges. Keep the semantic mapping policy identifier unchanged. +- Rewrite combat task/instruction assets around detecting ordered turns and + discrete interrupting events. Remove round, action, target, declaration, + resolution, outcome, and summary duties. Remove the shared immediate- + resolution message from the combat manifest. +- Once both spell and combat manifests no longer use it, delete + `common-dnd-immediate-resolution.md` and its shared and module asset + registrations. Retain all other shared evidence, identity, reference, NPC, + and transcript assets in their cache-friendly order. + +### 3.2 Simplify combat normalization and validation + +- Keep combat normalization at `dnd.combat_turns.normalize.v1`. Continue to + display-normalize and registry-canonicalize actors, canonicalize evidence, + order records chronologically, and collapse exact duplicates by actor, turn + kind, and complete valid evidence. Delete action/target/prose normalization + and warning codes. +- Keep the invariant policy identifier unchanged. It checks canonical actor + display, canonical and chronological evidence, stable event ordering, and + absence of duplicate event identities; it performs no nested-action checks. +- Keep the shape policy identifier unchanged and validate only list presence, + nonblank actor, allowed turn kind, and nonempty source references. +- Keep the source-relatedness policy identifier unchanged and compare only the + actor against cited transcript material. Remove declaration-token heuristics + and their now-unused helpers. Keep source-reference validation unchanged if + its semantics did not change. +- Simplify merge/clone behavior to copy only retained values and source refs. + Preserve stable ordering and defensive ownership. + +### 3.3 Update owners and tests + +- Update codec, private schema, extractor, normalizer, validators, merge, + registration, pipeline integration, CLI fixtures, and examples to the minimal + shape. Rewrite or delete rich-schema fixtures rather than retaining them as + compatibility cases. +- Delete tests whose sole policy was round/action/summary handling. Add or + rewrite focused tests for strict schemas, enum validation, mapping and + source-ID assignment, actor canonicalization through the redesigned NPC + registry, chronology, exact duplicate collapse, and invariant validation. +- Update `docs/integrations/dnd-combat-turn-artifacts.md` as the canonical + redesigned v1 contract and update current internal module/LLM documentation + for the implemented behavior. + +### Stage 3 completion criteria + +- No production combat type, prompt, schema, policy, normalizer, fixture, or + current documentation refers to rounds, actions, summaries, declarations, + targets, resolutions, or action categories. +- The immediate-resolution shared asset has no remaining registration or file. +- Focused combat and pipeline tests pass, and `go test ./...` passes. + +## Stage 4: Complete The Repository-Wide Cutover + +### 4.1 Audit contract identity and stale surface area + +- Search code, embedded assets, tests, examples, and current documentation for + all removed field names and unintended v2 schema/prompt names. Removed fields + may remain only in historical ADR or roadmap context; active schema, prompt, + fixture, and current-behavior surfaces must describe the minimal v1 contract. +- Verify the three artifact registrations still bind their original kinds and + exact redesigned Go types through extract, merge, normalize, codec, and + validators. +- Verify all prompt manifests still point at their v1 private schemas, stable + reference material precedes module-variable material, and transcript content + remains last. Do not add a change-detector test for prompt message count, + prose, shared-prefix content, or prefix length. +- Verify pre-redesign development checkpoints naturally miss through changed + prompt/schema content and generated-dependency fingerprints. Do not add + migration or compatibility handling for disposable pre-release state. + +### 4.2 Exercise representative assembled behavior + +- Keep unit case matrices at their owning schema, validator, normalizer, and + registry boundaries. At the assembled-pipeline boundary, retain only + representative tests proving each redesigned lane is registered, strict + JSON/schema failures are attributed to the correct validator, semantic + failures are attributed to the correct domain validator, and generated NPC + output is accepted by later spell/combat stages. +- Cover an ordered multi-step run in which NPC extraction produces the minimal + artifact and spell/combat consume its name projection. Assert that downstream + event source references point only to the current transcript and not to NPC + registry evidence. +- Verify CLI logical output and manifests retain the v1 schema/prompt identities, + full NPC registry provenance, and the relevant component fingerprints without + exposing prompt bodies or reference payloads. + +### 4.3 Finish documentation and lifecycle state + +- Reconcile the three integration documents, `docs/internal/modules.md`, and + `docs/internal/llm.md` with the final code. Correct any stale validator-order + descriptions while doing so; current docs must describe the registered order, + not preserve an older generic ordering example. +- Mark ADR-0009 `Accepted` and the feature roadmap `Implemented` when the code, + tests, and current-behavior documentation all land. Remove the completed item + from `docs/roadmap/future.md`; retain the feature roadmap and ADR as design + rationale unless the repository's normal roadmap-retirement practice calls + for moving the completed roadmap later. +- Record any human-reviewed rich/minimal model evaluation separately from CI + results. Evaluation may motivate later prompt tuning but does not reopen the + approved durable minimal field set within this implementation. + +### 4.4 Final verification + +Run, in order: + +```sh +gofmt -w go test ./... go vet ./... go build ./cmd/notarius -go test -race ./internal/modules/dnd/... ./internal/framework/pipeline ./internal/cli ./internal/modules/integration +go test -race ./internal/modules/dnd/... +git diff --check ``` -Review current-behavior documentation for stale statements that private spell, -NPC, or combat schemas own semantic validation. Confirm the scene schema and -prompt remain unchanged: their enumerations are explicitly communicated and -scene-plan construction has a distinct structural mapping boundary. +If the repository-wide race command exposes an unrelated, pre-existing failure, +document it with the narrower affected package result; do not weaken or skip +ordinary tests for the changed D&D packages. + +### Stage 4 completion criteria + +- All three lanes use only their minimal v1 contracts from model response + through durable output. +- No compatibility shim, removed-field policy, unintended v2 asset, or stale + current-behavior documentation remains. +- Generated NPC references are name-only for LLM input and remain full-fidelity + for durable provenance. +- Repository tests, vet, build, race checks for the changed domain, and diff + hygiene checks pass. ## Open Questions -None. The stages above define the validation ownership, default ordering, -compatibility policy, diagnostic behavior, and test boundaries required for -implementation. +None. The feature roadmap, ADR, and fixed decisions above define the cutover, +pre-release schema, evidence, projection, validation, testing, and documentation +policies needed to implement each stage without further product decisions. diff --git a/docs/roadmap/minimal-dnd-extraction-contracts.md b/docs/roadmap/minimal-dnd-extraction-contracts.md new file mode 100644 index 0000000..9b9ea2e --- /dev/null +++ b/docs/roadmap/minimal-dnd-extraction-contracts.md @@ -0,0 +1,220 @@ +# Minimal D&D Extraction Contracts + +**Status:** Proposed + +## Intent + +Redesign the D&D spell, NPC, and combat-turn artifacts around the principle in +[ADR-0009](../adr/0009-minimal-evidence-grounded-extraction-artifacts.md): +each extractor should answer one narrow question with the smallest useful set +of source-grounded fields. + +The redesign favors extraction precision, evidence quality, valid-output rate, +smaller-model reliability, and lower prompt and response cost over descriptive +richness. It removes synthesis responsibilities rather than preserving obsolete +fields as optional, nullable, empty, or application-generated placeholders. + +## Goals + +- Make every model-produced field necessary to the artifact's core question. +- Require direct transcript evidence for every extracted record. +- Remove overlapping prose, inferred enrichment, and nested structures without + a demonstrated consumer. +- Keep catalog and identity references as disambiguation aids rather than + evidence. +- Preserve deterministic canonicalization, evidence ordering, exact + deduplication, identity assignment, and bounded domain diagnostics where + those responsibilities still apply. +- Reduce downstream prompt material to the fields a consumer actually needs. +- Keep the unpublished v1 identities while replacing their pre-release shapes + in place. + +## Non-Goals + +- Generating session narrative, rules analysis, biographies, relationship + graphs, encounter summaries, or prose descriptions. +- Preserving removed fields for source compatibility through empty strings, + nullable values, or synthetic defaults. +- Adding fuzzy entity resolution, LLM-assisted enrichment, or a general schema + migration framework. +- Treating campaign references, catalogs, or earlier artifacts as evidence that + an event occurred in the current transcript. +- Combining the three D&D artifact families into one model call. + +## Shared Contract Policy + +All three artifacts remain ordered lists. Each record contains at least one +source reference, and the complete reference collection supports every +model-produced field in that record. Source IDs continue to be assigned by the +application from the current input; the model returns only source-unit ranges. + +Private LLM schemas remain strict about their transport envelope: required +fields, JSON types, object and array shape, nullability where applicable, and +unknown-field rejection. Deterministic validators continue to own semantic +rules such as nonblank identities, catalog membership, enum membership, +positive and resolvable source units, and canonical normalized invariants. + +Prompts retain the shared D&D evidence, identity, reference, NPC-grounding, and +transcript assets that remain relevant. Module-specific task and instruction +assets must delete duties associated with removed fields. The existing +cache-friendly ordering keeps stable shared and reference material before the +chunk-variable transcript. + +## Spell Cast + +The spell extractor answers: + +> Which spell was cast, by which in-world caster, and where is that event +> established in the source? + +The artifact kind remains `dnd/spell-list`. Its durable schema remains v1. +Each spell-cast record contains exactly: + +- `caster`: required nonblank in-world display identity; +- `spell`: required nonblank canonical or catalog-resolvable spell name; and +- `source_refs`: one or more current-source evidence ranges. + +The model-facing response contains the same fields except for application-owned +`source_id` values within references. The current `effect` and +`narrative_description` fields are removed from the private response, public Go +type, durable schema, codec, validators, normalizer, fixtures, and integration +contract. + +The inclusion boundary remains an actual casting event or an unambiguously +declared casting attempt, not a spell mention, hypothetical plan, rules +discussion, or catalog match. The spell catalog helps recognize and canonicalize +the name but never establishes that a cast occurred. + +Normalization continues to canonicalize spell names, canonicalize evidence, +and collapse exact duplicate events using caster, canonical spell name, and +complete valid evidence. It performs no prose selection or merging. + +## NPC Registry + +The NPC extractor answers: + +> Which individually identifiable non-player characters are established in the +> source, and where is each identity established? + +The artifact kind remains `dnd/npc-list`. Its durable schema remains v1. +Each durable NPC record contains exactly: + +- `id`: deterministic application-assigned identity derived under the NPC + identity policy; +- `name`: required nonblank source-supported display identity; and +- `source_refs`: one or more evidence ranges supporting that identity. + +The private model response omits `id` and reference `source_id` values. A +`name` may be a proper name or a stable, individually distinguishing title or +alias supported by the transcript. The extractor does not invent descriptive +labels for anonymous creatures, crowds, or generic roles. + +The current `aliases`, `description`, and `relationships` fields are removed +from the private response, public Go type, durable schema, codec, validators, +normalizer, registry, fixtures, and integration contract. Normalization +consolidates only identities supported by the retained name policy and unions +exact evidence; it does not infer alias equivalence or relationships. + +Spell and combat consumers receive a prompt projection containing only the +canonical NPC names needed for identity grounding. Application-owned NPC IDs +remain available to deterministic registry and normalization code but are not +sent to a model that cannot return or otherwise consume them. NPC source +references remain provenance in the durable registry and are not included as +current-transcript evidence or copied into downstream event artifacts. + +Encounter context is deliberately not a scalar NPC registry field. Dialogue, +combat alignment, presence, or third-party mention can vary across occurrences. +If a demonstrated consumer needs that information, add a separate ordered +NPC-occurrence artifact whose records contain `name`, a small mutually +exclusive context enum, and `source_refs`. A candidate starting vocabulary is +`dialogue`, `combat_ally`, `combat_opponent`, `noncombat_presence`, `mentioned`, +and `other`; its exact semantics require a separate feature decision. + +## Combat Event + +The combat extractor answers: + +> Which in-world participant took a turn or discrete interrupting combat +> event, what kind of event was it, and where is it established in the source? + +The existing `dnd/combat-turn-list` artifact kind and v1 durable-schema identity +remain. Each record contains exactly: + +- `actor`: required nonblank in-world display identity; +- `turn_kind`: one of `turn`, `reaction`, `legendary_action`, `lair_action`, or + `other`; and +- `source_refs`: one or more current-source evidence ranges. + +The current `round`, `actions`, and `summary` fields, including nested action +categories, declarations, targets, and resolutions, are removed from the +private response, public Go types, durable schema, codec, validators, +normalizer, fixtures, and integration contract. + +Normalization continues to display-normalize and registry-canonicalize actors, +canonicalize evidence, order events by source position, and collapse exact +duplicates using actor, turn kind, and complete valid evidence. It no longer +normalizes targets, declarations, summaries, or resolutions. + +If action-level facts later have a demonstrated consumer, they belong in a +separate combat-action artifact rather than restoring a nested synthesis +contract to combat-turn detection. Spell casts and future item events remain +owned by their dedicated artifact lanes. + +## Pre-Release Schema Policy + +Notarius and these contracts are pre-release. The existing v1 artifacts and +private model-response schemas have not been published as compatibility +contracts, so their shapes change in place. The implementation does not retain +the rich pre-redesign shape, add v2 assets, migrate old output, support multiple +versions, or preserve old fixtures solely for compatibility testing. + +Artifact kinds, schema keys and IDs, schema names and versions, prompt IDs and +versions, module keys, capabilities, and media types all remain unchanged. +Changing prompt and schema content invalidates the relevant content-addressed +development state; any remaining pre-redesign local output or cache is +disposable and may be regenerated. + +| Lane | Durable v1 schema | Private model-response v1 schema | Prompt ID | +| --- | --- | --- | --- | +| Spell cast | ID `notarius.dnd.spells`, name `notarius_dnd_spells_v1` | key `dnd_spells`, ID `notarius.dnd.spells`, name `notarius_dnd_spells_v1` | `dnd.spells` | +| NPC registry | ID `notarius.dnd.npcs`, name `notarius_dnd_npcs_v1` | key `dnd_npcs_llm`, ID `notarius.dnd.npcs.llm`, name `notarius_dnd_npcs_llm_v1` | `dnd.npcs` | +| Combat event | ID `notarius.dnd.combat_turns`, name `notarius_dnd_combat_turns_v1` | key `dnd_combat_turns_llm`, ID `notarius.dnd.combat_turns.llm`, name `notarius_dnd_combat_turns_llm_v1` | `dnd.combat_turns` | + +## Quality And Evaluation + +The implemented contract should protect the remaining meaningful risks: + +- private schemas accept only the new structural envelopes; +- durable codecs strictly round-trip the redesigned contracts; +- domain validators own blank identities, enum and catalog membership, and + invalid evidence; +- normalizers preserve source-grounded values while applying only their stated + deterministic transformations; +- generated NPC references expose the minimal identity projection and never + become event evidence; and +- representative assembled pipelines attribute retries and rejections to the + owning domain boundary. + +Post-cutover model evaluation should compare the rich-schema baseline and the +minimal-schema result on a small human-reviewed transcript set using: + +- event/entity precision and recall; +- caster, NPC, and combat-actor attribution accuracy; +- source-range validity and evidence sufficiency; +- catalog and enum accuracy; +- structurally valid completion rate and exhausted-retry rate; +- unsupported-claim rate; and +- input/output tokens, latency, and model cost. + +The evaluation exists to identify prompt or model-quality follow-up work, not +to gate the approved minimal contract on live-provider behavior. Human review +and live model calls are evaluation aids rather than deterministic CI gates. + +## Documentation Ownership + +The spell, NPC, and combat integration documents are the canonical owners of +their redesigned durable schemas once implemented. Internal LLM and module +documents own the corresponding current prompt, DTO, validator, normalizer, and +NPC prompt-projection behavior. This roadmap owns the desired end state and +policy; [the implementation plan](implementation.md) owns sequencing and +completion criteria until the cutover is complete.