diff --git a/docs/roadmap/future.md b/docs/roadmap/future.md index c830d35..ae97ec5 100644 --- a/docs/roadmap/future.md +++ b/docs/roadmap/future.md @@ -22,6 +22,48 @@ not as committed release dates. spell, combat, interaction, and scene-description lanes after real-world use. Add more complex chunking only in response to demonstrated failures. +### Canonical Entity Registries And Occurrences + +- Establish a consistent module and artifact naming convention that makes the + two entity-extraction roles explicit. Prefer `dnd/-registry` for a + module that identifies canonical nouns and `dnd/-occurrences` for a + downstream module that consumes that registry and extracts transcript-backed + evidence about those nouns. Apply the convention coherently to NPCs, + locations, and items rather than preserving the current mixture of bare + plurals, `interactions`, `occurrences`, and `events` names. +- Treat this as a contract-level change rather than a prompt-asset rename. A + full feature roadmap should inventory and update module IDs, artifact kinds + and schema identities where applicable, pipeline dependencies, generated + references, checkpoints, configuration examples, tests, and documentation. + Because Notarius remains pre-release, prefer a clean migration over aliases + or compatibility shims unless the repository has acquired a concrete need + for them before implementation. +- Define the registry contract around stable, canonicalizable entity names. + In particular, tighten location extraction to include explicitly named + in-world places, such as `Waterdeep` or `The Yawning Portal`, while excluding + generic, temporary, relative, or purely descriptive references such as `the + room`, `the bar`, `the hallway`, or `upstairs`. Determine eligibility by + whether the transcript establishes a stable name or unique designation, not + by capitalization alone, because transcript capitalization is unreliable. +- Keep registry identification separate from occurrence evidence. A downstream + occurrence extractor may resolve a generic reference such as `the bar` to an + existing named registry entry when transcript context supports that + coreference, but it must not create a new canonical entity merely from the + generic reference. The occurrence itself must remain independently supported + by transcript ranges; registry membership is grounding, not event evidence. +- Define equivalent eligibility and coreference rules for NPC and item + registries so future entity types follow one architectural pattern while + retaining their domain-specific occurrence kinds. The full roadmap should + decide how unnamed but uniquely identifiable NPCs and items are represented, + and whether an item registry is introduced before or alongside renaming the + existing item-event lane. +- Evaluate the renamed and tightened contracts against representative + transcripts. Review false-positive registry entries, missed named entities, + coreference accuracy, unsupported occurrence links, and the behavior of + cost-effective smaller models. Keep these semantic changes separate from the + prompt-asset consolidation so regressions can be attributed to the correct + work set. + ## Shared Normalization And Quality Work ### Generic LLM-Assisted Deduplication diff --git a/docs/roadmap/implementation.md b/docs/roadmap/implementation.md new file mode 100644 index 0000000..76ed5c2 --- /dev/null +++ b/docs/roadmap/implementation.md @@ -0,0 +1,618 @@ +# D&D Prompt Asset Cleanup Implementation Plan + +## Objective + +Implement the target state defined in +[D&D Prompt Asset Cleanup](prompts.md): standardize D&D prompt asset names, +clarify transcript scopes, consolidate fixed module instructions, remove prompt +prose duplication, normalize the two outlying private response-schema +identities, and document the resulting authoring conventions. + +This plan changes internal LLM-facing assets and their owning registration code. +It does not change durable artifact contracts, public prompt IDs, module IDs, +pipeline configuration, reference bindings, profile selection, or extraction +policy. In particular, the proposed registry/occurrence module renaming and the +proper-name-only location policy in [Future Work](future.md) are not part of +this implementation. + +The stages are ordered dependencies. Complete and validate each stage before +starting the next. Each stage is sized for one gpt-5.6-terra implementation +prompt. + +## Implementation Rules + +Apply these rules throughout all stages: + +- Read `docs/policy/architecture.md`, `docs/policy/documentation.md`, and + `docs/policy/testing.md` before making changes. +- Keep the root `assets` Go package content-only. Do not add prompt composition, + schema selection, registration, or D&D policy to that package. +- Preserve every public PromptKit prompt `id`, prompt version, default profile, + input name, input requirement, output format, validation mode, and repair + setting unless a stage below expressly changes private schema identity. +- Preserve the message orders and cache boundaries specified in `prompts.md`. + Consolidating fixed prose is not authorization to move variable material or + family-specific policy into the system message. +- Treat prompt prose as an owned contract, not as text to concatenate + mechanically. Merge `task.md` and `instructions.md` by editing the result into + one coherent `instructions.md`, removing duplicated transitions and rules. +- Begin with minimal, non-repetitive prose. Do not retain or reintroduce a + duplicated instruction merely because repetition might help a model. That + decision requires later observational evidence. +- Preserve all lane-specific domain semantics identified in `prompts.md`, even + when removing generic output, evidence, reference, or schema restatements. +- Keep all shared filenames prefixed with `common-dnd-`. +- Do not add compatibility aliases for retired private asset names or private + schema identities. Notarius is pre-release, and one-time prompt fingerprint, + checkpoint, and provider-cache invalidation is accepted. +- Keep default tests deterministic, offline, and credential-free. Do not call a + live LLM during implementation or make live model quality an automated gate. +- Do not add exact prompt snapshots, exact shared-prefix lengths, or tests that + enumerate every private asset merely to detect change. Retain tests of actual + preparation, input placement, cache boundaries, schema registration, and the + byte-identical extraction prefix. + +## Stage 1: Establish Shared System And Transcript-Scope Assets + +### Goal + +Give universal instructions and each form of transcript material one correct +shared owner, then migrate every current D&D prompt to the appropriate scope in +one coherent change. + +### Required changes + +1. Revise `assets/dnd/shared/prompts/common-dnd-system.md` so it contains all + and only behavior applicable to scene chunking, every extraction lane, and + both entity normalizers: + + - the D&D transcript-processing role; + - reliance only on supplied inputs; + - awareness of transcription errors, repetition, incomplete sentences, and + misheard proper nouns; and + - exactly one schema-conforming JSON object with no explanatory prose. + + Refer to supplied inputs generally. Do not limit the statement to transcript + and reference inputs, because grounded extractors and normalizers receive + other material. Do not place rendered inputs, transcript-evidence mechanics, + identity policy, reference policy, registry policy, or reconciliation policy + in the system message. + +2. Revise + `assets/dnd/shared/prompts/common-dnd-extraction-evidence.md` to own only + extraction evidence and source-range policy. Retain transcript-only evidence, + integer range endpoints, omission of `source_id`, and narrow non-contiguous + ranges. Remove the universal JSON-only/output-schema sentences now owned by + the system message. Avoid repeating detailed campaign-reference or registry + semantics owned by their input fragments. + +3. Replace `assets/dnd/shared/prompts/common-dnd-transcript.md` with: + + - `common-dnd-transcript-full.md`, which renders the complete ordered session + for scene chunking and does not imply that the complete session is one + scene; + - `common-dnd-transcript-chunk.md`, which renders one extraction chunk, + limits reporting and inference to that chunk, and states that unit IDs + retain source-wide meaning; and + - `common-dnd-transcript-windows.md`, which renders selected normalization + evidence windows and states that they may be incomplete, non-contiguous, + or overlapping and that absence outside them is not evidence. + + Each file must render `{{ input "transcript" }}` exactly once. Scope files + describe material completeness and permissible use; do not move scene, + extraction-evidence, or reconciliation policy into them. + +4. Update `internal/modules/dnd/shared/assets.go` so `sharedPromptPaths` exposes + the three new names and no longer exposes `common-dnd-transcript.md`. + +5. Update every owning `promptAssetManifest` and current prompt YAML definition + to select exactly one transcript-scope asset: + + | Scope asset | Prompt IDs | + | --- | --- | + | `common-dnd-transcript-full.md` | `dnd.scenes` | + | `common-dnd-transcript-chunk.md` | `dnd.spells`, `dnd.npcs`, `dnd.combat_turns`, `dnd.enemy_events`, `dnd.item_events`, `dnd.npc_interactions`, `dnd.scene_descriptions`, `dnd.locations`, `dnd.location_occurrences` | + | `common-dnd-transcript-windows.md` | `dnd.npcs.normalize`, `dnd.locations.normalize` | + + Preserve the existing message position and cache control while changing the + referenced scope filename. Extraction prompts must continue to share the + same rendered prefix through the chunk transcript. + +6. Update focused prompt-asset and shared-asset tests. A focused assertion that + an owning prompt family selects its required transcript scope is warranted; + do not replace it with exact prose snapshots or a full shared-file inventory. + Expand `internal/modules/dnd/register/prompt_cache_test.go` to include the + location and location-occurrence extraction prompts in its byte-identical + prefix check while retaining the existing other extraction lanes. + +### Acceptance criteria + +- The old `common-dnd-transcript.md` is absent and unreferenced. +- Every D&D prompt prepares with exactly the transcript semantics supplied by + its caller. +- The shared system message is byte-identical across all D&D prompts and + contains no variable input. +- The extraction-evidence fragment no longer repeats the universal output rule. +- Extraction prompts retain a byte-identical rendered prefix through campaign + references and the chunk transcript when given identical inputs. +- No public prompt, module, durable artifact, or pipeline contract changes. + +### Validation + +```sh +go test ./internal/modules/dnd/shared/... ./internal/modules/dnd/... +git diff --check +``` + +## Stage 2: Standardize Prompt And Context Filenames + +### Goal + +Make prompt definitions and variable context assets predictable without changing +their rendered content or message order. + +### Required changes + +1. Rename every prompt-ID-derived YAML file below to `prompt.yaml` and update its + owning `promptAssetManifest` entry to mount `prompt.yaml` from + `prompts/prompt.yaml`: + + - `assets/dnd/scenes/prompts/` + - `assets/dnd/spells/prompts/` + - `assets/dnd/npcs/extract/prompts/` + - `assets/dnd/npcs/normalize/prompts/` + - `assets/dnd/combat-turns/prompts/` + - `assets/dnd/enemy-events/prompts/` + - `assets/dnd/item-events/prompts/` + - `assets/dnd/npc-interactions/prompts/` + - `assets/dnd/scene-descriptions/prompts/` + - `assets/dnd/locations/extract/prompts/` + - `assets/dnd/locations/normalize/prompts/` + - `assets/dnd/location-occurrences/prompts/` + + Leave the YAML `id` fields unchanged. PromptKit continues to distinguish the + definitions through each manifest's existing `ModuleDir`. + +2. Apply these context-fragment renames and update both YAML `content_file` + references and manifest `ModuleFiles` entries: + + - `assets/dnd/spells/prompts/catalog.md` to `spell-catalog.md`; + - `assets/dnd/enemy-events/prompts/grounding.md` to + `combat-grounding.md`; and + - `assets/dnd/location-occurrences/prompts/locations.md` to + `location-registry.md`. + + Do not edit their prompt prose in this mechanical stage. + +3. Update tests that intentionally exercise the production asset layout: + + - In `internal/modules/dnd/register/register_test.go`, retain a registrar- + boundary check for one canonical `prompt.yaml` per maintained prompt, but + remove lists of every task, instruction, context, and shared fragment. + - In `internal/cli/production_contract_test.go`, remove private prompt-file + inventory assertions already owned by the D&D registrar and module prompt- + preparation tests. Retain its CLI composition and public catalog contract + checks. + - Update module-local tests only where a renamed file is intentionally part + of the manifest/preparation behavior. + - Do not rewrite generic `internal/framework/llm` or + `internal/framework/promptfs` fixtures merely because they use arbitrary + noncanonical filenames; those tests prove generic filesystem behavior, not + the D&D authoring convention. + +4. Search for and remove all production references to the retired filenames. + Do not preserve copies or aliases. + +### Acceptance criteria + +- Every D&D prompt subtree contains `prompt.yaml` and no prompt-ID-derived YAML + definition. +- The three renamed context files have descriptive names and render in their + original message positions. +- Every module manifest mounts the canonical names and still produces a prompt + fingerprint. +- Production-composition tests protect registration behavior without + enumerating every private content file. +- All prompts prepare through PromptKit with unchanged IDs, versions, inputs, + profiles, output modes, and cache controls. + +### Validation + +```sh +go test ./internal/modules/dnd/... ./internal/cli/... +git diff --check +``` + +## Stage 3: Consolidate Scene And Entity-Normalization Instructions + +### Goal + +Adopt the single `instructions.md` convention for scene chunking, scene +description, NPC normalization, and location normalization while preserving +their distinct message-order contracts. + +### Required changes + +1. For scene chunking, merge `task.md` and `instructions.md` into one coherent + `instructions.md` and delete `task.md`. Retain: + + - the definition of a coherent scene and meaningful boundary conditions; + - reasons to split and reasons not to split; + - complete ordered transcript coverage with no gaps or overlaps; and + - inclusive, positive source-unit endpoints. + + Remove the generic JSON-only rule now owned by the system prompt and field- + exclusivity prose already enforced by the private response schema. In + `prompt.yaml`, place the combined instructions after campaign references and + before the full transcript. It has no cache control; references and the final + full transcript retain ephemeral cache control. + +2. For scene descriptions, merge `task.md` and `instructions.md` into one + `instructions.md` and delete `task.md`. Retain the one-accepted-chunk scope, + all `kind` definitions and precedence, and title and summary semantics. + Remove schema field-list exclusions and universal JSON-only language. Keep + the combined final instructions message ephemeral. + +3. For both NPC and location normalizers, rename `task.md` to + `instructions.md`, update `prompt.yaml` and the manifest, and edit the prose + against `common-dnd-entity-reconciliation.md`: + + - NPC instructions own same-individual alias judgment and canonical display- + name preferences. + - Location instructions own same-place identity judgment, separation of + nested or uncertain places, and canonical display-name selection. + - The shared reconciliation fragment alone owns opaque keys, valid group + membership, canonical-member constraints, invention prohibitions, and + omission of uncertain groups. + + Preserve the normalizer order: system, module instructions, cacheable shared + reconciliation policy, candidates, cacheable transcript windows. + +4. Update module-local prompt preparation tests to assert meaningful relative + ordering, input isolation, and cache boundaries. Remove exact message-count + assertions whose only purpose was to preserve the former task/instruction + split. + +### Acceptance criteria + +- These four prompt directories each contain one module-owned + `instructions.md` and no `task.md`. +- Scene and normalizer domain policy remains complete but does not repeat + universal system, response-schema, or shared reconciliation rules. +- Scene chunking still receives the full transcript last; normalizers still + receive candidates before transcript windows. +- Prompt preparation and existing scene/normalization behavior tests pass. + +### Validation + +```sh +go test ./internal/modules/dnd/chunk/scenes ./internal/modules/dnd/extract/scenedescriptions ./internal/modules/dnd/normalize/npcs ./internal/modules/dnd/normalize/locations +git diff --check +``` + +## Stage 4: Consolidate NPC, Location, And Item-Event Instructions + +### Goal + +Consolidate three comparatively self-contained extraction prompts and remove +their shared or schema-enforced repetition without changing lane semantics. + +### Required changes + +1. Merge each prompt's `task.md` and `instructions.md` into its existing + `instructions.md`, delete `task.md`, and update `prompt.yaml` and the owning + manifest for: + + - NPC extraction; + - location extraction; and + - item-event extraction. + +2. NPC instructions must retain individually identifiable non-PC inclusion, + proper-name/stable-title/alias eligibility, all existing exclusion + categories, observed display spelling, and the prohibition on invented + anonymous labels. Remove generic narrow-range, reference-not-evidence, + schema-field, and lore-output restatements already owned elsewhere. + +3. Location instructions must retain the current implemented location policy, + including the current allowance for a generic label used for a specific + place, supported place categories, alias/nesting handling, and domain + exclusions. Do **not** implement the future proper-name-only registry policy + in this stage. Remove only generic evidence, reference, source-range, and + schema-shape repetition. + +4. Item-event instructions must retain event eligibility, all five kind + definitions, currency treatment, quantity semantics, holder transitions, + collective `party` behavior, transfer restrictions, and the distinction + between discovery and acquisition. Remove only generic evidence-range, + output-envelope, and schema-field restatements. + +5. Keep each consolidated extraction instruction as the final ephemeral + message. Update focused tests to protect preparation, inputs, output schema, + and cache placement without asserting the retired two-message structure. + +### Acceptance criteria + +- The three prompt directories contain no `task.md`. +- No domain inclusion, exclusion, event-kind, currency, holder, or location + behavior changes. +- Generic evidence and response-shape rules have one shared/schema owner. +- All three prompts prepare and their extractor tests remain green. + +### Validation + +```sh +go test ./internal/modules/dnd/extract/npcs ./internal/modules/dnd/extract/locations ./internal/modules/dnd/extract/itemevents +git diff --check +``` + +## Stage 5: Consolidate Spell And Combat-Turn Instructions + +### Goal + +Consolidate the spell and combat-turn prompts while keeping their identity, +catalog, grounding, and event semantics in the correct assets. + +### Required changes + +1. Merge `task.md` and `instructions.md` into `instructions.md`, delete + `task.md`, and update `prompt.yaml` and manifests for spells and combat turns. + +2. Spell module instructions own: + + - inclusion of actual casts and unambiguous declared casting attempts; + - exclusion of mentions, plans, rules discussion, and unsupported catalog + matches; + - in-world caster attribution, including player-character and GM-narrated + casts; and + - the lane-specific requirement that evidence collectively support caster, + spell, and casting event. + + `spell-catalog.md` alone owns the rendered catalog, exact canonical spelling, + and the fact that the catalog is recognition context rather than evidence. + Shared identity and evidence fragments own their generic rules. + +3. Combat-turn instructions own combat-event eligibility and exclusions, + chronology, unmatched actors, turn-level evidence grouping, and the + `turn_kind` meanings. Remove repeated player/NPC identity grounding, + transcript-only evidence, source-range mechanics, and schema-field lists + owned by shared assets or the response schema. + +4. Keep both consolidated instruction messages final and ephemeral. Preserve + the spell catalog and optional NPC grounding as separate variable-context + messages after evidence policy and before module instructions. + +### Acceptance criteria + +- Both prompt directories contain no `task.md`. +- Spell recognition remains catalog-constrained and caster attribution remains + evidence-grounded. +- Combat-turn categories, event boundaries, actor policy, and evidence grouping + are unchanged. +- Neither prompt repeats generic shared identity, evidence, or output-format + policy. +- Prompt preparation, spell schema/catalog tests, combat tests, and the shared + extraction-prefix test pass. + +### Validation + +```sh +go test ./internal/modules/dnd/extract/spells ./internal/modules/dnd/extract/combatturns ./internal/modules/dnd/register +git diff --check +``` + +## Stage 6: Consolidate Grounded Occurrence And Enemy-Event Instructions + +### Goal + +Finish module prompt consolidation for the three registry- or artifact-grounded +event extractors. + +### Required changes + +1. Merge `task.md` and `instructions.md` into `instructions.md`, delete + `task.md`, and update `prompt.yaml` and manifests for: + + - NPC interactions; + - location occurrences; and + - enemy events. + +2. NPC-interaction instructions own registry membership as an eligibility + constraint, occurrence boundaries, all interaction kinds and precedence, + combat-alignment splitting, and domain exclusions. The shared NPC fragment + owns registry rendering, canonical grounding, and its non-evidence status; + shared identity and evidence fragments own their generic rules. + +3. Location-occurrence instructions own occurrence eligibility, all occurrence + kinds and precedence, handling of speculation, independent overlapping + facts, and domain exclusions. `location-registry.md` owns exact ID/name + rendering and registry-not-evidence semantics. Remove repeated schema fields, + `source_id`, narrow-range, and registry-context language from module prose. + +4. Enemy-event instructions own event eligibility, the five event kinds and + outcome definitions, one-engagement-per-scene policy, exclusions, and naming + of registered or grounded unnamed opponents. `combat-grounding.md` and the + shared NPC fragment own rendering and the non-evidence status of auxiliary + inputs. The shared evidence fragment owns source-range mechanics. Remove the + repeated response-field list and generic evidence statements. + +5. Keep each consolidated instruction message final and ephemeral, with all + registry and grounding messages before it and after the chunk transcript. + +### Acceptance criteria + +- No D&D prompt directory contains `task.md`. +- All occurrence kinds, precedence rules, enemy outcomes, splitting rules, and + registry constraints remain intact. +- Registry and auxiliary artifacts remain grounding only and never become + transcript evidence. +- All three prompts prepare, and focused extractor, registry, invariant, and + cache-prefix tests pass. + +### Validation + +```sh +go test ./internal/modules/dnd/extract/npcinteractions ./internal/modules/dnd/extract/locationoccurrences ./internal/modules/dnd/extract/enemyevents ./internal/modules/dnd/register +git diff --check +``` + +## Stage 7: Normalize Private Scene And Spell Schema Identities + +### Goal + +Bring the two outlying private LLM response schemas into the naming convention +already used by the other D&D prompt schemas, without touching durable artifact +schemas. + +### Required changes + +1. Normalize the scene private response schema: + + - rename `assets/dnd/scenes/schemas/dnd_scenes.v1.json` to + `dnd_scenes_llm.v1.json`; + - change its `$id` from `notarius.dnd.scenes` to + `notarius.dnd.scenes.llm`; + - change the scene `prompt.yaml` `schema_path` accordingly; and + - update `internal/modules/dnd/chunk/scenes/schema.go` to use key + `dnd_scenes_llm`, ID `notarius.dnd.scenes.llm`, name + `notarius_dnd_scenes_llm_v1`, and the renamed asset path. + +2. Normalize the already correctly named spell private response schema: + + - keep the filename `dnd_spells_llm.v1.json`; + - change its `$id` from `notarius.dnd.spells` to + `notarius.dnd.spells.llm`; and + - update `internal/modules/dnd/extract/spells/schema.go` to use key + `dnd_spells_llm`, ID `notarius.dnd.spells.llm`, and name + `notarius_dnd_spells_llm_v1`. + +3. Keep both versions at `v1`; these are private pre-release response contracts, + not a compatibility migration. Do not alter required fields, property types, + enum values, `additionalProperties`, or any owning Go response type. + +4. Update focused schema-loading, prompt-preparation, registrar, and production + composition expectations. Carefully distinguish the private spell response + schema from the durable spell artifact schema under + `internal/modules/dnd/codec/spells/assets/schemas/`; the durable schema and + `docs/integrations/dnd-spell-artifacts.md` must not change. + +5. Remove every reference to the retired private scene filename and old private + IDs/names/keys. Do not add aliases. + +### Acceptance criteria + +- Scene and spell private schemas use `_llm`/`.llm` consistently in filenames + where applicable, registry keys, schema IDs, and response-schema names. +- Both schemas still load and validate the same private Go response shapes. +- Durable scene chunk-map and spell artifact contracts are unchanged. +- Registrar and module prompt preparation tests pass without schema collisions. + +### Validation + +```sh +go test ./internal/modules/dnd/chunk/scenes ./internal/modules/dnd/extract/spells ./internal/modules/dnd/register ./internal/cli +git diff --check +``` + +## Stage 8: Document Conventions And Perform Final Verification + +### Goal + +Make the implemented conventions discoverable in their canonical documentation, +remove stale references, and verify the complete repository. + +### Required changes + +1. Rewrite the prompt-construction portion of `docs/internal/dnd.md` to describe + current implemented behavior: + + - the canonical `prompt.yaml` and `instructions.md` local names; + - module manifests as the owner of selection, order, and fingerprinting; + - the `common-dnd-` visual provenance prefix; + - the single-owner hierarchy for universal, family, input, module, and schema + rules; + - the eligibility test for system-prompt content; + - the full transcript, chunk transcript, and transcript-window assets and + their consumers; + - the final extraction, scene-chunking, and entity-normalization message + orders and cache controls; + - the rule against default prompt repetition and the observational-evidence + threshold for reintroducing it; and + - the content-only `assets` boundary, with a link to ADR-0011 rather than a + duplicate rationale. + + Remove the current descriptions of separate task and instructions messages + and the generic transcript fragment. + +2. Do not add an assets-package README or a new ADR. Do not modify ADR-0011. + `docs/internal/llm.md` should remain provider-neutral and should not duplicate + D&D filenames or ordering. Update it only if an earlier stage changed a + provider-neutral PromptKit preparation or cache-control fact; no such change + is expected by this plan. + +3. Review all changed prompt definitions and manifests as a set. Confirm that: + + - every prompt has one system message using the shared system asset; + - every prompt selects exactly one correct transcript-scope asset; + - every extraction prompt retains the stable shared prefix and final + ephemeral module instructions; + - scene chunking and normalizers retain their intentional distinct orders; + - no module-specific prompt repeats a rule already owned by its selected + shared assets or response schema; and + - the root `assets` Go package remains a single minimal filesystem file with + no business logic or internal/PromptKit dependency. + +4. Search for stale production references to: + + - `common-dnd-transcript.md`; + - prompt-ID-derived YAML filenames; + - D&D `task.md` prompt files; + - `catalog.md`, `grounding.md`, and the location-occurrence `locations.md`; + - `dnd_scenes.v1.json` and the old scene private schema identity; and + - the old spell private schema key, ID, and response-schema name. + + Exclude historical roadmap text only when the reference intentionally + explains a retired name. There should be no stale reference in current code, + assets, tests, or implemented documentation. + +5. Review tests for policy compliance. Retain behavior-level assertions for + preparation, schema compatibility, input isolation, cache placement, and the + identical extraction prefix. Remove any leftover assertion whose only value + is freezing exact prompt prose, message count, shared-prefix length, or a + complete private asset inventory. + +6. Run formatting where applicable and the complete offline repository checks. + Do not perform live model evaluation. Report that human evaluation with + representative transcripts and cost-effective target models remains the + next observational step after implementation; do not reintroduce prompt + repetition before that evidence exists. + +### Acceptance criteria + +- `docs/internal/dnd.md` accurately describes the implemented prompt layout, + ownership, transcript scopes, and cache ordering without duplicating external + contracts. +- No new ADR or assets README exists, and ADR-0011 remains unchanged. +- No stale production references or retired files remain. +- All maintained prompts prepare offline, all schemas register, all repository + tests and static checks pass, and the CLI builds. +- The final diff contains no durable artifact, public module/configuration, or + pipeline behavior change. + +### Validation + +```sh +go test ./... +go vet ./... +go build ./cmd/notarius +git diff --check +``` + +Run `gofmt` on the Go files changed by these stages before the commands above. +Inspect the diff and ensure formatting did not rewrite unrelated user changes; +never format an entire directory when the worktree contains unrelated edits. + +## Open Questions + +None. The feature roadmap and this plan make the prompt ownership, transcript +scope, naming, schema identity, documentation, compatibility, and testing +decisions required for implementation. diff --git a/docs/roadmap/prompts.md b/docs/roadmap/prompts.md new file mode 100644 index 0000000..5ded9de --- /dev/null +++ b/docs/roadmap/prompts.md @@ -0,0 +1,463 @@ +# D&D Prompt Asset Cleanup + +## Purpose + +This roadmap defines the target state for the embedded D&D prompt and private +schema assets. The work is intended to make prompts easier to inspect, compare, +and revise; eliminate unnecessary repetition; preserve useful prompt-cache +boundaries; and establish conventions that future D&D modules can follow. + +The central design principle is that each instruction should have one clear +owner. Shared behavior belongs in a shared prompt asset, module-specific +behavior belongs in the module's instruction asset, variable grounding belongs +in a dedicated input asset, and purely structural requirements belong in the +JSON schema. + +## Current State + +The root `assets` package correctly centralizes embedded prompt and schema +content without owning D&D business logic. The D&D prompt assets nevertheless +retain several conventions inherited from their former module-local layouts: + +- prompt definitions use prompt-ID-derived filenames rather than a consistent + local filename; +- most prompt directories divide fixed module instructions between `task.md` + and `instructions.md`, although that division generally does not represent a + meaningful cache, trust, or input boundary; +- module-specific instructions repeat some rules already supplied by shared D&D + messages or enforced by their response schemas; +- one generic transcript fragment currently describes full-session input, + chunk-scoped input, and selected normalization windows even though those + materials have different completeness and inference boundaries; +- variable context assets use several naming styles, including generic names + such as `catalog.md` and `grounding.md`; +- the private scene and spell response-schema identifiers do not follow the + `_llm`/`.llm` convention used by the other private D&D schemas. + +These are internal authoring and maintenance inconsistencies. The durable +artifact contracts and the separation between content ownership in `assets` +and behavior ownership in `internal/modules` remain sound. + +## Goals + +The completed cleanup should: + +- give every D&D prompt directory the same predictable entry-point names; +- consolidate fixed module-specific instructions into one message unless a + distinct message boundary has a concrete purpose; +- remove module-specific restatements of rules already owned by shared assets; +- retain separate messages for shared policy, stable references, variable + transcript content, and other inputs where the boundary communicates trust, + provenance, reuse, or cache behavior; +- distinguish full-session transcripts, extraction chunks, and normalization + evidence windows through separate shared assets with explicit scope rules; +- keep all shared D&D filenames visibly identifiable in prompt definitions; +- make variable context filenames self-describing; +- normalize private LLM response-schema identities without changing durable + output schemas; +- document the conventions future prompt authors must follow; and +- preserve the rule that the root `assets` Go package contains no business + logic. + +## Decisions + +### Canonical prompt directory layout + +Every D&D prompt directory should use the following local layout: + +```text +prompts/ + prompt.yaml + instructions.md + [optional variable or context fragments] +``` + +The prompt definition filename is always `prompt.yaml`. The prompt's `id` field, +not its filename, remains the canonical global identifier. This is safe because +each module registers a distinct asset subtree with PromptKit; the local +`prompt.yaml` filenames do not share a virtual directory. + +Use the `.yaml` extension consistently. It matches the existing embedded prompt +and profile assets. + +For prompts that currently have both `task.md` and `instructions.md`, combine +their non-duplicative content into `instructions.md`. For prompts that currently +have only `task.md`, rename that asset to `instructions.md`. Do not preserve an +otherwise artificial two-message split merely to distinguish a short task +statement from its rules. + +### Retain the `common-dnd-` prefix + +Shared D&D assets must retain the `common-dnd-` filename prefix, including: + +- `common-dnd-system.md` +- `common-dnd-identity.md` +- `common-dnd-references.md` +- `common-dnd-transcript-full.md` +- `common-dnd-transcript-chunk.md` +- `common-dnd-transcript-windows.md` +- `common-dnd-extraction-evidence.md` +- `common-dnd-npcs.md` +- `common-dnd-entity-reconciliation.md` + +The prefix is a deliberate visual provenance marker. When reviewing a prompt +definition, an author should be able to distinguish shared messages from +module-owned messages immediately. Because asset references are explicit and +machine-resolved, retaining the prefix has no meaningful mechanical cost. + +### One owner per instruction + +Prompt prose should follow this ownership hierarchy: + +1. Universal D&D prompting behavior belongs in a shared D&D asset. +2. Rules shared by a prompt family belong in the narrowest applicable shared + asset. +3. Semantics of a variable input belong in the fragment that renders that + input. +4. Lane-specific extraction, classification, or normalization policy belongs in + that lane's `instructions.md`. +5. Object shape, required fields, enum values, and rejection of unknown fields + belong in the JSON schema unless prose is needed to explain their domain + meaning. + +Module-specific prompt assets should omit instructions already provided by the +shared prompt set. In particular, they should not routinely repeat generic rules +about transcript-only evidence, references not being evidence, omission of +`source_id`, narrow transcript ranges, schema-only output, or explanatory text. + +This cleanup should begin with the minimal instruction set that is both correct +and non-repetitive. Repetition may be reintroduced only when evaluation produces +repeatable observational evidence that a target model needs reinforcement. Any +such repetition should be narrow, placed close to the rule's canonical owner, +and documented as a deliberate model-reliability tradeoff rather than allowed to +become accidental drift. + +This rule does not authorize removal of domain semantics. Similar wording is not +duplication when it defines a distinct lane contract, such as event precedence, +holder transitions, scene-kind definitions, identity inclusion criteria, or the +relationship between a combat actor and an extracted event. + +### System-prompt eligibility + +`common-dnd-system.md` should own every fixed instruction that genuinely applies +to every maintained D&D LLM prompt. Consolidating those universal rules into the +system message gives them one authoritative owner, applies the appropriate +instruction priority, and preserves one byte-identical initial message across +chunking, extraction, and normalization prompts. + +An instruction belongs in the shared system message only when it is: + +- applicable to scene chunking, every extraction lane, and both entity + normalizers; +- independent of prompt-specific or variable inputs; +- static across calls; and +- appropriate to treat as a high-priority behavioral constraint. + +The system message should therefore own the D&D transcript-processing role, +reliance only on supplied inputs, treatment of transcription uncertainty, and +the requirement to return exactly one schema-conforming JSON object without +explanatory prose. + +Rules shared by only a subset of prompts must remain in the narrowest applicable +shared user-message fragment. Transcript-evidence and citation rules, identity +rules, campaign-reference semantics, registry grounding, and reconciliation +policy are not universal system instructions. No rendered transcript, +reference, registry, catalog, candidate, or other variable material belongs in +the system message. + +### Shared prompt ownership + +The shared D&D files should have clear, non-overlapping responsibilities: + +| Asset | Canonical responsibility | +| --- | --- | +| `common-dnd-system.md` | D&D domain role, reliance on supplied inputs, treatment of transcription uncertainty, and the universal requirement to emit exactly one schema-conforming JSON object without commentary. | +| `common-dnd-identity.md` | Rules for resolving supported in-world identities and the distinction between disambiguation and participation evidence. | +| `common-dnd-references.md` | Rendering and interpretation of campaign references, including that references provide context rather than establishing transcript events. | +| `common-dnd-transcript-full.md` | Rendering the complete ordered session transcript and identifying it as complete session input. | +| `common-dnd-transcript-chunk.md` | Rendering one extraction chunk, limiting the task to that chunk, and explaining that its unit IDs retain their source-wide meaning. | +| `common-dnd-transcript-windows.md` | Rendering selected reconciliation evidence windows and explaining that they may be incomplete, non-contiguous, or overlapping. | +| `common-dnd-extraction-evidence.md` | Transcript-evidence policy, source-range construction, omission of `source_id`, and preference for narrow supporting ranges. | +| `common-dnd-npcs.md` | Rendering and interpreting an upstream NPC registry as grounding rather than evidence. | +| `common-dnd-entity-reconciliation.md` | Safe duplicate-group proposal mechanics shared by entity normalizers. | + +As part of normalization, universal output-format prose should move to +`common-dnd-system.md`, while evidence-specific prose should remain in +`common-dnd-extraction-evidence.md`. The system message should refer generally +to supplied inputs rather than only transcripts and references, because +normalizers and grounded extractors receive additional input types. + +### Transcript-scope ownership + +The current `common-dnd-transcript.md` should be replaced rather than retained +as an alias. All three replacements may render the same prompt input named +`transcript`, but each must accurately describe the material supplied by its +caller: + +- `common-dnd-transcript-full.md` is used by scene chunking. It identifies the + material as the complete ordered session and permits whole-session coverage + decisions without implying that the entire session is one scene. +- `common-dnd-transcript-chunk.md` is used by every extraction lane. It states + that only the current chunk is in scope, that source-wide unit IDs remain + authoritative, and that the model must not infer or report events outside the + supplied chunk. +- `common-dnd-transcript-windows.md` is used by NPC and location normalization. + It states that the material consists of selected evidence windows that may be + incomplete, non-contiguous, or overlapping. The model may use those windows + to evaluate candidate identity but must not treat absence outside them as + evidence. + +The scope fragments should describe input completeness and permissible use; +they should not absorb scene policy, extraction evidence mechanics, or entity +reconciliation rules owned by other assets. Future prompt families must select +the fragment matching the material actually passed by code rather than choosing +one based merely on an input variable named `transcript`. + +### Module-specific ownership + +After shared duplication is removed, each module's `instructions.md` should +retain only its substantive lane policy: + +| Prompt | Module-owned policy | +| --- | --- | +| Scene chunking | Scene-boundary rules, ordered coverage, and valid treatment of gaps or overlaps. | +| Scene descriptions | `kind` classification definitions and precedence, plus title and summary semantics. | +| NPC extraction | Inclusion and exclusion rules for individually identifiable NPCs. | +| NPC interactions | Interaction kinds, precedence, occurrence splitting, and NPC-specific event semantics. | +| Location extraction | Inclusion and exclusion rules for identifiable in-world places. | +| Location occurrences | Occurrence kinds, precedence, splitting, and location-specific event semantics. | +| Spell extraction | What constitutes a cast, catalog-name policy, and caster attribution. | +| Combat turns | Combat event boundaries, event kinds, actor semantics, and combat-specific inclusion rules. | +| Item events | Event kinds, possession transitions, holder semantics, quantities, and currency treatment. | +| Enemy events | Enemy-event kinds, outcome semantics, encounter identity, and combat-grounding rules. | +| NPC normalization | NPC-specific duplicate criteria and canonical-name selection. | +| Location normalization | Location-specific duplicate criteria and canonical-name selection. | + +Schema field lists should not be copied into prose solely to restate the schema. +Prose may still explain how a field is interpreted or how two fields relate when +the schema cannot express that semantic rule. + +### Message boundaries and cache ordering + +Combining `task.md` and `instructions.md` must not lead to indiscriminate message +consolidation. Separate messages remain appropriate when they preserve a shared +byte-identical prefix, distinguish fixed instructions from variable material, +or identify the provenance and trust level of an input. + +Extraction prompts should retain this general order: + +```text +system +identity +campaign references +chunk transcript +evidence policy +optional grounding or variable context +module instructions +``` + +The scene chunker should retain this general order: + +```text +system +campaign references +module instructions +full transcript +``` + +Entity normalizers should retain this general order: + +```text +system +module instructions +shared entity-reconciliation policy +candidates +transcript windows +``` + +Existing cache controls should remain attached to the same semantic boundaries +unless PromptKit's contract requires a mechanical adjustment. The cleanup must +not move variable transcript or candidate content into an earlier stable prefix +merely to reduce message count. + +### Variable context asset names + +Optional non-shared fragments should describe the input they render rather than +use generic names. Normalize the current assets as follows where applicable: + +- `catalog.md` to `spell-catalog.md`; +- `grounding.md` to `combat-grounding.md`; +- `locations.md` to `location-registry.md`. + +The name `candidates.md` may remain in normalizer-local directories because its +meaning is clear in that context. Future context assets should use similarly +specific nouns and should remain separate from fixed `instructions.md` content. + +### Private response-schema identity + +Private schemas used only to constrain LLM responses should consistently use +the `_llm` filename/key/name convention and the `.llm` schema-ID suffix already +used by most D&D modules. + +Normalize the scene schema to: + +- file: `dnd_scenes_llm.v1.json`; +- schema ID: `notarius.dnd.scenes.llm`; +- registry key: `dnd_scenes_llm`; +- response-schema name: `notarius_dnd_scenes_llm_v1`. + +Normalize the spell schema metadata to: + +- file: `dnd_spells_llm.v1.json` (already named correctly); +- schema ID: `notarius.dnd.spells.llm`; +- registry key: `dnd_spells_llm`; +- response-schema name: `notarius_dnd_spells_llm_v1`. + +Private schema filenames must remain globally unique. They should not be renamed +to a generic local name because the schema registry exposes a shared namespace. +Durable artifact schema identifiers and shapes are out of scope and must not +change. + +### Explicit prompt definitions remain explicit + +The repeated `version`, `default_profile`, input, message, output, and schema +declarations in `prompt.yaml` files should remain explicit. They are declarative +configuration, not prompt-prose duplication. Introducing a generator, template +layer, or Go builder for these small definitions would make inspection harder +and create another synchronization boundary. + +Similarly, module manifests should continue to declare the assets their prompt +uses. Shared asset selection is module behavior and does not belong in the root +`assets` package. + +## Compatibility and Cache Effects + +This is an intentional internal prompt revision. Renaming prompt definition +files, consolidating messages, editing shared prose, and normalizing private +schema identifiers will change prompt and component fingerprints. Existing +checkpoints may no longer be reusable, and backend prompt caches will experience +a one-time cold start. Those fingerprint, checkpoint, and cache effects are +accepted consequences of reaching the target prompt layout. + +The following contracts must remain stable: + +- public prompt IDs; +- module IDs and configuration fields; +- default profile selection; +- required and optional prompt inputs; +- durable artifact schema IDs and artifact shapes; +- extraction and normalization semantics, except for removing unintended prompt + repetition; and +- pipeline dependency behavior. + +No backward-compatible copies of old private prompt assets or schema aliases are +required while the application remains pre-release. + +## Documentation and Maintenance Rules + +The implemented convention should be documented in the canonical internal D&D +and LLM documentation rather than in an assets-package README that could drift. +The documentation should explain: + +- the canonical `prompt.yaml` and `instructions.md` names; +- why shared assets retain the `common-dnd-` prefix; +- the instruction-ownership hierarchy; +- the eligibility test for moving a rule into `common-dnd-system.md`; +- the full, chunk, and evidence-window transcript contracts and which prompt + families select each one; +- when a separate message boundary is warranted; +- the cache-oriented message ordering; +- the prohibition on business logic in `assets`; and +- the evidence threshold for deliberately repeating an instruction. + +The Go file in the root `assets` package should remain a minimal embedding and +filesystem-access layer. Prompt composition, schema selection, module policy, +and validation logic must remain in their existing owning packages. + +`docs/internal/dnd.md` should be the canonical owner of these D&D prompt +authoring conventions and the concrete message-order patterns. Its existing +prompt-construction section should be revised when the assets change rather +than creating an `assets` README or duplicating the rules in contributor +orientation. `docs/internal/llm.md` should receive only any short update needed +to describe provider-neutral PromptKit preparation or cache-control mechanics; +it should link to the D&D guide instead of repeating domain-specific asset names +or ordering. + +No new ADR is warranted for this cleanup. ADR-0011 already records the +architecturally significant decisions: physical centralization, content-only +package ownership, and retention of prompt semantics and ordering by modules. +The filename, instruction-ownership, transcript-scope, and system-message rules +are internal authoring conventions that should be allowed to evolve with model +evaluation. They do not create a new dependency boundary or reverse an accepted +architectural decision. If a later change promotes these conventions into a +cross-domain runtime-enforced prompt composition contract, that broader +decision should receive its own ADR at that time; ADR-0011 must not be edited to +retrofit the new rationale. + +## Validation and Evaluation Policy + +Automated validation should establish that the refactor preserves executable +contracts without freezing prompt prose. It should cover: + +- preparation of every embedded prompt through PromptKit; +- prompt input declarations and required-input behavior; +- private response-schema registration and compatibility with the owning Go + response types; +- expected cache controls and meaningful stable-prefix boundaries; +- selection of the transcript-scope fragment appropriate to each prompt + family, verified through prompt preparation and owning-module behavior rather + than exact prose snapshots; +- manifest references after asset renames; +- absence of references to retired filenames and private schema identifiers; + and +- the full repository test suite. + +Tests must not assert exact prompt wording, exact message counts, shared-prefix +lengths, or asset inventories solely as change detectors. Such tests would make +intentional prompt iteration unnecessarily expensive and conflict with the +project's testing policy. + +Model quality should be evaluated by running representative transcripts through +all affected lanes, with particular attention to cost-effective smaller models. +Review should compare extraction correctness, unsupported claims, missing +events, source-range quality, and schema reliability. The initial evaluation +should use the minimal non-repetitive prompts. A repeated instruction should be +restored only if failures recur and there is a credible causal reason that +reinforcement will help. + +## Non-Goals + +This roadmap does not include: + +- changes to durable D&D artifact schemas; +- new extraction lanes or pipeline behavior; +- changes to profile selection or model settings; +- a prompt-definition generator or templating framework; +- moving business logic into `assets`; +- collapsing all prompt material into one message; +- placing family-specific rules or rendered variable content in the system + message; +- preserving old prompt fingerprints or warm backend cache entries; or +- introducing brittle prompt snapshots or change-detector tests. + +## Target End State + +Every D&D prompt is discoverable through a local `prompt.yaml`, has one fixed +module-owned `instructions.md`, and uses separately named context fragments only +where inputs or cache boundaries justify them. Shared D&D assets remain visibly +marked with `common-dnd-` and have documented, non-overlapping responsibilities. +Module instructions contain no routine repetition of shared or schema-enforced +rules, while retaining all lane-specific domain semantics. + +The shared system message contains all and only universal, static D&D behavior. +Scene chunking renders the explicitly full transcript fragment, extraction +lanes render the explicitly chunk-scoped fragment, and entity normalizers render +the explicitly incomplete evidence-window fragment. No prompt presents one +scope as another merely because all three inputs use the name `transcript`. + +Private scene and spell response schemas follow the established LLM-only naming +convention, durable contracts are unchanged, all manifests and PromptKit +definitions prepare successfully, and human evaluation begins from a minimal +non-repetitive prompt set. Future modules can follow the same conventions +without copying an existing extractor and inheriting accidental prose or naming +drift.