From 74c793e6a1fb13d2e96cf3f246c45bf6ba183504 Mon Sep 17 00:00:00 2001 From: Eric Rakestraw Date: Sun, 26 Jul 2026 13:30:36 +0000 Subject: [PATCH] Document D&D module conventions --- docs/development.md | 2 +- docs/internal/dnd.md | 119 ++++++ docs/internal/modules.md | 827 +++++--------------------------------- docs/internal/overview.md | 8 +- 4 files changed, 224 insertions(+), 732 deletions(-) create mode 100644 docs/internal/dnd.md diff --git a/docs/development.md b/docs/development.md index 05b68d7..c90d64e 100644 --- a/docs/development.md +++ b/docs/development.md @@ -20,7 +20,7 @@ implemented component map. | CLI composition or command behavior | [CLI Internals](internal/cli.md) and [CLI Reference](cli.md) | The internal guide owns composition and command flow; the reference owns public syntax. | | Configuration loading, resolution, or user-visible configuration behavior | [Configuration Internals](internal/configuration.md) and [Configuration](config.md) | The internal guide owns loading and resolution mechanics; the reference owns the configuration contract. | | Pipeline resolution or execution | [Pipeline Internals](internal/pipeline.md) | It documents profiles, references, validation, retries, checkpoints, and runner behavior. | -| Production modules or validators | [Module Internals](internal/modules.md) and [D&D integration contracts](integrations/) | The generic module guide owns extension mechanics; D&D artifact contracts own durable output shapes. | +| Production modules or validators | [Module Internals](internal/modules.md), [D&D Module Internals](internal/dnd.md), and [D&D integration contracts](integrations/) | The generic guide owns extension mechanics, the D&D guide owns shared family conventions, and the contracts own durable output shapes. | | LLM clients, prompts, schemas, profiles, or scheduling | [LLM Runtime](internal/llm.md) | It documents the transport boundary and Scriptorium integration. | | Output, cache, resume, or debug artifacts | [Run State Internals](internal/state.md), [Operations](operations.md), and [Configuration](config.md) | These separate implementation details, operator behavior, and configuration contracts. | | External input formats, artifact schemas, or durable output files | [Integration Contracts](integrations/) | Integration documents define external and durable data contracts. | diff --git a/docs/internal/dnd.md b/docs/internal/dnd.md new file mode 100644 index 0000000..7372eca --- /dev/null +++ b/docs/internal/dnd.md @@ -0,0 +1,119 @@ +# D&D Module Internals + +This guide records the conventions shared by the production D&D module family. +It complements [Module Internals](modules.md), which owns generic registration +and extension mechanics, and [Configuration](../config.md), which owns the +selectable keys, bindings, reference syntax, and default validator chains. + +## Durable Artifact Contracts + +The six lanes have separate durable wire contracts. This guide deliberately +does not repeat their JSON shapes or schemas. + +| Lane | Durable contract | +| --- | --- | +| Spells | [spell artifacts](../integrations/dnd-spell-artifacts.md) | +| NPCs | [NPC artifacts](../integrations/dnd-npc-artifacts.md) | +| Combat turns | [combat-turn artifacts](../integrations/dnd-combat-turn-artifacts.md) | +| Item events | [item-event artifacts](../integrations/dnd-item-event-artifacts.md) | +| NPC interactions | [NPC-interaction artifacts](../integrations/dnd-npc-interaction-artifacts.md) | +| Scene descriptions | [scene-description artifacts](../integrations/dnd-scene-description-artifacts.md) | + +## Family Composition + +The D&D registrar registers the family’s artifact codecs, extractors, typed +append-order mergers, normalizers, validators, prompt assets, and default +validator chains. Each extractor and normalizer has a stable module spec, +strict option decoding, and a typed builder. Configuration remains the +canonical owner of the exact keys and validator order. + +Private structured-LLM response schemas are deliberately minimal. They reject +invalid JSON structure, missing required fields, incompatible types, and +unknown fields, while preserving semantic candidates for deterministic +validation. Do not promote a private response envelope into a durable schema; +the contracts above define durable data. + +## Prompt Construction + +D&D extractors assemble prompts from an ordered manifest of shared and +module-owned assets. Reuse the shared D&D system, evidence, identity, +reference, and transcript assets instead of copying their text into individual +modules. A manifest’s declared sequence, including cache-control placement, is +part of the prompt behavior, and the chunk transcript is the final message. +Preserve that order when changing an extractor or its assets so prompt-cache +behavior remains stable. + +All extractors use the shared prompt-input preparation rules. The current chunk +is copied into transcript material; player, party, glossary, and compatible +campaign references are context for disambiguation, not source evidence. +Reference prompt material is canonically ordered before it is rendered, which +keeps equivalent inputs stable across runs. + +## Evidence, Candidates, And Normalization + +The current transcript is the only durable evidence source. Extractors assign +the current source identity, preserve candidate evidence ranges for validators, +and canonically order or remove exact duplicate ranges without asking the +model to repair semantic errors. Campaign context and generated artifacts may +ground names or control routing, but they never establish evidence for a D&D +result. + +Default chains keep responsibilities separate: structural validators assess the +candidate, source-reference validators resolve cited ranges against the current +source, durable-schema validation checks an approved representation, and +relatedness validators report advisory evidence concerns. The configured order +is documented in +[Configuration](../config.md#production-validator-keys-and-default-chains). + +Normalizers are deterministic for spells, combat turns, item events, NPC +interactions, and scene descriptions. They canonicalize display values and +evidence, use source-document order for stable output, and issue bounded +warnings for changes or collapsed duplicates. The NPC normalizer is the +intentional exception: it first produces a deterministic candidate set, then +uses a bounded structured-LLM proposal to reconcile identity groups. Invalid +or unusable proposals retain the deterministic result and surface retry or +fallback diagnostics; the model does not directly replace durable records. + +## Generated References And Grounding + +Normalized D&D artifacts can be handed to a later step through a generated +reference binding. The framework verifies artifact compatibility and retains +producer provenance; consumers resolve the handed-off artifact into an +immutable, validated projection for each operation. External files are checked +during preparation, while generated artifacts are resolved at the handoff. + +NPC registries are names-only grounding projections: they may canonicalize +actors for spells and combat turns and are required for NPC interactions, but +they do not supply evidence. Scene-description registries are eligibility-only +projections: they retain the current chunk’s classification data, not scene +prose or evidence, and exist to route combat extraction. + +## Lane-Specific Rules + +The following differences are intentional and should remain explicit when a +shared helper changes. + +| Lane | Intentional behavior | +| --- | --- | +| Spells | May use a spell-catalog overlay and optional NPC grounding; the catalog validator supplies domain-specific semantic checks. | +| NPCs | Does not consume an NPC registry. Its normalizer is the LLM-assisted reconciliation exception described above. | +| Combat turns | Requires a scene-description artifact. It calls the LLM only for an exact `combat` classification; exact non-combat classifications return an accepted empty result, while missing or mismatched classifications return an empty result with a bounded warning. Optional NPC grounding never becomes evidence. | +| Item events | Uses campaign context for disambiguation but has no NPC-registry or scene-description dependency. | +| NPC interactions | Requires the normalized NPC registry at extraction and normalization, using it for canonical actor grounding only. | +| Scene descriptions | Produces the classifications consumed by combat routing; it does not consume an NPC registry or provide evidence for combat artifacts. | + +The combat and scene-description contracts describe their exact handoff and +empty-result behavior in more detail: +[combat turns](../integrations/dnd-combat-turn-artifacts.md) and +[scene descriptions](../integrations/dnd-scene-description-artifacts.md). + +## Focused Verification + +When changing D&D behavior, test the affected codec, extractor, normalizer, +validator, prompt-asset manifest, and registry projection. Also test generated +handoffs at the integration boundary and run the full D&D module suite: + +~~~sh +go test ./internal/modules/dnd/... +go test ./internal/modules/integration/... +~~~ diff --git a/docs/internal/modules.md b/docs/internal/modules.md index 6f336f1..77ef39b 100644 --- a/docs/internal/modules.md +++ b/docs/internal/modules.md @@ -1,728 +1,99 @@ -# Module And Validator Internals - -Production module and validator implementations live under their domain-first -trees in `internal/modules`. -The selectable keys, configuration options, reference slots, and default -validator chain are canonical in the -[module](../config.md#implemented-production-modules) and -[validator](../config.md#implemented-production-validators) catalogs in -Configuration. - -## Extension Pattern - -A stage module package provides a stable key, constructor, contract -implementation, `ModuleSpec`, `Register`, and focused behavior and registration -tests. A validator package follows the same pattern with `ValidatorSpec` and the -validator registry. Package-family registrars compose those leaf registrations -into the production catalog and own family-level policy such as default -validator chains and prompt asset collection. - -Production input, chunk, output, and D&D spell-, NPC-, combat-, item-event-, interaction-, and scene-description-extract packages -register strict option decoders and run-local builders. Preparation decodes their options into -implementation-owned values and injects dependencies plus the materialized -reference set for the selected target. Each builder receives an isolated clone -of that set; input and output builders receive no references. The spell, NPC, -combat, item-event, interaction, and scene-description extractors are typed over the canonical D&D model. D&D validators, merge, -and normalize use typed variants; JSON representation validators use serialized -requests; and unconditional validators expose separate chunk and typed -variants. The D&D production registrar registers the canonical typed spell, -NPC, combat, item-event, interaction, and scene-description implementations, including their kind-specific merge and -normalize behavior. - -For D&D artifact defaults, generic JSON syntax validation runs first. Rejecting -domain validators then own semantic diagnostics before generic JSON Schema -validation provides the final rejecting representation backstop; warning-only -relatedness validators run last. This default composition does not reorder an -explicitly configured validator chain. - -Prepared extractors, extract validators, and codecs may be reused concurrently -by the run-wide extract pool. Production implementations are immutable after -construction: they retain only typed options, immutable assets, or the shared -concurrency-safe LLM client. Implementations that introduce mutable state must -synchronize that state without creating a separate provider scheduler. - -Specs expose capability and execution metadata without constructing an -implementation. Registry entries separately expose option validation and -run-local construction. Chunk, extract, merge, and normalize modules that accept -auxiliary material declare identical reference slots from both -`ReferenceSlots()` and `ModuleSpec().ReferenceSlots`; registration tests enforce -that agreement. Runtime delivery uses the corresponding stage request's -`References` field. - -LLM-backed extensions own their prompt definitions and response schemas under -package-local embedded assets. Shared filesystem composition belongs in -`internal/framework/promptfs`; reusable D&D prompt fragments, reference -declarations, prompt-input assembly, and source-unit/citation helpers belong in -`internal/modules/dnd/shared`, which owns operation-scoped indexed -source-reference validation, citation traversal, ordering and canonicalization, -plus bounded D&D diagnostics. The -D&D scene chunker and spell, NPC, combat-turn, item-event, NPC-interaction, and scene-description extractors use ordered -package-local prompt manifests for both rendering and prompt fingerprinting, so -only the shared fragments each prompt actually renders participate in either -operation. Extraction prompts place stable shared and lane-specific context -before the variable transcript and use shared assets for wording common across -lanes. The canonical ordering and cache-boundary policy is documented in -[LLM Runtime](llm.md#dd-extraction-prompt-ordering-and-cache-boundaries). Stage -contracts expose only Notarius structured-completion types, not Scriptorium -public types. - -The shared `PrepareChunkExtraction` helper owns common extraction preflight and -transcript material preparation for the spell, NPC, combat-turn, item-event, -NPC-interaction, and scene-description extractors. It validates common request -state, clones supplied source metadata, falls back to the materialized chunk -when content is absent, checks that content remains chunk-identical, and fills -only the common default fields. Extractors retain receiver, dependency, and -lane-specific checks locally and wrap helper errors with their module context. - -Reference material may inform a module or prompt but must not become source -evidence. The resolver and materializer behavior is described in -[Pipeline Internals](pipeline.md#reference-materialization). - -## Domain Reference Data - -### `internal/modules/dnd/spells/catalog` - -The spell catalog package owns the embedded, versioned D&D 5e 2014 SRD spell -reference data. Its strict JSON asset contains one canonical record per spell, -including spell level and all applicable class memberships. `LoadSRD5E2014` -validates catalog identity, provenance metadata, ordering, uniqueness, levels, -classes, aliases, and lookup-key collisions before exposing immutable copies. - -Lookup is case-insensitive and normalizes whitespace and common apostrophe -variants while preserving source punctuation in canonical display names. The -catalog contains 319 unique spells and 779 class memberships. Source and -license details live beside the asset in `SOURCES.md`. This domain-owned data is -separate from `internal/modules/dnd/shared`, which is reserved for reusable -prompt and source-reference machinery. - -`ResolveEffectiveCatalog` builds the immutable recognition view used by the -spell extractor and catalog validator. It starts with the embedded SRD catalog -and optionally applies one strict JSON overlay from the `spell_catalog` item in -a materialized reference set. Overlay catalogs are ordered by ID, may add names -and aliases, and may augment an existing canonical spell without replacing its -display name. Cross-spell lookup collisions are errors. The effective view -exposes sorted canonical names, normalized lookup, overlay identities, and a -semantic digest; overlay content remains contextual reference material rather -than source evidence. Its external JSON contract is defined in the -[spell-catalog overlay contract](../integrations/dnd-spell-catalog-overlays.md). - -### `internal/modules/dnd/npcs/identity`, `internal/modules/dnd/npcs/registry`, and `internal/modules/dnd/codec/npcs` - -The NPC identity package owns Unicode comparison keys, deterministic -`npc:sha256:` IDs, display normalization, and whole-registry collision issues. -The registry package resolves one optional normalized artifact through the -strict codec, validates whole-registry identity, canonicalizes its JSON, and -provides immutable records, a names-only prompt projection, distinct durable -and projection digests, count, and exact canonical-name lookup. External files cross this boundary during -preparation; generated artifacts cross it at the ordered step handoff. It owns -the `npcs` slot and its bounded, content-safe validation failures. NPC source -references are durable provenance and are not treated as evidence for a -consuming pipeline. The NPC codec owns the strict durable `dnd/npc-list` JSON -boundary and exposes candidate versus approved encode/decode operations. The -shared `internal/modules/dnd/codec/candidatejson` package supplies strict typed -candidate JSON mechanics; each artifact codec retains its own durable schema -and approved-value policy. - -### `internal/modules/dnd/scenedescriptions/registry` - -The scene-description registry owns the required `scene_descriptions` control -reference used by combat extraction. It decodes exactly one approved scene-list -artifact through the scene-description codec and retains only scene ID, exact -source reference, and kind. Titles, summaries, original bytes, paths, and -prompt material do not cross this domain boundary. - -An external reference is validated during preparation; an unbound seed is -permitted only while a configured generated reference awaits the ordered -handoff. At operation time, a generated artifact overrides the seed and is -resolved into an immutable view safe for concurrent extract jobs. Matching is -strictly exact by chunk ID, source ID, start unit ID, and end unit ID, producing -an exact, missing, or mismatched result. Only an exact result exposes kind. - -The registry's semantic eligibility digest is derived from a sorted projection -of ID, exact range, and kind. It ignores titles, summaries, and input order; -the unbound view has a stable empty projection digest. Combat extractor -metadata and checkpoint identity use this semantic boundary for external -references, while generated artifact identity and dependencies remain owned by -the framework handoff. - -The `internal/modules/dnd/codec/combatturns` package owns the durable -`dnd/combat-turn-list` schema and candidate versus approved JSON boundary. It -is registered by the production D&D family registrar for the selectable combat -lane. - -The `internal/modules/dnd/codec/itemevents` package owns the durable -`dnd/item-event-list` schema and candidate versus approved JSON boundary. It is -registered by the production D&D family registrar. Its external contract is -defined in the [D&D item-event artifact contract](../integrations/dnd-item-event-artifacts.md). - -The `internal/modules/dnd/codec/npcinteractions` package owns the durable -`dnd/npc-interaction-list` schema and candidate versus approved JSON boundary. -It is registered by the production D&D family registrar for the selectable -interaction lane. Its external contract is documented in the -[D&D NPC interaction artifact contract](../integrations/dnd-npc-interaction-artifacts.md). - -The `internal/modules/dnd/codec/scenedescriptions` package owns the durable -`dnd/scene-description-list` schema and candidate versus approved JSON boundary. -It is registered by the production D&D family registrar. Its external contract -is documented in the -[D&D scene-description artifact contract](../integrations/dnd-scene-description-artifacts.md). - -## Input Adapter - -### `internal/modules/seriatim/input/transcript` - -The adapter decodes the supported transcript JSON, selects the source identity, -computes canonical source provenance, validates segments, and maps each segment -into a generic source unit with a self-reference plus speaker and timestamp -metadata. It accepts no module options. Its spec advertises the transcript -capabilities consumed by D&D modules. - -Parsing is strict about required values and duplicate unit IDs but deliberately -ignores unrelated Seriatim fields. The external format and derived-identity -rules are defined in the -[Seriatim contract](../integrations/seriatim.md). - -## Chunkers - -Chunkers implement `contracts.Chunker.Plan`. A plan identifies ordered source -unit ranges and may carry optional namespaced JSON annotations; it does not -contain materialized chunk content. The framework canonicalizes annotations, -validates ranges against the current source, and materializes chunk IDs, -indexes, references, content, units, and generic metadata. Materialized source -unit metadata is independently owned. Annotation -namespaces remain optional data: generic framework code and downstream modules -must not require D&D scene annotations or import `dnd/scenes`. - -### `internal/modules/generic/chunk/units` - -The generic chunker validates the source document and returns ranges over units -in configured windows. Overlap changes the next window start but never reorders -units. Framework materialization derives the resulting chunk identity and -generic metadata from those ranges. - -The accepted options and defaults are defined in -[Configuration](../config.md#implemented-production-modules). Generic -framework validation canonicalizes the returned unit slices before extraction. -The chunker decodes its options during construction and retains only the typed -window settings used by `Plan`. - -### `internal/modules/dnd/chunk/scenes` - -The scene chunker prepares a structured Scriptorium request from the full -transcript, session, and optional D&D reference inputs. It validates the model's -inclusive source-unit endpoints against document position and converts them -into deterministic plan ranges. Preparation injects the shared structured LLM -client into the chunker; `Plan` supplies only the run-specific profile, session, -source, references, and metadata. - -Scene validation requires sequential, contiguous, non-overlapping coverage from -the first source unit through the last. Its private response contains only the -boundary endpoints; the accepted plan has no D&D-specific annotations and -produces no boundary warnings. Malformed structured output is returned as an -error; there is no fallback chunker. - -The package embeds its prompt and response schema and reports their non-secret -identity and hashes through singleton module metadata. Shared D&D assets supply -reference declarations and prompt inputs; their user-facing keys and accepted -file types remain canonical in [Configuration](../config.md). - -## Extractor - -### `internal/modules/dnd/extract/spells` - -The spell extractor prepares a structured request from one chunk, the -chunk-scoped source input, the session, and optional D&D reference inputs. It -decodes the model response, assigns the generic source identity to every source -reference, canonicalizes duplicate references, orders spell casts by their -earliest valid source-document position, and returns `dnd.SpellList`. - -Its private response schema admits only the structural transport envelope: -required fields, JSON types, array and object shapes, and unknown-field -rejection. It maps integer source-unit candidates directly without repairing -semantic values, so the deterministic shape, catalog, and source-reference -validators own blank values, empty evidence, and invalid or unresolved ranges. - -The extractor owns its private model-response DTO, embedded prompt, LLM response -schema, strict option decoder, injected shared LLM client, and prompt/schema -manifest metadata. During preparation it resolves the optional `spell_catalog` -reference into an immutable effective catalog and adds a generated -canonical-name-only JSON input to every structured completion request. Overlay -failures therefore stop construction before source parsing or an LLM call; -campaign references remain separate disambiguation inputs and never become -source evidence. - -The prompt includes only actual casting events and unambiguous declared casting -attempts. Spell mentions, plans, rules discussion, and catalog matches without -a casting event are excluded. Shared extraction-evidence and identity rules -require transcript-supported caster and spell facts, while the catalog, -campaign references, and NPC names only disambiguate source text. Structural -source validation remains deterministic; semantic evidence sufficiency is -enforced through extraction policy and evaluation. - -Both the extractor and deterministic catalog validator expose -the effective base-plus-overlay semantic digest as scoped prepared-component -checkpoint identity. Raw overlay provenance independently covers file-byte -changes, while the semantic digest also invalidates reuse when the embedded -catalog or catalog composition changes. The extractor additionally fingerprints -its complete prompt assets and private response schema, so either semantic -contract changing invalidates previously recorded extraction checkpoints. The -separate `internal/modules/dnd/codec/spells` package -owns the durable schema and stable JSON representation for artifact kind -`dnd/spell-list`. The runner keeps the result typed through validators and later -stages, using the codec only for checkpoint, debug, and output boundaries. -Shared D&D helpers keep prompt input names and source-unit reference conversion -consistent with the scene chunker. - -The extractor also declares the optional `npcs` registry slot and consumes the -immutable registry boundary from `internal/modules/dnd/npcs/registry`. An -external registry is prepared before execution; a generated registry is -validated and supplied at operation time. Bound external registries add only -the full `npc_registry_digest` and `npc_count` to module metadata. The local -`npc_registry` checkpoint fingerprint always covers the names-only projection, -including its exact unbound value. Generated bindings are represented by -framework handoff provenance and dependency fingerprints. The unbound prompt -input is exactly `{"npcs":[]}` and has no registry provenance. -The shared NPC grounding fragment is placed immediately after the common -campaign reference message and is included in the spell prompt fingerprint. - -The durable payload and manifest metadata shapes are defined in the -[D&D spell artifact contract](../integrations/dnd-spell-artifacts.md). - -### `internal/modules/dnd/extract/npcs` - -The NPC extractor maps private model output to the canonical `dnd.NPCList`, -assigns source identity and deterministic NPC IDs, and preserves source -references for deterministic validation. It uses the shared campaign -references only for disambiguation and does not consume the optional NPC -registry slot. Its prompt and private response schema are package-owned. The -private response contains only a name and model-facing evidence ranges for each -record; anonymous groups, generic roles, invented labels, descriptions, -aliases, and relationships are outside its contract. The -prompt follows the shared D&D extraction ordering and cache policy documented -in [LLM Runtime](llm.md#dd-extraction-prompt-ordering-and-cache-boundaries). - -The private response schema owns only structural transport validation and maps -integer source-unit candidates unchanged. Required semantic content, non-empty -evidence, and valid source ranges are rejected by the deterministic shape and -source-reference validators. - -### `internal/modules/dnd/extract/scenedescriptions` - -The scene-description extractor makes one structured completion for each -accepted chunk and maps its private `kind`, `title`, and `summary` response to -one `dnd.SceneDescription`. It assigns the current chunk ID and exact range, -preserves kind without repair, and trims only title and summary whitespace. -Optional players, party, and glossary references can disambiguate prompt terms -but do not supply evidence. The package owns its private schema, prompt assets, -and mapping fingerprint; deterministic validators own the durable semantic -checks. The durable contract is defined in the -[D&D scene-description artifact contract](../integrations/dnd-scene-description-artifacts.md). - -### `internal/modules/dnd/extract/combatturns` - -The combat extractor requires the `scene_descriptions` reference and resolves -it through the immutable scene-description registry before it resolves NPC -grounding or constructs prompt inputs. It calls the LLM only for an exact -current-chunk match whose kind is `combat`. Exact `narrative`, `recap`, and -`meta` matches return an accepted empty `dnd.CombatTurnList`; missing or -mismatched coverage returns the same result with one bounded unavailable- -classification warning. These deterministic results do not consume retry -attempts. Scene descriptions are control context only and are not passed to the -combat prompt or copied into combat evidence. - -For eligible chunks, the extractor prepares one structured request using the -shared extraction-evidence, identity, campaign-reference, NPC-grounding, and -transcript prompt inputs. It maps the private response to -`dnd.CombatTurnList`, assigns the current source identity, removes exact -duplicate source ranges, and orders turns by valid source-document position -while preserving malformed candidate fields for deterministic validators. Its -package-owned private response schema enforces only the structural JSON -envelope; semantic artifact constraints remain with the validator chain. - -Prepared metadata and checkpoint fingerprints include prompt, response-schema, -mapping, and scene-gate identities. An external scene reference additionally -reports its semantic eligibility digest and count; generated identity remains -framework handoff provenance and dependency state. Neither surface retains -scene prose or payload bytes. The prompt follows the shared D&D extraction -ordering and cache policy documented in -[LLM Runtime](llm.md#dd-extraction-prompt-ordering-and-cache-boundaries). The -package exposes typed registration and is included in the production D&D -registrar with the default combat extraction chain. - -The combat normalizer accepts only the optional structured NPC registry. -Campaign references remain extractor-only LLM context and are not materialized -for deterministic normalization. - -### `internal/modules/dnd/extract/itemevents` - -The item-event extractor prepares one structured request from the accepted -chunk and optional campaign references, then maps private records to -`dnd.ItemEventList` with the current source identity. It declares only optional -`glossary`, `party`, `players`, and deprecated `roster` reference slots; these -can disambiguate names but never supply evidence. It has no NPC, -scene-description, or item-registry dependency. - -The private response schema owns structural transport validation. The extractor -preserves candidate category, holder, quantity, and source-range values for the -deterministic validators, removes exact duplicate ranges, and source-orders -events. The source-reference validator requires citations to fit the current -accepted chunk. Prompt, response-schema, and mapping identities participate in -checkpoint identity. The durable schema is owned separately by -`internal/modules/dnd/codec/itemevents`. - -### `internal/modules/dnd/extract/npcinteractions` - -The NPC interaction extractor requires the structured `npcs` registry slot. It -uses the registry's names-only prompt projection with shared extraction -evidence, identity, and transcript material, then maps private model records to -`dnd.NPCInteractionList` with the current source identity. Registry source -references are never reused as interaction evidence. The private response -schema carries only name, bounded interaction kind, and source-unit ranges; -deterministic validators own registry membership, source validity, and -relatedness. Extract-stage source validation additionally requires every cited -range to be wholly contained in the current materialized chunk. Prompt, schema, -mapping, and the names-only registry projection -participate in checkpoint identity, while generated producer identity remains -framework provenance. - -The shared D&D source-reference order defines canonical evidence ordering. The -domain-owned `internal/modules/dnd/npcinteractions` package defines occurrence -ordering, valid-evidence eligibility, and collision-safe exact identity. The -interaction normalizer and normalized invariants validator consume those -rules, so their production and checking paths cannot drift. Normalizer and -relatedness warning lists use the shared D&D diagnostic cap and emit a final -omission-summary warning when truncated. - -### `internal/modules/dnd/normalize/npcs` - -The NPC normalizer deterministically trims display names, recomputes IDs, -canonicalizes evidence, and consolidates equal comparison keys before semantic -work. Records are eligible for the document-level identity call only when they -have a non-empty comparison key and wholly valid current-document references. -It sends private candidate names and source ranges plus coalesced, cited -transcript windows to its own prompt; stable NPC IDs and the durable artifact -shape are not prompt inputs. - -The private structured response proposes groups of supplied names and a -canonical supplied name. Deterministic comparison-key resolution validates each -group, discards unsafe or overlapping groups, and independently applies safe -ones. Application preserves earliest record order, unions canonical evidence, -and derives the final canonical ID. Invalid structured output and discarded -groups request framework retry with a safe fallback; bounded diagnostics become -durable only on final fallback exhaustion. - -The normalizer records prompt and response-schema identities and digests, -identity and normalization policies, and semantic-context policy and radius as -manifest metadata. Its local checkpoint fingerprints cover the prompt, response -schema, identity policy, normalization policy, and semantic-context policy so a -meaningful behavior change invalidates prior normalize reuse. - -## Merger And Normalizer - -### `internal/modules/generic/merge/appendorder` - -The merger passes typed values to an injected combine function in framework -source-chunk order. The D&D registrar specializes it for all six artifact -lists; each append merger preserves collection presence and order while giving -the result independently owned nested source-reference slices. - -### `internal/modules/generic/normalize/noop` - -The normalizer returns the merged domain value unchanged and is reusable for -any registered artifact type. - -### `internal/modules/dnd/normalize/spells` - -The typed spell normalizer resolves the optional `spell_catalog` reference into -the same immutable SRD-plus-overlay effective catalog used by spell extraction -and catalog validation. It performs no LLM calls. For each spell cast it -canonicalizes recognized names using the catalog's case, whitespace, -apostrophe, and alias rules; canonicalizes source references with the shared -document-aware order; removes only exact reference duplicates; and emits -bounded, scoped warnings for each mutation or unresolved name. - -After those per-cast changes, it collapses only casts with the same canonical -spell, case-folded and whitespace-normalized caster, and complete non-empty -valid source-reference set. It retains the first occurrence and its caster, -source references, and stable order. Unknown names, empty or invalid evidence, -and adjacent or overlapping but different ranges remain unchanged for -validation. - -The normalizer exposes the effective catalog digest as its independently scoped -`effective_catalog` checkpoint fingerprint and reports catalog base ID, digest, -and overlay IDs as manifest metadata. Catalog contents, reference paths, and -raw overlay bytes are not included in either surface. The normalize-stage -reference is stage-local, so an overlay-capable pipeline binds the catalog -independently for extraction and normalization. - -### `internal/modules/dnd/normalize/combatturns` - -The combat normalizer prepares an external NPC registry before execution or -receives a generated registry at the ordered step handoff, then uses the -immutable view during runtime. It display-normalizes actors, -rewrites canonical-name matches for actors, orders and deduplicates source -references, stable-sorts records by source-document position, and collapses -only exact duplicate identities with fully valid evidence. It deep-clones -output storage and emits bounded warnings scoped to merged input indexes. Its -metadata and fingerprints identify the normalization and NPC identity policies. -External bindings may contribute registry -digest/count metadata; generated identity is retained in framework provenance -and dependency fingerprints. The normalizer is included in the production D&D -registrar with the default combat normalization chain. - -### `internal/modules/dnd/normalize/itemevents` - -The item-event normalizer accepts no options or references and makes no LLM -calls. It trims display-edge whitespace in names and holders, canonicalizes -source references, source-orders events, and collapses only exact duplicates -with complete valid evidence. It does not create a ledger, calculate balances, -resolve aliases, infer quantities or holders, or reconcile nearby events. Its -policy fingerprint and bounded warnings identify deterministic normalization; -the matching invariant validator checks the resulting order and duplicate rule. - -### `internal/modules/dnd/normalize/npcinteractions` - -The interaction normalizer requires the same immutable NPC registry. It -canonicalizes exact registry-name matches, orders and de-duplicates source -references, stable-sorts occurrences by source-document position, and collapses -only exact interaction identities with valid evidence. It does not infer, -merge, or summarize distinct occurrences. Its metadata and fingerprints expose -the normalization and NPC identity policies; generated registry identity stays -in framework provenance and checkpoint dependencies. - -### `internal/modules/dnd/normalize/scenedescriptions` - -The scene-description normalizer has no options or references. It validates -each source range against the source document, trims title and summary -whitespace, orders records by source position then ID, removes only exactly -identical records, and rejects conflicting reused IDs or ranges. Its policy -fingerprint identifies this deterministic behavior; the matching invariant -validator checks the normalized result in the production chain. - -## Output Encoder - -### `internal/modules/generic/output/json` - -The JSON encoder sorts normalized results by lane, derives collision-checked -safe logical names, pretty-prints JSON payloads, and assembles the logical index, -manifest, rejected-result, warning, and lane files. Invalid JSON, unsupported -media types, unsafe names, and sanitized-name collisions are errors. - -Its strict `include_chunk_map` option is disabled by default. When enabled, it -validates the framework-supplied accepted chunk map through its codec and adds -the pipeline-wide `chunk-map.json` plus its index descriptor; it does not treat -the map as a lane payload. The external shape is owned by the -[Accepted Chunk Map contract](../integrations/chunk-map.md). - -The encoder returns logical files only. The CLI places them on disk, and the -[JSON output contract](../integrations/json-output.md) defines their external -paths and schemas. - -## Generic Validators - -The generic validator implementations live under -`internal/modules/generic/validate`. - -The unconditional accept and reject validators provide explicit chunk and -typed-artifact variants used primarily for controlled composition and tests. - -The serialized JSON syntax validator uses `encoding/json` to reject malformed -representation bytes. The serialized JSON Schema validator requires schema -bytes, parses the instance and schema with `jsonschema`, and distinguishes -payload rejection from schema loading or compilation errors. The framework -serialized-validation request carries either canonical chunk bytes or artifact -codec bytes according to its target context. Neither validator calls the LLM. - -## D&D Spell Validators - -All four validators receive `dnd.SpellList` directly. The shape validator -rejects a missing list, blank caster or spell names, and empty reference lists. -The catalog validator defers when shape is invalid, then checks every non-empty -spell name against the immutable effective SRD and overlay catalog. It accepts -normalized canonical names and aliases without rewriting the artifact; unknown -names reject the complete result with bounded, stable index/name diagnostics. The -source-reference validator defers malformed shapes, validates every cited -range, and reports all range defects through a bounded aggregate while -preserving `invalid_source_refs`. The relatedness validator resolves all cited -ranges through the shared document-order traversal, then warns when a normalized -consecutive spell-name token sequence is absent from the cited source text. -Invalid shape -or cited ranges produce no relatedness warnings; the shape and source-reference -validators own those defects. - -These validators are deterministic. Shape, source-reference, and relatedness -each expose a local semantic `policy` checkpoint fingerprint. The catalog -validator instead exposes its effective catalog digest as its semantic -checkpoint identity and does not add a separate policy fingerprint. Their -selectable keys and production order are defined in -[Configuration](../config.md#implemented-production-validators); their durable -payload rules are defined in the -[artifact contract](../integrations/dnd-spell-artifacts.md). - -## D&D NPC Validators - -NPC shape validation checks the required ID and name strings, list presence, and source-reference -shape. The source-reference validator defers malformed shapes, checks -current-document identity, unit existence, and range ordering, and reports all -defects through bounded aggregates. Source relatedness uses the shared -document-order traversal and normalized consecutive-token matching, emitting at -most one bounded warning per record when the canonical name does not occur near -its cited text. Invalid shape or cited ranges produce no relatedness warnings. -Normalize identity validation checks deterministic IDs, canonical names, and -duplicate canonical-name or ID ownership. -All are deterministic and expose the policy fingerprints used by the -production chains. - -## D&D Combat Validators - -Combat shape validation owns the required list, actor, supported turn kind, and -non-empty source-reference collection. Combat source-reference validation defers invalid -shape, checks source identity, unit existence, and range order, and reports all -defects through bounded aggregates. Combat source-relatedness defers invalid -shape or ranges, uses the shared traversal to combine overlapping cited units -in document order, and emits at most one bounded advisory warning per turn for -an unrelated actor. Actors use normalized consecutive-token matching. The -normalized-invariants validator owns actor display normalization, canonical -source-reference order, chronology, and exact duplicate identity; it defers -shape and source-reference failures. All four validators are deterministic and -expose local policy fingerprints. In the registered defaults, JSON syntax runs -first; combat shape, normalized invariants when applicable, and source-reference -validation precede JSON Schema validation; warning-only relatedness runs last. - -## D&D Item-Event Validators - -Item-event shape validation owns the required list, non-empty name, supported -category, category-and-holder combination, positive optional quantity, and -non-empty source-reference collection. Source-reference validation defers -malformed shapes, checks current-source identity and ordered ranges, and during -extraction requires every citation to fit the accepted chunk. Relatedness is -advisory and warning-only: it checks the event name against cited transcript -text while deferring malformed candidates and invalid ranges to their blocking -owners. The normalized-invariants validator owns display normalization, -canonical source-reference order, chronology, and exact duplicate identity. -All four validators are deterministic and expose policy fingerprints. The -registered chains run syntax and blocking checks before durable JSON Schema; -relatedness remains last. - -## D&D NPC Interaction Validators - -Interaction shape validation owns the required list, registry name, supported -kind, and non-empty source-reference collection. Registry validation checks -exact membership in the required immutable NPC registry. Source-reference and -relatedness validation use the current transcript only; malformed candidates -are deferred by later validators and produce no relatedness warning. The -normalized-invariants validator owns canonical registry names, source-reference -order, chronology, and exact duplicate identity. The production chains run -shape, registry, and source-reference checks before JSON Schema validation; -relatedness remains warning-only and last. - -## D&D Scene Description Validators - -Scene-description shape validation owns the non-empty list, trimmed ID and -prose, closed kind, and basic source-reference shape. Extract-stage source -validation additionally requires the one record to attach exactly to the -current accepted chunk; later source validation checks source membership. -Relatedness checks the title and summary independently against only their cited -transcript range and emits bounded advisory warnings. The normalized-invariants -validator owns ordering, exact duplicate elimination, and conflicting ID or -range detection. The production chains run shape and source-reference checks -before JSON Schema validation; the warning-only relatedness check is last. - -## Production Registration - -Production composition occurs through family registrars. The CLI allocates one -complete framework registry set and one LLM asset registry. It invokes -`internal/modules/generic/register`, -`internal/modules/seriatim/register`, and `internal/modules/dnd/register` in -that order, then exposes the matching catalog for resolution. The generic and -Seriatim registrars own their production leaf registrations. The D&D registrar -owns D&D leaf registrations, typed spell, NPC, combat, item-event, interaction, and scene-description default-validator -chains, typed append-order specializations, and D&D prompt/schema asset -collection. Its registration helpers group module, validator, prompt-asset, and -chain composition while retaining artifact-specific merge and clone behavior in -the registrar. - -Concrete implementation packages do not import generic implementation -packages directly. A concrete family's `register` package is its composition -point for specializing reusable generic implementations, while the generic -registrar composes only generic children. - -Core and framework production packages do not import production extensions. -CLI production code is the sole application composition root for extensions -and imports only exact family registrar packages. Other production packages, -including commands and newly introduced package trees, do not import module -packages directly. Compatibility tests in the CLI, core, and framework trees -may import roots and implementation leaves directly. Other non-module tests do -not receive that exemption. White-box tests within module families retain the -production family boundaries. `internal/modules/integration` is test -infrastructure: its black-box tests may compose multiple families, but it is -not a production module family or production dependency target. - -## Adding An Extension - -When adding a production module or validator: - -1. implement the stage or validator contract and package-local key; -2. expose and test its spec, constructor, and registration function; -3. keep format or domain parsing inside the concrete package; -4. add package-owned prompt/schema assets when the extension is LLM-backed; - new LLM-backed D&D extraction modules must follow the stable-to-variable - prompt ordering, shared-asset ownership, and cache-boundary policy in - [LLM Runtime](llm.md#dd-extraction-prompt-ordering-and-cache-boundaries), or - document the implemented exception and its evidence there; -5. register it through its package-family registrar and add a default chain - there only when production policy requires one; -6. add resolution and composition coverage for capabilities, options, - references, and validation behavior; -7. update the selectable-key catalog in [Configuration](../config.md), the - relevant external contract, this inventory, and maintained examples when - user-visible behavior changes. - -Do not add the extension to `docs/development.md`; that file routes by task and -does not inventory implementations. - -### D&D Extractor Contract - -New D&D extractors preserve these package-owned responsibilities: - -- Reject unknown options unless an option namespace is intentionally - extensible, and use shared common preflight while retaining receiver, - dependency, and lane-specific checks locally. -- Return independently owned results and exposed metadata that callers may - safely mutate. -- Keep the private response DTO, structural response schema and its identity, - provider-response mapping, durable artifact conversion, and lane diagnostics - in the owning package. -- Include every stable semantic input that can change durable output in - checkpoint identity. Consider prompt, schema, mapping, canonicalization, - prepared reference projections, identity, normalization, and trimming where - applicable. -- Add focused behavioral coverage where the lane's risks warrant it, including - construction and registration, option rejection, preflight, provider - failures, structured decoding, mapping and ownership, prompt - role/input/cache order, and checkpoint invalidation. - -Prompt ordering, shared-asset ownership, cache boundaries, and private-schema -rules are defined in [LLM Runtime](llm.md#dd-extraction-prompt-ordering-and-cache-boundaries). -[Pipeline Internals](pipeline.md#reference-materialization) owns reference -materialization, and its [checkpoint hooks](pipeline.md#checkpoint-and-debug-hooks) -define checkpoint behavior. Follow [Architecture](../policy/architecture.md#source-and-domain-boundaries) -for ownership boundaries and the [Testing Policy](../policy/testing.md) when -selecting durable coverage. This contract intentionally does not prescribe -prompt prose or length, hashes, test counts, filenames, fixture layouts, or -generic implementation builders. - -## Tests To Inspect - -- Package-local `*_test.go` files under the module or validator being changed. -- `internal/framework/pipeline/typed_resolution_test.go`: typed registry, spec, - and heterogeneous artifact composition. -- `internal/framework/pipeline/profile_test.go`: framework binding defaults and - profile resolution. -- `internal/cli/production_contract_test.go`: production catalog, config - resolution, and composition smoke coverage. -- `internal/cli/example_contract_test.go`: maintained example ownership. -- `internal/framework/promptfs/*_test.go` and - `internal/modules/dnd/shared/*_test.go`: shared prompt and reference assembly. -- `internal/modules/integration/*_test.go`: black-box composition across - production extension domains. +# Module Internals + +This guide owns the mechanics for implementing and registering production +modules. [Configuration](../config.md) owns selectable keys, binding syntax, +reference configuration, and default validator chains. Durable input and output +shapes belong in [integration contracts](../integrations/). + +The D&D family has additional shared conventions and domain-specific +exceptions. See [D&D Module Internals](dnd.md) rather than adding them here. + +## Module Boundary + +A module is a typed implementation registered for one pipeline stage. Its +`ModuleSpec` is the public-to-the-framework declaration of its stable key, +stage, required and provided capabilities, artifact kind, and accepted +reference slots. The framework uses that declaration to resolve a configured +binding before it builds the implementation. + +Implementations that accept options must provide both an option validator and +a builder. The validator is used while resolving configuration; the builder +decodes the same options and constructs the implementation from the prepared +`BuildRequest`. Reject unknown options in both paths. A builder receives only +the dependencies and materialized references that the framework prepared for +that operation, so it must not re-read configuration or files. + +Registry helpers register the typed builder for a stage-specific registry. +They are preferable to hand-written untyped registration because they retain +the artifact type at the framework boundary. Registrars validate the registries +they need, register each leaf implementation, and add any family-owned assets +or default validator chains. They return contextual errors so production +composition fails at startup rather than at the first run. + +## Production Composition + +Production composition is intentionally split by family: + +- The generic registrar provides the unit chunker, generic JSON validators, + and JSON output encoder. +- The Seriatim registrar provides the transcript input adapter. Its external + input behavior is defined by the [Seriatim contract](../integrations/seriatim.md). +- The D&D registrar provides its codecs, extractors, mergers, normalizers, + validators, prompt assets, and default chains. Its behavioral conventions + are documented in [D&D Module Internals](dnd.md). + +The CLI owns the composition that invokes these registrars. A module package +may register its own family but must not assemble the CLI or make framework +packages depend on production extensions. + +## Adding Or Changing A Module + +1. Choose the pipeline stage and the typed artifact boundary. Put external + input or durable artifact formats in the relevant integration contract, + not in this guide or in a private LLM response type. +2. Define a stable `ModuleSpec` with the exact capabilities and reference + slots needed for the operation. Model a producer/consumer handoff as an + artifact-compatible slot; configuration then chooses an external file or a + generated binding. +3. Implement strict option decoding, construction, and the typed stage + interface. Preserve caller ownership: do not retain mutable request data + and return defensive copies where an implementation exposes stored data. +4. Register the module through its typed registry helper and add it to the + owning family registrar. Add a default validator chain only when that + family owns the behavior; otherwise require an explicit compatible chain. +5. Update the selectable-key and chain reference in + [Configuration](../config.md#production-module-keys), the applicable + integration contract, and focused tests. Keep the configuration document + as the sole list of production keys and validator order. + +## Validation And References + +Validators operate on the value produced at their configured stage. A default +chain is ordered behavior, not a set: JSON parsing, structural checks, +domain-specific checks, durable-schema checks, and advisory checks may have +different responsibilities and failure handling. The active default chains and +override rules are maintained in +[Configuration](../config.md#production-validator-keys-and-default-chains). + +Reference slots are part of the module specification. They describe the +accepted artifact kind, media type, size, and whether a binding is required; +the framework validates those constraints before construction. An external +reference is materialized during preparation. A generated reference is a +compatible normalized artifact handed from an earlier pipeline step at +operation time. The configuration reference rules, including precedence and +ordered-handoff requirements, are maintained in +[Configuration](../config.md#references-and-ordered-handoffs). + +## Focused Verification + +Exercise the leaf implementation and its registration path when changing a +module. Registry and registrar tests cover duplicate keys, required registries, +and typed construction; pipeline resolution tests cover capabilities, options, +and reference compatibility. Domain packages should additionally test their +codecs, validators, normalizers, and any integration handoffs they own. + +Run the affected package tests while iterating. The complete module suite is: + +~~~sh +go test ./internal/modules/... +~~~ diff --git a/docs/internal/overview.md b/docs/internal/overview.md index dcae244..97cc000 100644 --- a/docs/internal/overview.md +++ b/docs/internal/overview.md @@ -48,9 +48,11 @@ the CLI composition boundary. composition, and path safety. - [LLM Runtime](llm.md): structured completion, scheduling, prompt assets, profiles, and secret handling. -- [Module Internals](modules.md): extension registration and production module - mechanics. Durable D&D and Seriatim data shapes remain in the - [integration contracts](../integrations/). +- [Module Internals](modules.md): generic extension registration, module + construction, validation, and reference mechanics. +- [D&D Module Internals](dnd.md): shared D&D extractor conventions, generated + reference projections, and lane-specific exceptions. Durable D&D and + Seriatim data shapes remain in the [integration contracts](../integrations/). Use this map to find an owner, then read the focused document and its tests before changing behavior.