Document D&D module conventions
This commit is contained in:
@@ -1,728 +1,99 @@
|
||||
# Module And Validator Internals
|
||||
|
||||
Production module and validator implementations live under their domain-first
|
||||
trees in `internal/modules`.
|
||||
The selectable keys, configuration options, reference slots, and default
|
||||
validator chain are canonical in the
|
||||
[module](../config.md#implemented-production-modules) and
|
||||
[validator](../config.md#implemented-production-validators) catalogs in
|
||||
Configuration.
|
||||
|
||||
## Extension Pattern
|
||||
|
||||
A stage module package provides a stable key, constructor, contract
|
||||
implementation, `ModuleSpec`, `Register`, and focused behavior and registration
|
||||
tests. A validator package follows the same pattern with `ValidatorSpec` and the
|
||||
validator registry. Package-family registrars compose those leaf registrations
|
||||
into the production catalog and own family-level policy such as default
|
||||
validator chains and prompt asset collection.
|
||||
|
||||
Production input, chunk, output, and D&D spell-, NPC-, combat-, item-event-, interaction-, and scene-description-extract packages
|
||||
register strict option decoders and run-local builders. Preparation decodes their options into
|
||||
implementation-owned values and injects dependencies plus the materialized
|
||||
reference set for the selected target. Each builder receives an isolated clone
|
||||
of that set; input and output builders receive no references. The spell, NPC,
|
||||
combat, item-event, interaction, and scene-description extractors are typed over the canonical D&D model. D&D validators, merge,
|
||||
and normalize use typed variants; JSON representation validators use serialized
|
||||
requests; and unconditional validators expose separate chunk and typed
|
||||
variants. The D&D production registrar registers the canonical typed spell,
|
||||
NPC, combat, item-event, interaction, and scene-description implementations, including their kind-specific merge and
|
||||
normalize behavior.
|
||||
|
||||
For D&D artifact defaults, generic JSON syntax validation runs first. Rejecting
|
||||
domain validators then own semantic diagnostics before generic JSON Schema
|
||||
validation provides the final rejecting representation backstop; warning-only
|
||||
relatedness validators run last. This default composition does not reorder an
|
||||
explicitly configured validator chain.
|
||||
|
||||
Prepared extractors, extract validators, and codecs may be reused concurrently
|
||||
by the run-wide extract pool. Production implementations are immutable after
|
||||
construction: they retain only typed options, immutable assets, or the shared
|
||||
concurrency-safe LLM client. Implementations that introduce mutable state must
|
||||
synchronize that state without creating a separate provider scheduler.
|
||||
|
||||
Specs expose capability and execution metadata without constructing an
|
||||
implementation. Registry entries separately expose option validation and
|
||||
run-local construction. Chunk, extract, merge, and normalize modules that accept
|
||||
auxiliary material declare identical reference slots from both
|
||||
`ReferenceSlots()` and `ModuleSpec().ReferenceSlots`; registration tests enforce
|
||||
that agreement. Runtime delivery uses the corresponding stage request's
|
||||
`References` field.
|
||||
|
||||
LLM-backed extensions own their prompt definitions and response schemas under
|
||||
package-local embedded assets. Shared filesystem composition belongs in
|
||||
`internal/framework/promptfs`; reusable D&D prompt fragments, reference
|
||||
declarations, prompt-input assembly, and source-unit/citation helpers belong in
|
||||
`internal/modules/dnd/shared`, which owns operation-scoped indexed
|
||||
source-reference validation, citation traversal, ordering and canonicalization,
|
||||
plus bounded D&D diagnostics. The
|
||||
D&D scene chunker and spell, NPC, combat-turn, item-event, NPC-interaction, and scene-description extractors use ordered
|
||||
package-local prompt manifests for both rendering and prompt fingerprinting, so
|
||||
only the shared fragments each prompt actually renders participate in either
|
||||
operation. Extraction prompts place stable shared and lane-specific context
|
||||
before the variable transcript and use shared assets for wording common across
|
||||
lanes. The canonical ordering and cache-boundary policy is documented in
|
||||
[LLM Runtime](llm.md#dd-extraction-prompt-ordering-and-cache-boundaries). Stage
|
||||
contracts expose only Notarius structured-completion types, not Scriptorium
|
||||
public types.
|
||||
|
||||
The shared `PrepareChunkExtraction` helper owns common extraction preflight and
|
||||
transcript material preparation for the spell, NPC, combat-turn, item-event,
|
||||
NPC-interaction, and scene-description extractors. It validates common request
|
||||
state, clones supplied source metadata, falls back to the materialized chunk
|
||||
when content is absent, checks that content remains chunk-identical, and fills
|
||||
only the common default fields. Extractors retain receiver, dependency, and
|
||||
lane-specific checks locally and wrap helper errors with their module context.
|
||||
|
||||
Reference material may inform a module or prompt but must not become source
|
||||
evidence. The resolver and materializer behavior is described in
|
||||
[Pipeline Internals](pipeline.md#reference-materialization).
|
||||
|
||||
## Domain Reference Data
|
||||
|
||||
### `internal/modules/dnd/spells/catalog`
|
||||
|
||||
The spell catalog package owns the embedded, versioned D&D 5e 2014 SRD spell
|
||||
reference data. Its strict JSON asset contains one canonical record per spell,
|
||||
including spell level and all applicable class memberships. `LoadSRD5E2014`
|
||||
validates catalog identity, provenance metadata, ordering, uniqueness, levels,
|
||||
classes, aliases, and lookup-key collisions before exposing immutable copies.
|
||||
|
||||
Lookup is case-insensitive and normalizes whitespace and common apostrophe
|
||||
variants while preserving source punctuation in canonical display names. The
|
||||
catalog contains 319 unique spells and 779 class memberships. Source and
|
||||
license details live beside the asset in `SOURCES.md`. This domain-owned data is
|
||||
separate from `internal/modules/dnd/shared`, which is reserved for reusable
|
||||
prompt and source-reference machinery.
|
||||
|
||||
`ResolveEffectiveCatalog` builds the immutable recognition view used by the
|
||||
spell extractor and catalog validator. It starts with the embedded SRD catalog
|
||||
and optionally applies one strict JSON overlay from the `spell_catalog` item in
|
||||
a materialized reference set. Overlay catalogs are ordered by ID, may add names
|
||||
and aliases, and may augment an existing canonical spell without replacing its
|
||||
display name. Cross-spell lookup collisions are errors. The effective view
|
||||
exposes sorted canonical names, normalized lookup, overlay identities, and a
|
||||
semantic digest; overlay content remains contextual reference material rather
|
||||
than source evidence. Its external JSON contract is defined in the
|
||||
[spell-catalog overlay contract](../integrations/dnd-spell-catalog-overlays.md).
|
||||
|
||||
### `internal/modules/dnd/npcs/identity`, `internal/modules/dnd/npcs/registry`, and `internal/modules/dnd/codec/npcs`
|
||||
|
||||
The NPC identity package owns Unicode comparison keys, deterministic
|
||||
`npc:sha256:` IDs, display normalization, and whole-registry collision issues.
|
||||
The registry package resolves one optional normalized artifact through the
|
||||
strict codec, validates whole-registry identity, canonicalizes its JSON, and
|
||||
provides immutable records, a names-only prompt projection, distinct durable
|
||||
and projection digests, count, and exact canonical-name lookup. External files cross this boundary during
|
||||
preparation; generated artifacts cross it at the ordered step handoff. It owns
|
||||
the `npcs` slot and its bounded, content-safe validation failures. NPC source
|
||||
references are durable provenance and are not treated as evidence for a
|
||||
consuming pipeline. The NPC codec owns the strict durable `dnd/npc-list` JSON
|
||||
boundary and exposes candidate versus approved encode/decode operations. The
|
||||
shared `internal/modules/dnd/codec/candidatejson` package supplies strict typed
|
||||
candidate JSON mechanics; each artifact codec retains its own durable schema
|
||||
and approved-value policy.
|
||||
|
||||
### `internal/modules/dnd/scenedescriptions/registry`
|
||||
|
||||
The scene-description registry owns the required `scene_descriptions` control
|
||||
reference used by combat extraction. It decodes exactly one approved scene-list
|
||||
artifact through the scene-description codec and retains only scene ID, exact
|
||||
source reference, and kind. Titles, summaries, original bytes, paths, and
|
||||
prompt material do not cross this domain boundary.
|
||||
|
||||
An external reference is validated during preparation; an unbound seed is
|
||||
permitted only while a configured generated reference awaits the ordered
|
||||
handoff. At operation time, a generated artifact overrides the seed and is
|
||||
resolved into an immutable view safe for concurrent extract jobs. Matching is
|
||||
strictly exact by chunk ID, source ID, start unit ID, and end unit ID, producing
|
||||
an exact, missing, or mismatched result. Only an exact result exposes kind.
|
||||
|
||||
The registry's semantic eligibility digest is derived from a sorted projection
|
||||
of ID, exact range, and kind. It ignores titles, summaries, and input order;
|
||||
the unbound view has a stable empty projection digest. Combat extractor
|
||||
metadata and checkpoint identity use this semantic boundary for external
|
||||
references, while generated artifact identity and dependencies remain owned by
|
||||
the framework handoff.
|
||||
|
||||
The `internal/modules/dnd/codec/combatturns` package owns the durable
|
||||
`dnd/combat-turn-list` schema and candidate versus approved JSON boundary. It
|
||||
is registered by the production D&D family registrar for the selectable combat
|
||||
lane.
|
||||
|
||||
The `internal/modules/dnd/codec/itemevents` package owns the durable
|
||||
`dnd/item-event-list` schema and candidate versus approved JSON boundary. It is
|
||||
registered by the production D&D family registrar. Its external contract is
|
||||
defined in the [D&D item-event artifact contract](../integrations/dnd-item-event-artifacts.md).
|
||||
|
||||
The `internal/modules/dnd/codec/npcinteractions` package owns the durable
|
||||
`dnd/npc-interaction-list` schema and candidate versus approved JSON boundary.
|
||||
It is registered by the production D&D family registrar for the selectable
|
||||
interaction lane. Its external contract is documented in the
|
||||
[D&D NPC interaction artifact contract](../integrations/dnd-npc-interaction-artifacts.md).
|
||||
|
||||
The `internal/modules/dnd/codec/scenedescriptions` package owns the durable
|
||||
`dnd/scene-description-list` schema and candidate versus approved JSON boundary.
|
||||
It is registered by the production D&D family registrar. Its external contract
|
||||
is documented in the
|
||||
[D&D scene-description artifact contract](../integrations/dnd-scene-description-artifacts.md).
|
||||
|
||||
## Input Adapter
|
||||
|
||||
### `internal/modules/seriatim/input/transcript`
|
||||
|
||||
The adapter decodes the supported transcript JSON, selects the source identity,
|
||||
computes canonical source provenance, validates segments, and maps each segment
|
||||
into a generic source unit with a self-reference plus speaker and timestamp
|
||||
metadata. It accepts no module options. Its spec advertises the transcript
|
||||
capabilities consumed by D&D modules.
|
||||
|
||||
Parsing is strict about required values and duplicate unit IDs but deliberately
|
||||
ignores unrelated Seriatim fields. The external format and derived-identity
|
||||
rules are defined in the
|
||||
[Seriatim contract](../integrations/seriatim.md).
|
||||
|
||||
## Chunkers
|
||||
|
||||
Chunkers implement `contracts.Chunker.Plan`. A plan identifies ordered source
|
||||
unit ranges and may carry optional namespaced JSON annotations; it does not
|
||||
contain materialized chunk content. The framework canonicalizes annotations,
|
||||
validates ranges against the current source, and materializes chunk IDs,
|
||||
indexes, references, content, units, and generic metadata. Materialized source
|
||||
unit metadata is independently owned. Annotation
|
||||
namespaces remain optional data: generic framework code and downstream modules
|
||||
must not require D&D scene annotations or import `dnd/scenes`.
|
||||
|
||||
### `internal/modules/generic/chunk/units`
|
||||
|
||||
The generic chunker validates the source document and returns ranges over units
|
||||
in configured windows. Overlap changes the next window start but never reorders
|
||||
units. Framework materialization derives the resulting chunk identity and
|
||||
generic metadata from those ranges.
|
||||
|
||||
The accepted options and defaults are defined in
|
||||
[Configuration](../config.md#implemented-production-modules). Generic
|
||||
framework validation canonicalizes the returned unit slices before extraction.
|
||||
The chunker decodes its options during construction and retains only the typed
|
||||
window settings used by `Plan`.
|
||||
|
||||
### `internal/modules/dnd/chunk/scenes`
|
||||
|
||||
The scene chunker prepares a structured Scriptorium request from the full
|
||||
transcript, session, and optional D&D reference inputs. It validates the model's
|
||||
inclusive source-unit endpoints against document position and converts them
|
||||
into deterministic plan ranges. Preparation injects the shared structured LLM
|
||||
client into the chunker; `Plan` supplies only the run-specific profile, session,
|
||||
source, references, and metadata.
|
||||
|
||||
Scene validation requires sequential, contiguous, non-overlapping coverage from
|
||||
the first source unit through the last. Its private response contains only the
|
||||
boundary endpoints; the accepted plan has no D&D-specific annotations and
|
||||
produces no boundary warnings. Malformed structured output is returned as an
|
||||
error; there is no fallback chunker.
|
||||
|
||||
The package embeds its prompt and response schema and reports their non-secret
|
||||
identity and hashes through singleton module metadata. Shared D&D assets supply
|
||||
reference declarations and prompt inputs; their user-facing keys and accepted
|
||||
file types remain canonical in [Configuration](../config.md).
|
||||
|
||||
## Extractor
|
||||
|
||||
### `internal/modules/dnd/extract/spells`
|
||||
|
||||
The spell extractor prepares a structured request from one chunk, the
|
||||
chunk-scoped source input, the session, and optional D&D reference inputs. It
|
||||
decodes the model response, assigns the generic source identity to every source
|
||||
reference, canonicalizes duplicate references, orders spell casts by their
|
||||
earliest valid source-document position, and returns `dnd.SpellList`.
|
||||
|
||||
Its private response schema admits only the structural transport envelope:
|
||||
required fields, JSON types, array and object shapes, and unknown-field
|
||||
rejection. It maps integer source-unit candidates directly without repairing
|
||||
semantic values, so the deterministic shape, catalog, and source-reference
|
||||
validators own blank values, empty evidence, and invalid or unresolved ranges.
|
||||
|
||||
The extractor owns its private model-response DTO, embedded prompt, LLM response
|
||||
schema, strict option decoder, injected shared LLM client, and prompt/schema
|
||||
manifest metadata. During preparation it resolves the optional `spell_catalog`
|
||||
reference into an immutable effective catalog and adds a generated
|
||||
canonical-name-only JSON input to every structured completion request. Overlay
|
||||
failures therefore stop construction before source parsing or an LLM call;
|
||||
campaign references remain separate disambiguation inputs and never become
|
||||
source evidence.
|
||||
|
||||
The prompt includes only actual casting events and unambiguous declared casting
|
||||
attempts. Spell mentions, plans, rules discussion, and catalog matches without
|
||||
a casting event are excluded. Shared extraction-evidence and identity rules
|
||||
require transcript-supported caster and spell facts, while the catalog,
|
||||
campaign references, and NPC names only disambiguate source text. Structural
|
||||
source validation remains deterministic; semantic evidence sufficiency is
|
||||
enforced through extraction policy and evaluation.
|
||||
|
||||
Both the extractor and deterministic catalog validator expose
|
||||
the effective base-plus-overlay semantic digest as scoped prepared-component
|
||||
checkpoint identity. Raw overlay provenance independently covers file-byte
|
||||
changes, while the semantic digest also invalidates reuse when the embedded
|
||||
catalog or catalog composition changes. The extractor additionally fingerprints
|
||||
its complete prompt assets and private response schema, so either semantic
|
||||
contract changing invalidates previously recorded extraction checkpoints. The
|
||||
separate `internal/modules/dnd/codec/spells` package
|
||||
owns the durable schema and stable JSON representation for artifact kind
|
||||
`dnd/spell-list`. The runner keeps the result typed through validators and later
|
||||
stages, using the codec only for checkpoint, debug, and output boundaries.
|
||||
Shared D&D helpers keep prompt input names and source-unit reference conversion
|
||||
consistent with the scene chunker.
|
||||
|
||||
The extractor also declares the optional `npcs` registry slot and consumes the
|
||||
immutable registry boundary from `internal/modules/dnd/npcs/registry`. An
|
||||
external registry is prepared before execution; a generated registry is
|
||||
validated and supplied at operation time. Bound external registries add only
|
||||
the full `npc_registry_digest` and `npc_count` to module metadata. The local
|
||||
`npc_registry` checkpoint fingerprint always covers the names-only projection,
|
||||
including its exact unbound value. Generated bindings are represented by
|
||||
framework handoff provenance and dependency fingerprints. The unbound prompt
|
||||
input is exactly `{"npcs":[]}` and has no registry provenance.
|
||||
The shared NPC grounding fragment is placed immediately after the common
|
||||
campaign reference message and is included in the spell prompt fingerprint.
|
||||
|
||||
The durable payload and manifest metadata shapes are defined in the
|
||||
[D&D spell artifact contract](../integrations/dnd-spell-artifacts.md).
|
||||
|
||||
### `internal/modules/dnd/extract/npcs`
|
||||
|
||||
The NPC extractor maps private model output to the canonical `dnd.NPCList`,
|
||||
assigns source identity and deterministic NPC IDs, and preserves source
|
||||
references for deterministic validation. It uses the shared campaign
|
||||
references only for disambiguation and does not consume the optional NPC
|
||||
registry slot. Its prompt and private response schema are package-owned. The
|
||||
private response contains only a name and model-facing evidence ranges for each
|
||||
record; anonymous groups, generic roles, invented labels, descriptions,
|
||||
aliases, and relationships are outside its contract. The
|
||||
prompt follows the shared D&D extraction ordering and cache policy documented
|
||||
in [LLM Runtime](llm.md#dd-extraction-prompt-ordering-and-cache-boundaries).
|
||||
|
||||
The private response schema owns only structural transport validation and maps
|
||||
integer source-unit candidates unchanged. Required semantic content, non-empty
|
||||
evidence, and valid source ranges are rejected by the deterministic shape and
|
||||
source-reference validators.
|
||||
|
||||
### `internal/modules/dnd/extract/scenedescriptions`
|
||||
|
||||
The scene-description extractor makes one structured completion for each
|
||||
accepted chunk and maps its private `kind`, `title`, and `summary` response to
|
||||
one `dnd.SceneDescription`. It assigns the current chunk ID and exact range,
|
||||
preserves kind without repair, and trims only title and summary whitespace.
|
||||
Optional players, party, and glossary references can disambiguate prompt terms
|
||||
but do not supply evidence. The package owns its private schema, prompt assets,
|
||||
and mapping fingerprint; deterministic validators own the durable semantic
|
||||
checks. The durable contract is defined in the
|
||||
[D&D scene-description artifact contract](../integrations/dnd-scene-description-artifacts.md).
|
||||
|
||||
### `internal/modules/dnd/extract/combatturns`
|
||||
|
||||
The combat extractor requires the `scene_descriptions` reference and resolves
|
||||
it through the immutable scene-description registry before it resolves NPC
|
||||
grounding or constructs prompt inputs. It calls the LLM only for an exact
|
||||
current-chunk match whose kind is `combat`. Exact `narrative`, `recap`, and
|
||||
`meta` matches return an accepted empty `dnd.CombatTurnList`; missing or
|
||||
mismatched coverage returns the same result with one bounded unavailable-
|
||||
classification warning. These deterministic results do not consume retry
|
||||
attempts. Scene descriptions are control context only and are not passed to the
|
||||
combat prompt or copied into combat evidence.
|
||||
|
||||
For eligible chunks, the extractor prepares one structured request using the
|
||||
shared extraction-evidence, identity, campaign-reference, NPC-grounding, and
|
||||
transcript prompt inputs. It maps the private response to
|
||||
`dnd.CombatTurnList`, assigns the current source identity, removes exact
|
||||
duplicate source ranges, and orders turns by valid source-document position
|
||||
while preserving malformed candidate fields for deterministic validators. Its
|
||||
package-owned private response schema enforces only the structural JSON
|
||||
envelope; semantic artifact constraints remain with the validator chain.
|
||||
|
||||
Prepared metadata and checkpoint fingerprints include prompt, response-schema,
|
||||
mapping, and scene-gate identities. An external scene reference additionally
|
||||
reports its semantic eligibility digest and count; generated identity remains
|
||||
framework handoff provenance and dependency state. Neither surface retains
|
||||
scene prose or payload bytes. The prompt follows the shared D&D extraction
|
||||
ordering and cache policy documented in
|
||||
[LLM Runtime](llm.md#dd-extraction-prompt-ordering-and-cache-boundaries). The
|
||||
package exposes typed registration and is included in the production D&D
|
||||
registrar with the default combat extraction chain.
|
||||
|
||||
The combat normalizer accepts only the optional structured NPC registry.
|
||||
Campaign references remain extractor-only LLM context and are not materialized
|
||||
for deterministic normalization.
|
||||
|
||||
### `internal/modules/dnd/extract/itemevents`
|
||||
|
||||
The item-event extractor prepares one structured request from the accepted
|
||||
chunk and optional campaign references, then maps private records to
|
||||
`dnd.ItemEventList` with the current source identity. It declares only optional
|
||||
`glossary`, `party`, `players`, and deprecated `roster` reference slots; these
|
||||
can disambiguate names but never supply evidence. It has no NPC,
|
||||
scene-description, or item-registry dependency.
|
||||
|
||||
The private response schema owns structural transport validation. The extractor
|
||||
preserves candidate category, holder, quantity, and source-range values for the
|
||||
deterministic validators, removes exact duplicate ranges, and source-orders
|
||||
events. The source-reference validator requires citations to fit the current
|
||||
accepted chunk. Prompt, response-schema, and mapping identities participate in
|
||||
checkpoint identity. The durable schema is owned separately by
|
||||
`internal/modules/dnd/codec/itemevents`.
|
||||
|
||||
### `internal/modules/dnd/extract/npcinteractions`
|
||||
|
||||
The NPC interaction extractor requires the structured `npcs` registry slot. It
|
||||
uses the registry's names-only prompt projection with shared extraction
|
||||
evidence, identity, and transcript material, then maps private model records to
|
||||
`dnd.NPCInteractionList` with the current source identity. Registry source
|
||||
references are never reused as interaction evidence. The private response
|
||||
schema carries only name, bounded interaction kind, and source-unit ranges;
|
||||
deterministic validators own registry membership, source validity, and
|
||||
relatedness. Extract-stage source validation additionally requires every cited
|
||||
range to be wholly contained in the current materialized chunk. Prompt, schema,
|
||||
mapping, and the names-only registry projection
|
||||
participate in checkpoint identity, while generated producer identity remains
|
||||
framework provenance.
|
||||
|
||||
The shared D&D source-reference order defines canonical evidence ordering. The
|
||||
domain-owned `internal/modules/dnd/npcinteractions` package defines occurrence
|
||||
ordering, valid-evidence eligibility, and collision-safe exact identity. The
|
||||
interaction normalizer and normalized invariants validator consume those
|
||||
rules, so their production and checking paths cannot drift. Normalizer and
|
||||
relatedness warning lists use the shared D&D diagnostic cap and emit a final
|
||||
omission-summary warning when truncated.
|
||||
|
||||
### `internal/modules/dnd/normalize/npcs`
|
||||
|
||||
The NPC normalizer deterministically trims display names, recomputes IDs,
|
||||
canonicalizes evidence, and consolidates equal comparison keys before semantic
|
||||
work. Records are eligible for the document-level identity call only when they
|
||||
have a non-empty comparison key and wholly valid current-document references.
|
||||
It sends private candidate names and source ranges plus coalesced, cited
|
||||
transcript windows to its own prompt; stable NPC IDs and the durable artifact
|
||||
shape are not prompt inputs.
|
||||
|
||||
The private structured response proposes groups of supplied names and a
|
||||
canonical supplied name. Deterministic comparison-key resolution validates each
|
||||
group, discards unsafe or overlapping groups, and independently applies safe
|
||||
ones. Application preserves earliest record order, unions canonical evidence,
|
||||
and derives the final canonical ID. Invalid structured output and discarded
|
||||
groups request framework retry with a safe fallback; bounded diagnostics become
|
||||
durable only on final fallback exhaustion.
|
||||
|
||||
The normalizer records prompt and response-schema identities and digests,
|
||||
identity and normalization policies, and semantic-context policy and radius as
|
||||
manifest metadata. Its local checkpoint fingerprints cover the prompt, response
|
||||
schema, identity policy, normalization policy, and semantic-context policy so a
|
||||
meaningful behavior change invalidates prior normalize reuse.
|
||||
|
||||
## Merger And Normalizer
|
||||
|
||||
### `internal/modules/generic/merge/appendorder`
|
||||
|
||||
The merger passes typed values to an injected combine function in framework
|
||||
source-chunk order. The D&D registrar specializes it for all six artifact
|
||||
lists; each append merger preserves collection presence and order while giving
|
||||
the result independently owned nested source-reference slices.
|
||||
|
||||
### `internal/modules/generic/normalize/noop`
|
||||
|
||||
The normalizer returns the merged domain value unchanged and is reusable for
|
||||
any registered artifact type.
|
||||
|
||||
### `internal/modules/dnd/normalize/spells`
|
||||
|
||||
The typed spell normalizer resolves the optional `spell_catalog` reference into
|
||||
the same immutable SRD-plus-overlay effective catalog used by spell extraction
|
||||
and catalog validation. It performs no LLM calls. For each spell cast it
|
||||
canonicalizes recognized names using the catalog's case, whitespace,
|
||||
apostrophe, and alias rules; canonicalizes source references with the shared
|
||||
document-aware order; removes only exact reference duplicates; and emits
|
||||
bounded, scoped warnings for each mutation or unresolved name.
|
||||
|
||||
After those per-cast changes, it collapses only casts with the same canonical
|
||||
spell, case-folded and whitespace-normalized caster, and complete non-empty
|
||||
valid source-reference set. It retains the first occurrence and its caster,
|
||||
source references, and stable order. Unknown names, empty or invalid evidence,
|
||||
and adjacent or overlapping but different ranges remain unchanged for
|
||||
validation.
|
||||
|
||||
The normalizer exposes the effective catalog digest as its independently scoped
|
||||
`effective_catalog` checkpoint fingerprint and reports catalog base ID, digest,
|
||||
and overlay IDs as manifest metadata. Catalog contents, reference paths, and
|
||||
raw overlay bytes are not included in either surface. The normalize-stage
|
||||
reference is stage-local, so an overlay-capable pipeline binds the catalog
|
||||
independently for extraction and normalization.
|
||||
|
||||
### `internal/modules/dnd/normalize/combatturns`
|
||||
|
||||
The combat normalizer prepares an external NPC registry before execution or
|
||||
receives a generated registry at the ordered step handoff, then uses the
|
||||
immutable view during runtime. It display-normalizes actors,
|
||||
rewrites canonical-name matches for actors, orders and deduplicates source
|
||||
references, stable-sorts records by source-document position, and collapses
|
||||
only exact duplicate identities with fully valid evidence. It deep-clones
|
||||
output storage and emits bounded warnings scoped to merged input indexes. Its
|
||||
metadata and fingerprints identify the normalization and NPC identity policies.
|
||||
External bindings may contribute registry
|
||||
digest/count metadata; generated identity is retained in framework provenance
|
||||
and dependency fingerprints. The normalizer is included in the production D&D
|
||||
registrar with the default combat normalization chain.
|
||||
|
||||
### `internal/modules/dnd/normalize/itemevents`
|
||||
|
||||
The item-event normalizer accepts no options or references and makes no LLM
|
||||
calls. It trims display-edge whitespace in names and holders, canonicalizes
|
||||
source references, source-orders events, and collapses only exact duplicates
|
||||
with complete valid evidence. It does not create a ledger, calculate balances,
|
||||
resolve aliases, infer quantities or holders, or reconcile nearby events. Its
|
||||
policy fingerprint and bounded warnings identify deterministic normalization;
|
||||
the matching invariant validator checks the resulting order and duplicate rule.
|
||||
|
||||
### `internal/modules/dnd/normalize/npcinteractions`
|
||||
|
||||
The interaction normalizer requires the same immutable NPC registry. It
|
||||
canonicalizes exact registry-name matches, orders and de-duplicates source
|
||||
references, stable-sorts occurrences by source-document position, and collapses
|
||||
only exact interaction identities with valid evidence. It does not infer,
|
||||
merge, or summarize distinct occurrences. Its metadata and fingerprints expose
|
||||
the normalization and NPC identity policies; generated registry identity stays
|
||||
in framework provenance and checkpoint dependencies.
|
||||
|
||||
### `internal/modules/dnd/normalize/scenedescriptions`
|
||||
|
||||
The scene-description normalizer has no options or references. It validates
|
||||
each source range against the source document, trims title and summary
|
||||
whitespace, orders records by source position then ID, removes only exactly
|
||||
identical records, and rejects conflicting reused IDs or ranges. Its policy
|
||||
fingerprint identifies this deterministic behavior; the matching invariant
|
||||
validator checks the normalized result in the production chain.
|
||||
|
||||
## Output Encoder
|
||||
|
||||
### `internal/modules/generic/output/json`
|
||||
|
||||
The JSON encoder sorts normalized results by lane, derives collision-checked
|
||||
safe logical names, pretty-prints JSON payloads, and assembles the logical index,
|
||||
manifest, rejected-result, warning, and lane files. Invalid JSON, unsupported
|
||||
media types, unsafe names, and sanitized-name collisions are errors.
|
||||
|
||||
Its strict `include_chunk_map` option is disabled by default. When enabled, it
|
||||
validates the framework-supplied accepted chunk map through its codec and adds
|
||||
the pipeline-wide `chunk-map.json` plus its index descriptor; it does not treat
|
||||
the map as a lane payload. The external shape is owned by the
|
||||
[Accepted Chunk Map contract](../integrations/chunk-map.md).
|
||||
|
||||
The encoder returns logical files only. The CLI places them on disk, and the
|
||||
[JSON output contract](../integrations/json-output.md) defines their external
|
||||
paths and schemas.
|
||||
|
||||
## Generic Validators
|
||||
|
||||
The generic validator implementations live under
|
||||
`internal/modules/generic/validate`.
|
||||
|
||||
The unconditional accept and reject validators provide explicit chunk and
|
||||
typed-artifact variants used primarily for controlled composition and tests.
|
||||
|
||||
The serialized JSON syntax validator uses `encoding/json` to reject malformed
|
||||
representation bytes. The serialized JSON Schema validator requires schema
|
||||
bytes, parses the instance and schema with `jsonschema`, and distinguishes
|
||||
payload rejection from schema loading or compilation errors. The framework
|
||||
serialized-validation request carries either canonical chunk bytes or artifact
|
||||
codec bytes according to its target context. Neither validator calls the LLM.
|
||||
|
||||
## D&D Spell Validators
|
||||
|
||||
All four validators receive `dnd.SpellList` directly. The shape validator
|
||||
rejects a missing list, blank caster or spell names, and empty reference lists.
|
||||
The catalog validator defers when shape is invalid, then checks every non-empty
|
||||
spell name against the immutable effective SRD and overlay catalog. It accepts
|
||||
normalized canonical names and aliases without rewriting the artifact; unknown
|
||||
names reject the complete result with bounded, stable index/name diagnostics. The
|
||||
source-reference validator defers malformed shapes, validates every cited
|
||||
range, and reports all range defects through a bounded aggregate while
|
||||
preserving `invalid_source_refs`. The relatedness validator resolves all cited
|
||||
ranges through the shared document-order traversal, then warns when a normalized
|
||||
consecutive spell-name token sequence is absent from the cited source text.
|
||||
Invalid shape
|
||||
or cited ranges produce no relatedness warnings; the shape and source-reference
|
||||
validators own those defects.
|
||||
|
||||
These validators are deterministic. Shape, source-reference, and relatedness
|
||||
each expose a local semantic `policy` checkpoint fingerprint. The catalog
|
||||
validator instead exposes its effective catalog digest as its semantic
|
||||
checkpoint identity and does not add a separate policy fingerprint. Their
|
||||
selectable keys and production order are defined in
|
||||
[Configuration](../config.md#implemented-production-validators); their durable
|
||||
payload rules are defined in the
|
||||
[artifact contract](../integrations/dnd-spell-artifacts.md).
|
||||
|
||||
## D&D NPC Validators
|
||||
|
||||
NPC shape validation checks the required ID and name strings, list presence, and source-reference
|
||||
shape. The source-reference validator defers malformed shapes, checks
|
||||
current-document identity, unit existence, and range ordering, and reports all
|
||||
defects through bounded aggregates. Source relatedness uses the shared
|
||||
document-order traversal and normalized consecutive-token matching, emitting at
|
||||
most one bounded warning per record when the canonical name does not occur near
|
||||
its cited text. Invalid shape or cited ranges produce no relatedness warnings.
|
||||
Normalize identity validation checks deterministic IDs, canonical names, and
|
||||
duplicate canonical-name or ID ownership.
|
||||
All are deterministic and expose the policy fingerprints used by the
|
||||
production chains.
|
||||
|
||||
## D&D Combat Validators
|
||||
|
||||
Combat shape validation owns the required list, actor, supported turn kind, and
|
||||
non-empty source-reference collection. Combat source-reference validation defers invalid
|
||||
shape, checks source identity, unit existence, and range order, and reports all
|
||||
defects through bounded aggregates. Combat source-relatedness defers invalid
|
||||
shape or ranges, uses the shared traversal to combine overlapping cited units
|
||||
in document order, and emits at most one bounded advisory warning per turn for
|
||||
an unrelated actor. Actors use normalized consecutive-token matching. The
|
||||
normalized-invariants validator owns actor display normalization, canonical
|
||||
source-reference order, chronology, and exact duplicate identity; it defers
|
||||
shape and source-reference failures. All four validators are deterministic and
|
||||
expose local policy fingerprints. In the registered defaults, JSON syntax runs
|
||||
first; combat shape, normalized invariants when applicable, and source-reference
|
||||
validation precede JSON Schema validation; warning-only relatedness runs last.
|
||||
|
||||
## D&D Item-Event Validators
|
||||
|
||||
Item-event shape validation owns the required list, non-empty name, supported
|
||||
category, category-and-holder combination, positive optional quantity, and
|
||||
non-empty source-reference collection. Source-reference validation defers
|
||||
malformed shapes, checks current-source identity and ordered ranges, and during
|
||||
extraction requires every citation to fit the accepted chunk. Relatedness is
|
||||
advisory and warning-only: it checks the event name against cited transcript
|
||||
text while deferring malformed candidates and invalid ranges to their blocking
|
||||
owners. The normalized-invariants validator owns display normalization,
|
||||
canonical source-reference order, chronology, and exact duplicate identity.
|
||||
All four validators are deterministic and expose policy fingerprints. The
|
||||
registered chains run syntax and blocking checks before durable JSON Schema;
|
||||
relatedness remains last.
|
||||
|
||||
## D&D NPC Interaction Validators
|
||||
|
||||
Interaction shape validation owns the required list, registry name, supported
|
||||
kind, and non-empty source-reference collection. Registry validation checks
|
||||
exact membership in the required immutable NPC registry. Source-reference and
|
||||
relatedness validation use the current transcript only; malformed candidates
|
||||
are deferred by later validators and produce no relatedness warning. The
|
||||
normalized-invariants validator owns canonical registry names, source-reference
|
||||
order, chronology, and exact duplicate identity. The production chains run
|
||||
shape, registry, and source-reference checks before JSON Schema validation;
|
||||
relatedness remains warning-only and last.
|
||||
|
||||
## D&D Scene Description Validators
|
||||
|
||||
Scene-description shape validation owns the non-empty list, trimmed ID and
|
||||
prose, closed kind, and basic source-reference shape. Extract-stage source
|
||||
validation additionally requires the one record to attach exactly to the
|
||||
current accepted chunk; later source validation checks source membership.
|
||||
Relatedness checks the title and summary independently against only their cited
|
||||
transcript range and emits bounded advisory warnings. The normalized-invariants
|
||||
validator owns ordering, exact duplicate elimination, and conflicting ID or
|
||||
range detection. The production chains run shape and source-reference checks
|
||||
before JSON Schema validation; the warning-only relatedness check is last.
|
||||
|
||||
## Production Registration
|
||||
|
||||
Production composition occurs through family registrars. The CLI allocates one
|
||||
complete framework registry set and one LLM asset registry. It invokes
|
||||
`internal/modules/generic/register`,
|
||||
`internal/modules/seriatim/register`, and `internal/modules/dnd/register` in
|
||||
that order, then exposes the matching catalog for resolution. The generic and
|
||||
Seriatim registrars own their production leaf registrations. The D&D registrar
|
||||
owns D&D leaf registrations, typed spell, NPC, combat, item-event, interaction, and scene-description default-validator
|
||||
chains, typed append-order specializations, and D&D prompt/schema asset
|
||||
collection. Its registration helpers group module, validator, prompt-asset, and
|
||||
chain composition while retaining artifact-specific merge and clone behavior in
|
||||
the registrar.
|
||||
|
||||
Concrete implementation packages do not import generic implementation
|
||||
packages directly. A concrete family's `register` package is its composition
|
||||
point for specializing reusable generic implementations, while the generic
|
||||
registrar composes only generic children.
|
||||
|
||||
Core and framework production packages do not import production extensions.
|
||||
CLI production code is the sole application composition root for extensions
|
||||
and imports only exact family registrar packages. Other production packages,
|
||||
including commands and newly introduced package trees, do not import module
|
||||
packages directly. Compatibility tests in the CLI, core, and framework trees
|
||||
may import roots and implementation leaves directly. Other non-module tests do
|
||||
not receive that exemption. White-box tests within module families retain the
|
||||
production family boundaries. `internal/modules/integration` is test
|
||||
infrastructure: its black-box tests may compose multiple families, but it is
|
||||
not a production module family or production dependency target.
|
||||
|
||||
## Adding An Extension
|
||||
|
||||
When adding a production module or validator:
|
||||
|
||||
1. implement the stage or validator contract and package-local key;
|
||||
2. expose and test its spec, constructor, and registration function;
|
||||
3. keep format or domain parsing inside the concrete package;
|
||||
4. add package-owned prompt/schema assets when the extension is LLM-backed;
|
||||
new LLM-backed D&D extraction modules must follow the stable-to-variable
|
||||
prompt ordering, shared-asset ownership, and cache-boundary policy in
|
||||
[LLM Runtime](llm.md#dd-extraction-prompt-ordering-and-cache-boundaries), or
|
||||
document the implemented exception and its evidence there;
|
||||
5. register it through its package-family registrar and add a default chain
|
||||
there only when production policy requires one;
|
||||
6. add resolution and composition coverage for capabilities, options,
|
||||
references, and validation behavior;
|
||||
7. update the selectable-key catalog in [Configuration](../config.md), the
|
||||
relevant external contract, this inventory, and maintained examples when
|
||||
user-visible behavior changes.
|
||||
|
||||
Do not add the extension to `docs/development.md`; that file routes by task and
|
||||
does not inventory implementations.
|
||||
|
||||
### D&D Extractor Contract
|
||||
|
||||
New D&D extractors preserve these package-owned responsibilities:
|
||||
|
||||
- Reject unknown options unless an option namespace is intentionally
|
||||
extensible, and use shared common preflight while retaining receiver,
|
||||
dependency, and lane-specific checks locally.
|
||||
- Return independently owned results and exposed metadata that callers may
|
||||
safely mutate.
|
||||
- Keep the private response DTO, structural response schema and its identity,
|
||||
provider-response mapping, durable artifact conversion, and lane diagnostics
|
||||
in the owning package.
|
||||
- Include every stable semantic input that can change durable output in
|
||||
checkpoint identity. Consider prompt, schema, mapping, canonicalization,
|
||||
prepared reference projections, identity, normalization, and trimming where
|
||||
applicable.
|
||||
- Add focused behavioral coverage where the lane's risks warrant it, including
|
||||
construction and registration, option rejection, preflight, provider
|
||||
failures, structured decoding, mapping and ownership, prompt
|
||||
role/input/cache order, and checkpoint invalidation.
|
||||
|
||||
Prompt ordering, shared-asset ownership, cache boundaries, and private-schema
|
||||
rules are defined in [LLM Runtime](llm.md#dd-extraction-prompt-ordering-and-cache-boundaries).
|
||||
[Pipeline Internals](pipeline.md#reference-materialization) owns reference
|
||||
materialization, and its [checkpoint hooks](pipeline.md#checkpoint-and-debug-hooks)
|
||||
define checkpoint behavior. Follow [Architecture](../policy/architecture.md#source-and-domain-boundaries)
|
||||
for ownership boundaries and the [Testing Policy](../policy/testing.md) when
|
||||
selecting durable coverage. This contract intentionally does not prescribe
|
||||
prompt prose or length, hashes, test counts, filenames, fixture layouts, or
|
||||
generic implementation builders.
|
||||
|
||||
## Tests To Inspect
|
||||
|
||||
- Package-local `*_test.go` files under the module or validator being changed.
|
||||
- `internal/framework/pipeline/typed_resolution_test.go`: typed registry, spec,
|
||||
and heterogeneous artifact composition.
|
||||
- `internal/framework/pipeline/profile_test.go`: framework binding defaults and
|
||||
profile resolution.
|
||||
- `internal/cli/production_contract_test.go`: production catalog, config
|
||||
resolution, and composition smoke coverage.
|
||||
- `internal/cli/example_contract_test.go`: maintained example ownership.
|
||||
- `internal/framework/promptfs/*_test.go` and
|
||||
`internal/modules/dnd/shared/*_test.go`: shared prompt and reference assembly.
|
||||
- `internal/modules/integration/*_test.go`: black-box composition across
|
||||
production extension domains.
|
||||
# Module Internals
|
||||
|
||||
This guide owns the mechanics for implementing and registering production
|
||||
modules. [Configuration](../config.md) owns selectable keys, binding syntax,
|
||||
reference configuration, and default validator chains. Durable input and output
|
||||
shapes belong in [integration contracts](../integrations/).
|
||||
|
||||
The D&D family has additional shared conventions and domain-specific
|
||||
exceptions. See [D&D Module Internals](dnd.md) rather than adding them here.
|
||||
|
||||
## Module Boundary
|
||||
|
||||
A module is a typed implementation registered for one pipeline stage. Its
|
||||
`ModuleSpec` is the public-to-the-framework declaration of its stable key,
|
||||
stage, required and provided capabilities, artifact kind, and accepted
|
||||
reference slots. The framework uses that declaration to resolve a configured
|
||||
binding before it builds the implementation.
|
||||
|
||||
Implementations that accept options must provide both an option validator and
|
||||
a builder. The validator is used while resolving configuration; the builder
|
||||
decodes the same options and constructs the implementation from the prepared
|
||||
`BuildRequest`. Reject unknown options in both paths. A builder receives only
|
||||
the dependencies and materialized references that the framework prepared for
|
||||
that operation, so it must not re-read configuration or files.
|
||||
|
||||
Registry helpers register the typed builder for a stage-specific registry.
|
||||
They are preferable to hand-written untyped registration because they retain
|
||||
the artifact type at the framework boundary. Registrars validate the registries
|
||||
they need, register each leaf implementation, and add any family-owned assets
|
||||
or default validator chains. They return contextual errors so production
|
||||
composition fails at startup rather than at the first run.
|
||||
|
||||
## Production Composition
|
||||
|
||||
Production composition is intentionally split by family:
|
||||
|
||||
- The generic registrar provides the unit chunker, generic JSON validators,
|
||||
and JSON output encoder.
|
||||
- The Seriatim registrar provides the transcript input adapter. Its external
|
||||
input behavior is defined by the [Seriatim contract](../integrations/seriatim.md).
|
||||
- The D&D registrar provides its codecs, extractors, mergers, normalizers,
|
||||
validators, prompt assets, and default chains. Its behavioral conventions
|
||||
are documented in [D&D Module Internals](dnd.md).
|
||||
|
||||
The CLI owns the composition that invokes these registrars. A module package
|
||||
may register its own family but must not assemble the CLI or make framework
|
||||
packages depend on production extensions.
|
||||
|
||||
## Adding Or Changing A Module
|
||||
|
||||
1. Choose the pipeline stage and the typed artifact boundary. Put external
|
||||
input or durable artifact formats in the relevant integration contract,
|
||||
not in this guide or in a private LLM response type.
|
||||
2. Define a stable `ModuleSpec` with the exact capabilities and reference
|
||||
slots needed for the operation. Model a producer/consumer handoff as an
|
||||
artifact-compatible slot; configuration then chooses an external file or a
|
||||
generated binding.
|
||||
3. Implement strict option decoding, construction, and the typed stage
|
||||
interface. Preserve caller ownership: do not retain mutable request data
|
||||
and return defensive copies where an implementation exposes stored data.
|
||||
4. Register the module through its typed registry helper and add it to the
|
||||
owning family registrar. Add a default validator chain only when that
|
||||
family owns the behavior; otherwise require an explicit compatible chain.
|
||||
5. Update the selectable-key and chain reference in
|
||||
[Configuration](../config.md#production-module-keys), the applicable
|
||||
integration contract, and focused tests. Keep the configuration document
|
||||
as the sole list of production keys and validator order.
|
||||
|
||||
## Validation And References
|
||||
|
||||
Validators operate on the value produced at their configured stage. A default
|
||||
chain is ordered behavior, not a set: JSON parsing, structural checks,
|
||||
domain-specific checks, durable-schema checks, and advisory checks may have
|
||||
different responsibilities and failure handling. The active default chains and
|
||||
override rules are maintained in
|
||||
[Configuration](../config.md#production-validator-keys-and-default-chains).
|
||||
|
||||
Reference slots are part of the module specification. They describe the
|
||||
accepted artifact kind, media type, size, and whether a binding is required;
|
||||
the framework validates those constraints before construction. An external
|
||||
reference is materialized during preparation. A generated reference is a
|
||||
compatible normalized artifact handed from an earlier pipeline step at
|
||||
operation time. The configuration reference rules, including precedence and
|
||||
ordered-handoff requirements, are maintained in
|
||||
[Configuration](../config.md#references-and-ordered-handoffs).
|
||||
|
||||
## Focused Verification
|
||||
|
||||
Exercise the leaf implementation and its registration path when changing a
|
||||
module. Registry and registrar tests cover duplicate keys, required registries,
|
||||
and typed construction; pipeline resolution tests cover capabilities, options,
|
||||
and reference compatibility. Domain packages should additionally test their
|
||||
codecs, validators, normalizers, and any integration handoffs they own.
|
||||
|
||||
Run the affected package tests while iterating. The complete module suite is:
|
||||
|
||||
~~~sh
|
||||
go test ./internal/modules/...
|
||||
~~~
|
||||
|
||||
Reference in New Issue
Block a user