notarius/docs/internal/modules.md

Module And Validator Internals

Production module and validator implementations live under their domain-first trees in internal/modules. The selectable keys, configuration options, reference slots, and default validator chain are canonical in the module and validator catalogs in Configuration.

Extension Pattern

A stage module package provides a stable key, constructor, contract implementation, ModuleSpec, Register, and focused behavior and registration tests. A validator package follows the same pattern with ValidatorSpec and the validator registry. Package-family registrars compose those leaf registrations into the production catalog and own family-level policy such as default validator chains and prompt asset collection.

Production input, chunk, output, and D&D spell-, combat-, and interaction-extract packages register strict option decoders and run-local builders. Preparation decodes their options into implementation-owned values and injects dependencies plus the materialized reference set for the selected target. Each builder receives an isolated clone of that set; input and output builders receive no references. The spell, combat, and interaction extractors are typed over the canonical D&D model. D&D validators, merge, and normalize use typed variants; JSON representation validators use serialized requests; and unconditional validators expose separate chunk and typed variants. The D&D production registrar registers the canonical typed spell, NPC, combat, and interaction implementations, including their kind-specific merge and normalize behavior.

For D&D artifact defaults, generic JSON syntax validation runs first. Rejecting domain validators then own semantic diagnostics before generic JSON Schema validation provides the final rejecting representation backstop; warning-only relatedness validators run last. This default composition does not reorder an explicitly configured validator chain.

Prepared extractors, extract validators, and codecs may be reused concurrently by the run-wide extract pool. Production implementations are immutable after construction: they retain only typed options, immutable assets, or the shared concurrency-safe LLM client. Implementations that introduce mutable state must synchronize that state without creating a separate provider scheduler.

Specs expose capability and execution metadata without constructing an implementation. Registry entries separately expose option validation and run-local construction. Chunk, extract, merge, and normalize modules that accept auxiliary material declare identical reference slots from both ReferenceSlots() and ModuleSpec().ReferenceSlots; registration tests enforce that agreement. Runtime delivery uses the corresponding stage request's References field.

LLM-backed extensions own their prompt definitions and response schemas under package-local embedded assets. Shared filesystem composition belongs in internal/framework/promptfs; reusable D&D prompt fragments, reference declarations, prompt-input assembly, and source-unit/citation helpers belong in internal/modules/dnd/shared, which also owns bounded D&D diagnostics. The D&D scene chunker and spell, NPC, combat-turn, and NPC-interaction extractors use ordered package-local prompt manifests for both rendering and prompt fingerprinting, so only the shared fragments each prompt actually renders participate in either operation. Extraction prompts place stable shared and lane-specific context before the variable transcript and use shared assets for wording common across lanes. The canonical ordering and cache-boundary policy is documented in LLM Runtime. Stage contracts expose only Notarius structured-completion types, not Scriptorium public types.

The shared ChunkPromptMaterial helper owns common transcript material preparation for the spell, NPC, combat-turn, and NPC-interaction extractors. It clones supplied source metadata, falls back to the materialized chunk when content is absent, checks that content remains chunk-identical, and fills only the common default fields. Extractors retain their request validation and wrap helper errors with their module context.

Reference material may inform a module or prompt but must not become source evidence. The resolver and materializer behavior is described in Pipeline Internals.

Domain Reference Data

internal/modules/dnd/spells/catalog

The spell catalog package owns the embedded, versioned D&D 5e 2014 SRD spell reference data. Its strict JSON asset contains one canonical record per spell, including spell level and all applicable class memberships. LoadSRD5E2014 validates catalog identity, provenance metadata, ordering, uniqueness, levels, classes, aliases, and lookup-key collisions before exposing immutable copies.

Lookup is case-insensitive and normalizes whitespace and common apostrophe variants while preserving source punctuation in canonical display names. The catalog contains 319 unique spells and 779 class memberships. Source and license details live beside the asset in SOURCES.md. This domain-owned data is separate from internal/modules/dnd/shared, which is reserved for reusable prompt and source-reference machinery.

ResolveEffectiveCatalog builds the immutable recognition view used by the spell extractor and catalog validator. It starts with the embedded SRD catalog and optionally applies one strict JSON overlay from the spell_catalog item in a materialized reference set. Overlay catalogs are ordered by ID, may add names and aliases, and may augment an existing canonical spell without replacing its display name. Cross-spell lookup collisions are errors. The effective view exposes sorted canonical names, normalized lookup, overlay identities, and a semantic digest; overlay content remains contextual reference material rather than source evidence. Its external JSON contract is defined in the spell-catalog overlay contract.

internal/modules/dnd/npcs/identity, internal/modules/dnd/npcs/registry, and internal/modules/dnd/codec/npcs

The NPC identity package owns Unicode comparison keys, deterministic npc:sha256: IDs, display normalization, and whole-registry collision issues. The registry package resolves one optional normalized artifact through the strict codec, validates whole-registry identity, canonicalizes its JSON, and provides immutable records, a names-only prompt projection, distinct durable and projection digests, count, and exact canonical-name lookup. External files cross this boundary during preparation; generated artifacts cross it at the ordered step handoff. It owns the npcs slot and its bounded, content-safe validation failures. NPC source references are durable provenance and are not treated as evidence for a consuming pipeline. The NPC codec owns the strict durable dnd/npc-list JSON boundary and exposes candidate versus approved encode/decode operations.

The internal/modules/dnd/codec/combatturns package owns the durable dnd/combat-turn-list schema and candidate versus approved JSON boundary. It is registered by the production D&D family registrar for the selectable combat lane.

The internal/modules/dnd/codec/npcinteractions package owns the durable dnd/npc-interaction-list schema and candidate versus approved JSON boundary. It is registered by the production D&D family registrar for the selectable interaction lane. Its external contract is documented in the D&D NPC interaction artifact contract.

Input Adapter

internal/modules/seriatim/input/transcript

The adapter decodes the supported transcript JSON, selects the source identity, computes canonical source provenance, validates segments, and maps each segment into a generic source unit with a self-reference plus speaker and timestamp metadata. It accepts no module options. Its spec advertises the transcript capabilities consumed by D&D modules.

Parsing is strict about required values and duplicate unit IDs but deliberately ignores unrelated Seriatim fields. The external format and derived-identity rules are defined in the Seriatim contract.

Chunkers

Chunkers implement contracts.Chunker.Plan. A plan identifies ordered source unit ranges and may carry optional namespaced JSON annotations; it does not contain materialized chunk content. The framework canonicalizes annotations, validates ranges against the current source, and materializes chunk IDs, indexes, references, content, units, and generic metadata. Materialized source unit metadata is independently owned. Annotation namespaces remain optional data: generic framework code and downstream modules must not require D&D scene annotations or import dnd/scenes.

internal/modules/generic/chunk/units

The generic chunker validates the source document and returns ranges over units in configured windows. Overlap changes the next window start but never reorders units. Framework materialization derives the resulting chunk identity and generic metadata from those ranges.

The accepted options and defaults are defined in Configuration. Generic framework validation canonicalizes the returned unit slices before extraction. The chunker decodes its options during construction and retains only the typed window settings used by Plan.

internal/modules/dnd/chunk/scenes

The scene chunker prepares a structured Scriptorium request from the full transcript, session, and optional D&D reference inputs. It validates the model's scene boundaries against source-unit IDs and converts them into deterministic plan ranges with optional scene annotations. Preparation injects the shared structured LLM client into the chunker; Plan supplies only the run-specific profile, session, source, references, and metadata.

Scene validation requires sequential, contiguous, non-overlapping coverage from the first source unit through the last. Scene descriptions, boundaries, confidence, and participants are module-owned annotations. Boundary caveats become warnings. Malformed structured output is returned as an error; there is no fallback chunker.

The package embeds its prompt and response schema and reports their non-secret identity and hashes through singleton module metadata. Shared D&D assets supply reference declarations and prompt inputs; their user-facing keys and accepted file types remain canonical in Configuration.

Extractor

internal/modules/dnd/extract/spells

The spell extractor prepares a structured request from one chunk, the chunk-scoped source input, the session, and optional D&D reference inputs. It decodes the model response, assigns the generic source identity to every source reference, canonicalizes duplicate references, orders spell casts by their earliest cited unit, and returns dnd.SpellList.

Its private response schema admits only the structural transport envelope: required fields, JSON types, array and object shapes, and unknown-field rejection. It maps integer source-unit candidates directly without repairing semantic values, so the deterministic shape, catalog, and source-reference validators own blank values, empty evidence, and invalid or unresolved ranges.

The extractor owns its private model-response DTO, embedded prompt, LLM response schema, strict option decoder, injected shared LLM client, and prompt/schema manifest metadata. During preparation it resolves the optional spell_catalog reference into an immutable effective catalog and adds a generated canonical-name-only JSON input to every structured completion request. Overlay failures therefore stop construction before source parsing or an LLM call; campaign references remain separate disambiguation inputs and never become source evidence.

The prompt includes only actual casting events and unambiguous declared casting attempts. Spell mentions, plans, rules discussion, and catalog matches without a casting event are excluded. Shared extraction-evidence and identity rules require transcript-supported caster and spell facts, while the catalog, campaign references, and NPC names only disambiguate source text. Structural source validation remains deterministic; semantic evidence sufficiency is enforced through extraction policy and evaluation.

Both the extractor and deterministic catalog validator expose the effective base-plus-overlay semantic digest as scoped prepared-component checkpoint identity. Raw overlay provenance independently covers file-byte changes, while the semantic digest also invalidates reuse when the embedded catalog or catalog composition changes. The extractor additionally fingerprints its complete prompt assets and private response schema, so either semantic contract changing invalidates previously recorded extraction checkpoints. The separate internal/modules/dnd/codec/spells package owns the durable schema and stable JSON representation for artifact kind dnd/spell-list. The runner keeps the result typed through validators and later stages, using the codec only for checkpoint, debug, and output boundaries. Shared D&D helpers keep prompt input names and source-unit reference conversion consistent with the scene chunker.

The extractor also declares the optional npcs registry slot and consumes the immutable registry boundary from internal/modules/dnd/npcs/registry. An external registry is prepared before execution; a generated registry is validated and supplied at operation time. Bound external registries add only the full npc_registry_digest and npc_count to module metadata. The local npc_registry checkpoint fingerprint always covers the names-only projection, including its exact unbound value. Generated bindings are represented by framework handoff provenance and dependency fingerprints. The unbound prompt input is exactly {"npcs":[]} and has no registry provenance. The shared NPC grounding fragment is placed immediately after the common campaign reference message and is included in the spell prompt fingerprint.

The durable payload and manifest metadata shapes are defined in the D&D spell artifact contract.

internal/modules/dnd/extract/npcs

The NPC extractor maps private model output to the canonical dnd.NPCList, assigns source identity and deterministic NPC IDs, and preserves source references for deterministic validation. It uses the shared campaign references only for disambiguation and does not consume the optional NPC registry slot. Its prompt and private response schema are package-owned. The private response contains only a name and model-facing evidence ranges for each record; anonymous groups, generic roles, invented labels, descriptions, aliases, and relationships are outside its contract. The prompt follows the shared D&D extraction ordering and cache policy documented in LLM Runtime.

The private response schema owns only structural transport validation and maps integer source-unit candidates unchanged. Required semantic content, non-empty evidence, and valid source ranges are rejected by the deterministic shape and source-reference validators.

internal/modules/dnd/extract/combatturns

The combat extractor prepares one structured request per supplied chunk using the shared extraction-evidence, identity, campaign-reference, NPC-grounding, and transcript prompt inputs. It maps the private response to dnd.CombatTurnList, assigns the current source identity, removes exact duplicate source ranges, and orders turns by valid source-document position while preserving malformed candidate fields for deterministic validators. Its package-owned private response schema enforces only the structural JSON envelope; semantic artifact constraints remain with the validator chain. Its prepared metadata and checkpoint fingerprints contain only prompt/schema/mapping identities plus an optional NPC registry digest. The prompt follows the shared D&D extraction ordering and cache policy documented in LLM Runtime. The package exposes typed registration and is included in the production D&D registrar with the default combat extraction chain.

The combat normalizer accepts only the optional structured NPC registry. Campaign references remain extractor-only LLM context and are not materialized for deterministic normalization.

internal/modules/dnd/extract/npcinteractions

The NPC interaction extractor requires the structured npcs registry slot. It uses the registry's names-only prompt projection with shared extraction evidence, identity, and transcript material, then maps private model records to dnd.NPCInteractionList with the current source identity. Registry source references are never reused as interaction evidence. The private response schema carries only name, bounded interaction kind, and source-unit ranges; deterministic validators own registry membership, source validity, and relatedness. Extract-stage source validation additionally requires every cited range to be wholly contained in the current materialized chunk. Prompt, schema, mapping, and the names-only registry projection participate in checkpoint identity, while generated producer identity remains framework provenance.

The domain-owned internal/modules/dnd/npcinteractions package defines canonical source-reference and occurrence ordering, valid-evidence eligibility, and collision-safe exact identity. The interaction normalizer and normalized invariants validator both consume those rules, so their production and checking paths cannot drift. Normalizer and relatedness warning lists use the shared D&D diagnostic cap and emit a final omission-summary warning when truncated.

internal/modules/dnd/normalize/npcs

The NPC normalizer performs deterministic identity-aware consolidation in merged input order. It consolidates only equal canonical-name comparison keys, retains the first display record, and unions exact source references. It exposes the identity policy as its local checkpoint fingerprint and emits bounded normalization warnings.

Merger And Normalizer

internal/modules/generic/merge/appendorder

The merger passes typed values to an injected combine function in framework source-chunk order. The D&D registrar specializes it with a spell-list append function.

internal/modules/generic/normalize/noop

The normalizer returns the merged domain value unchanged and is reusable for any registered artifact type.

internal/modules/dnd/normalize/spells

The typed spell normalizer resolves the optional spell_catalog reference into the same immutable SRD-plus-overlay effective catalog used by spell extraction and catalog validation. It performs no LLM calls. For each spell cast it canonicalizes recognized names using the catalog's case, whitespace, apostrophe, and alias rules; sorts source references by source identity and unit boundaries; removes only exact reference duplicates; and emits bounded, scoped warnings for each mutation or unresolved name.

After those per-cast changes, it collapses only casts with the same canonical spell, case-folded and whitespace-normalized caster, and complete non-empty valid source-reference set. It retains the first occurrence and its caster, source references, and stable order. Unknown names, empty or invalid evidence, and adjacent or overlapping but different ranges remain unchanged for validation.

The normalizer exposes the effective catalog digest as its independently scoped effective_catalog checkpoint fingerprint and reports catalog base ID, digest, and overlay IDs as manifest metadata. Catalog contents, reference paths, and raw overlay bytes are not included in either surface. The normalize-stage reference is stage-local, so an overlay-capable pipeline binds the catalog independently for extraction and normalization.

internal/modules/dnd/normalize/combatturns

The combat normalizer prepares an external NPC registry before execution or receives a generated registry at the ordered step handoff, then uses the immutable view during runtime. It display-normalizes actors, rewrites canonical-name matches for actors, orders and deduplicates source references, stable-sorts records by source-document position, and collapses only exact duplicate identities with fully valid evidence. It deep-clones output storage and emits bounded warnings scoped to merged input indexes. Its metadata and fingerprints identify the normalization and NPC identity policies. External bindings may contribute registry digest/count metadata; generated identity is retained in framework provenance and dependency fingerprints. The normalizer is included in the production D&D registrar with the default combat normalization chain.

internal/modules/dnd/normalize/npcinteractions

The interaction normalizer requires the same immutable NPC registry. It canonicalizes exact registry-name matches, orders and de-duplicates source references, stable-sorts occurrences by source-document position, and collapses only exact interaction identities with valid evidence. It does not infer, merge, or summarize distinct occurrences. Its metadata and fingerprints expose the normalization and NPC identity policies; generated registry identity stays in framework provenance and checkpoint dependencies.

Output Encoder

internal/modules/generic/output/json

The JSON encoder sorts normalized results by lane, derives collision-checked safe logical names, pretty-prints JSON payloads, and assembles the logical index, manifest, rejected-result, warning, and lane files. Invalid JSON, unsupported media types, unsafe names, and sanitized-name collisions are errors.

The encoder returns logical files only. The CLI places them on disk, and the JSON output contract defines their external paths and schemas.

Generic Validators

The generic validator implementations live under internal/modules/generic/validate.

The unconditional accept and reject validators provide explicit chunk and typed-artifact variants used primarily for controlled composition and tests.

The serialized JSON syntax validator uses encoding/json to reject malformed representation bytes. The serialized JSON Schema validator requires schema bytes, parses the instance and schema with jsonschema, and distinguishes payload rejection from schema loading or compilation errors. The framework serialized-validation request carries either canonical chunk bytes or artifact codec bytes according to its target context. Neither validator calls the LLM.

D&D Spell Validators

All four validators receive dnd.SpellList directly. The shape validator rejects a missing list, blank caster or spell names, and empty reference lists. The catalog validator defers when shape is invalid, then checks every non-empty spell name against the immutable effective SRD and overlay catalog. It accepts normalized canonical names and aliases without rewriting the artifact; unknown names reject the complete result with bounded, stable index/name diagnostics. The source-reference validator defers malformed shapes, validates every cited range, and reports all range defects through a bounded aggregate while preserving invalid_source_refs. The relatedness validator resolves all cited ranges through the shared document-order traversal, then warns when a normalized consecutive spell-name token sequence is absent from the cited source text. Invalid shape or cited ranges produce no relatedness warnings; the shape and source-reference validators own those defects.

These validators are deterministic. Shape, source-reference, and relatedness each expose a local semantic policy checkpoint fingerprint. The catalog validator instead exposes its effective catalog digest as its semantic checkpoint identity and does not add a separate policy fingerprint. Their selectable keys and production order are defined in Configuration; their durable payload rules are defined in the artifact contract.

D&D NPC Validators

NPC shape validation checks the required ID and name strings, list presence, and source-reference shape. The source-reference validator defers malformed shapes, checks current-document identity, unit existence, and range ordering, and reports all defects through bounded aggregates. Source relatedness uses the shared document-order traversal and normalized consecutive-token matching, emitting at most one bounded warning per record when the canonical name does not occur near its cited text. Invalid shape or cited ranges produce no relatedness warnings. Normalize identity validation checks deterministic IDs, canonical names, and duplicate canonical-name or ID ownership. All are deterministic and expose the policy fingerprints used by the production chains.

D&D Combat Validators

Combat shape validation owns the required list, actor, supported turn kind, and non-empty source-reference collection. Combat source-reference validation defers invalid shape, checks source identity, unit existence, and range order, and reports all defects through bounded aggregates. Combat source-relatedness defers invalid shape or ranges, uses the shared traversal to combine overlapping cited units in document order, and emits at most one bounded advisory warning per turn for an unrelated actor. Actors use normalized consecutive-token matching. The normalized-invariants validator owns actor display normalization, canonical source-reference order, chronology, and exact duplicate identity; it defers shape and source-reference failures. All four validators are deterministic and expose local policy fingerprints. In the registered defaults, JSON syntax runs first; combat shape, normalized invariants when applicable, and source-reference validation precede JSON Schema validation; warning-only relatedness runs last.

D&D NPC Interaction Validators

Interaction shape validation owns the required list, registry name, supported kind, and non-empty source-reference collection. Registry validation checks exact membership in the required immutable NPC registry. Source-reference and relatedness validation use the current transcript only; malformed candidates are deferred by later validators and produce no relatedness warning. The normalized-invariants validator owns canonical registry names, source-reference order, chronology, and exact duplicate identity. The production chains run shape, registry, and source-reference checks before JSON Schema validation; relatedness remains warning-only and last.

Production Registration

Production composition occurs through family registrars. The CLI allocates one complete framework registry set and one LLM asset registry. It invokes internal/modules/generic/register, internal/modules/seriatim/register, and internal/modules/dnd/register in that order, then exposes the matching catalog for resolution. The generic and Seriatim registrars own their production leaf registrations. The D&D registrar owns D&D leaf registrations, typed spell, NPC, combat, and interaction default-validator chains, typed append-order specializations, and D&D prompt/schema asset collection. Its registration helpers group module, validator, prompt-asset, and chain composition while retaining artifact-specific merge and clone behavior in the registrar.

Concrete implementation packages do not import generic implementation packages directly. A concrete family's register package is its composition point for specializing reusable generic implementations, while the generic registrar composes only generic children.

Core and framework production packages do not import production extensions. CLI production code is the sole application composition root for extensions and imports only exact family registrar packages. Other production packages, including commands and newly introduced package trees, do not import module packages directly. Compatibility tests in the CLI, core, and framework trees may import roots and implementation leaves directly. Other non-module tests do not receive that exemption. White-box tests within module families retain the production family boundaries. internal/modules/integration is test infrastructure: its black-box tests may compose multiple families, but it is not a production module family or production dependency target.

Adding An Extension

When adding a production module or validator:

1. implement the stage or validator contract and package-local key;
2. expose and test its spec, constructor, and registration function;
3. keep format or domain parsing inside the concrete package;
4. add package-owned prompt/schema assets when the extension is LLM-backed; new LLM-backed D&D extraction modules must follow the stable-to-variable prompt ordering, shared-asset ownership, and cache-boundary policy in LLM Runtime, or document the implemented exception and its evidence there;
5. register it through its package-family registrar and add a default chain there only when production policy requires one;
6. add resolution and composition coverage for capabilities, options, references, and validation behavior;
7. update the selectable-key catalog in Configuration, the relevant external contract, this inventory, and maintained examples when user-visible behavior changes.

Do not add the extension to docs/development.md; that file routes by task and does not inventory implementations.

Tests To Inspect

• Package-local *_test.go files under the module or validator being changed.
• internal/framework/pipeline/typed_resolution_test.go: typed registry, spec, and heterogeneous artifact composition.
• internal/framework/pipeline/profile_test.go: framework binding defaults and profile resolution.
• internal/cli/production_contract_test.go: production catalog, config resolution, and composition smoke coverage.
• internal/cli/example_contract_test.go: maintained example ownership.
• internal/framework/promptfs/*_test.go and internal/modules/dnd/shared/*_test.go: shared prompt and reference assembly.
• internal/modules/integration/*_test.go: black-box composition across production extension domains.