notarius/docs/internal/modules.md

# Module And Validator Internals

Production module and validator implementations live under their domain-first
trees in `internal/modules`.
The selectable keys, configuration options, reference slots, and default
validator chain are canonical in the
[module](../config.md#implemented-production-modules) and
[validator](../config.md#implemented-production-validators) catalogs in
Configuration.

## Extension Pattern

A stage module package provides a stable key, constructor, contract
implementation, `ModuleSpec`, `Register`, and focused behavior and registration
tests. A validator package follows the same pattern with `ValidatorSpec` and the
validator registry. Package-family registrars compose those leaf registrations
into the production catalog and own family-level policy such as default
validator chains and prompt asset collection.

Production input, chunk, output, and D&D spell-extract packages register strict
option decoders and run-local builders. Preparation decodes their options into
implementation-owned values and injects dependencies plus the materialized
reference set for the selected target. Each builder receives an isolated clone
of that set; input and output builders receive no references. The spell
extractor is typed over the canonical D&D model. D&D validators, merge, and
normalize use typed variants; JSON representation validators use serialized
requests; and unconditional validators expose separate chunk and typed
variants. The D&D production registrar registers only the canonical typed spell
implementations.

Prepared extractors, extract validators, and codecs may be reused concurrently
by the run-wide extract pool. Production implementations are immutable after
construction: they retain only typed options, immutable assets, or the shared
concurrency-safe LLM client. Implementations that introduce mutable state must
synchronize that state without creating a separate provider scheduler.

Specs expose capability and execution metadata without constructing an
implementation. Registry entries separately expose option validation and
run-local construction. Chunk, extract, merge, and normalize modules that accept
auxiliary material declare identical reference slots from both
`ReferenceSlots()` and `ModuleSpec().ReferenceSlots`; registration tests enforce
that agreement. Runtime delivery uses the corresponding stage request's
`References` field.

LLM-backed extensions own their prompt definitions and response schemas under
package-local embedded assets. Shared filesystem composition belongs in
`internal/framework/promptfs`; reusable D&D prompt fragments, reference
declarations, prompt-input assembly, and source-unit helpers belong in
`internal/modules/dnd/shared`. Stage contracts expose only Notarius structured-
completion types, not Scriptorium public types.

Reference material may inform a module or prompt but must not become source
evidence. The resolver and materializer behavior is described in
[Pipeline Internals](pipeline.md#reference-materialization).

## Domain Reference Data

### `internal/modules/dnd/spells/catalog`

The spell catalog package owns the embedded, versioned D&D 5e 2014 SRD spell
reference data. Its strict JSON asset contains one canonical record per spell,
including spell level and all applicable class memberships. `LoadSRD5E2014`
validates catalog identity, provenance metadata, ordering, uniqueness, levels,
classes, aliases, and lookup-key collisions before exposing immutable copies.

Lookup is case-insensitive and normalizes whitespace and common apostrophe
variants while preserving source punctuation in canonical display names. The
catalog contains 319 unique spells and 779 class memberships. Source and
license details live beside the asset in `SOURCES.md`. This domain-owned data is
separate from `internal/modules/dnd/shared`, which is reserved for reusable
prompt and source-reference machinery.

`ResolveEffectiveCatalog` builds the immutable recognition view used by the
spell extractor and catalog validator. It starts with the embedded SRD catalog
and optionally applies one strict JSON overlay from the `spell_catalog` item in
a materialized reference set. Overlay catalogs are ordered by ID, may add names
and aliases, and may augment an existing canonical spell without replacing its
display name. Cross-spell lookup collisions are errors. The effective view
exposes sorted canonical names, normalized lookup, overlay identities, and a
semantic digest; overlay content remains contextual reference material rather
than source evidence. Its external JSON contract is defined in the
[spell-catalog overlay contract](../integrations/dnd-spell-catalog-overlays.md).

## Input Adapter

### `internal/modules/seriatim/input/transcript`

The adapter decodes the supported transcript JSON, selects the source identity,
computes canonical source provenance, validates segments, and maps each segment
into a generic source unit with a self-reference plus speaker and timestamp
metadata. It accepts no module options. Its spec advertises the transcript
capabilities consumed by D&D modules.

Parsing is strict about required values and duplicate unit IDs but deliberately
ignores unrelated Seriatim fields. The external format and derived-identity
rules are defined in the
[Seriatim contract](../integrations/seriatim.md).

## Chunkers

Chunkers implement `contracts.Chunker.Plan`. A plan identifies ordered source
unit ranges and may carry optional namespaced JSON annotations; it does not
contain materialized chunk content. The framework canonicalizes annotations,
validates ranges against the current source, and materializes chunk IDs,
indexes, references, content, units, and generic metadata. Materialized source
unit metadata is independently owned. Annotation
namespaces remain optional data: generic framework code and downstream modules
must not require D&D scene annotations or import `dnd/scenes`.

### `internal/modules/generic/chunk/units`

The generic chunker validates the source document and returns ranges over units
in configured windows. Overlap changes the next window start but never reorders
units. Framework materialization derives the resulting chunk identity and
generic metadata from those ranges.

The accepted options and defaults are defined in
[Configuration](../config.md#implemented-production-modules). Generic
framework validation canonicalizes the returned unit slices before extraction.
The chunker decodes its options during construction and retains only the typed
window settings used by `Plan`.

### `internal/modules/dnd/chunk/scenes`

The scene chunker prepares a structured Scriptorium request from the full
transcript, session, and optional D&D reference inputs. It validates the model's
scene boundaries against source-unit IDs and converts them into deterministic
plan ranges with optional scene annotations. Preparation injects the shared
structured LLM client into the chunker; `Plan`
supplies only the run-specific profile, session, source, references, and
metadata.

Scene validation requires sequential, contiguous, non-overlapping coverage from
the first source unit through the last. Scene descriptions, boundaries,
confidence, and participants are module-owned annotations. Boundary caveats
become warnings. Malformed
structured output is returned as an error; there is no fallback chunker.

The package embeds its prompt and response schema and reports their non-secret
identity and hashes through singleton module metadata. Shared D&D assets supply
reference declarations and prompt inputs; their user-facing keys and accepted
file types remain canonical in [Configuration](../config.md).

## Extractor

### `internal/modules/dnd/extract/spells`

The spell extractor prepares a structured request from one chunk, the
chunk-scoped source input, the session, and optional D&D reference inputs. It
decodes the model response, assigns the generic source identity to every source
reference, canonicalizes duplicate references, orders spell casts by their
earliest cited unit, and returns `dnd.SpellList`.

The extractor owns its private model-response DTO, embedded prompt, LLM response
schema, strict option decoder, injected shared LLM client, and prompt/schema
manifest metadata. During preparation it resolves the optional `spell_catalog`
reference into an immutable effective catalog and adds a generated
canonical-name-only JSON input to every structured completion request. Overlay
failures therefore stop construction before source parsing or an LLM call;
campaign references remain separate disambiguation inputs and never become
source evidence. Both the extractor and deterministic catalog validator expose
the effective base-plus-overlay semantic digest as scoped prepared-component
checkpoint identity. Raw overlay provenance independently covers file-byte
changes, while the semantic digest also invalidates reuse when the embedded
catalog or catalog composition changes. The separate
`internal/modules/dnd/codec/spells` package
owns the durable schema and stable JSON representation for artifact kind
`dnd/spell-list`. The runner keeps the result typed through validators and later
stages, using the codec only for checkpoint, debug, and output boundaries.
Shared D&D helpers keep prompt input
names and source-unit reference conversion consistent with the scene chunker.

The durable payload and manifest metadata shapes are defined in the
[D&D spell artifact contract](../integrations/dnd-spell-artifacts.md).

## Merger And Normalizer

### `internal/modules/generic/merge/appendorder`

The merger passes typed values to an injected combine function in framework
source-chunk order. The D&D registrar specializes it with a spell-list append
function.

### `internal/modules/generic/normalize/noop`

The normalizer returns the merged domain value unchanged and is reusable for
any registered artifact type.

### `internal/modules/dnd/normalize/spells`

The typed spell normalizer resolves the optional `spell_catalog` reference into
the same immutable SRD-plus-overlay effective catalog used by spell extraction
and catalog validation. It performs no LLM calls. For each spell cast it
canonicalizes recognized names using the catalog's case, whitespace,
apostrophe, and alias rules; sorts source references by source identity and
unit boundaries; removes only exact reference duplicates; and emits bounded,
scoped warnings for each mutation or unresolved name.

After those per-cast changes, it collapses only casts with the same canonical
spell, case-folded and whitespace-normalized caster, and complete non-empty
valid source-reference set. It retains the first occurrence and its caster,
effect, narrative description, and stable order. Unknown names, empty or
invalid evidence, and adjacent or overlapping but different ranges remain
unchanged for validation.

The normalizer exposes the effective catalog digest as its independently scoped
`effective_catalog` checkpoint fingerprint and reports catalog base ID, digest,
and overlay IDs as manifest metadata. Catalog contents, reference paths, and
raw overlay bytes are not included in either surface. The normalize-stage
reference is stage-local, so an overlay-capable pipeline binds the catalog
independently for extraction and normalization.

## Output Encoder

### `internal/modules/generic/output/json`

The JSON encoder sorts normalized results by lane, derives collision-checked
safe logical names, pretty-prints JSON payloads, and assembles the logical index,
manifest, rejected-result, warning, and lane files. Invalid JSON, unsupported
media types, unsafe names, and sanitized-name collisions are errors.

The encoder returns logical files only. The CLI places them on disk, and the
[JSON output contract](../integrations/json-output.md) defines their external
paths and schemas.

## Generic Validators

The generic validator implementations live under
`internal/modules/generic/validate`.

The unconditional accept and reject validators provide explicit chunk and
typed-artifact variants used primarily for controlled composition and tests.

The serialized JSON syntax validator uses `encoding/json` to reject malformed
representation bytes. The serialized JSON Schema validator requires schema
bytes, parses the instance and schema with `jsonschema`, and distinguishes
payload rejection from schema loading or compilation errors. The framework
serialized-validation request carries either canonical chunk bytes or artifact
codec bytes according to its target context. Neither validator calls the LLM.

## D&D Spell Validators

All four validators receive `dnd.SpellList` directly. The shape validator
rejects missing or empty spell fields and empty reference lists. The catalog
validator defers when shape is invalid, then checks every non-empty spell name
against the immutable effective SRD and overlay catalog. It accepts normalized
canonical names and aliases without rewriting the artifact; unknown names
reject the complete result with bounded, stable index/name diagnostics. The
source-reference validator applies generic source-reference validation to every
cited range. The relatedness validator warns when a case-insensitive spell name
is absent from all cited source text.

These validators are deterministic. Their selectable keys and production order
are defined in
[Configuration](../config.md#implemented-production-validators); their durable
payload rules are defined in the
[artifact contract](../integrations/dnd-spell-artifacts.md).

## Production Registration

Production composition occurs through family registrars. The CLI allocates one
complete framework registry set and one LLM asset registry. It invokes
`internal/modules/generic/register`,
`internal/modules/seriatim/register`, and `internal/modules/dnd/register` in
that order, then exposes the matching catalog for resolution. The generic and
Seriatim registrars own their production leaf registrations. The D&D registrar
owns D&D leaf registrations, the typed spell default-validator chains, and D&D
prompt/schema asset collection.

Concrete implementation packages do not import generic implementation
packages directly. A concrete family's `register` package is its composition
point for specializing reusable generic implementations, while the generic
registrar composes only generic children.

Core and framework production packages do not import production extensions.
CLI production code is the sole application composition root for extensions
and imports only exact family registrar packages. Other production packages,
including commands and newly introduced package trees, do not import module
packages directly. Compatibility tests in the CLI, core, and framework trees
may import roots and implementation leaves directly. Other non-module tests do
not receive that exemption. White-box tests within module families retain the
production family boundaries. `internal/modules/integration` is test
infrastructure: its black-box tests may compose multiple families, but it is
not a production module family or production dependency target.

## Adding An Extension

When adding a production module or validator:

1. implement the stage or validator contract and package-local key;
2. expose and test its spec, constructor, and registration function;
3. keep format or domain parsing inside the concrete package;
4. add package-owned prompt/schema assets when the extension is LLM-backed;
5. register it through its package-family registrar and add a default chain
   there only when production policy requires one;
6. add resolution and composition coverage for capabilities, options,
   references, and validation behavior;
7. update the selectable-key catalog in [Configuration](../config.md), the
   relevant external contract, this inventory, and maintained examples when
   user-visible behavior changes.

Do not add the extension to `docs/development.md`; that file routes by task and
does not inventory implementations.

## Tests To Inspect

- Package-local `*_test.go` files under the module or validator being changed.
- `internal/framework/pipeline/typed_resolution_test.go`: typed registry, spec,
  and heterogeneous artifact composition.
- `internal/framework/pipeline/profile_test.go`: framework binding defaults and
  profile resolution.
- `internal/cli/production_contract_test.go`: production catalog, config
  resolution, and composition smoke coverage.
- `internal/cli/example_contract_test.go`: maintained example ownership.
- `internal/framework/promptfs/*_test.go` and
  `internal/modules/dnd/shared/*_test.go`: shared prompt and reference assembly.
- `internal/modules/integration/*_test.go`: black-box composition across
  production extension domains.