notarius/docs/internal/modules.md

Modules

Production modules live under internal/modules. Each module implements one contract from internal/framework/contracts, exposes a ModuleSpec, and registers itself with the matching pipeline registry.

The CLI production catalog currently registers only the modules listed here.

Contract Pattern

A production module package should provide:

• a stable module key;
• a constructor such as New;
• the relevant contract implementation;
• ModuleSpec;
• Register;
• focused tests for registration, options, contract behavior, and errors.

Module specs should describe capabilities accurately. Resolution uses specs to reject incompatible pipelines before execution.

Chunk, extract, and normalize modules that accept auxiliary reference material must declare slots through both ReferenceSlots() and ModuleSpec().ReferenceSlots. The runtime slot list and registry metadata should match so config validation can inspect slots without constructing module instances. A slot declaration names the slot, whether it is required, accepted media types, whether multiple items are allowed, and any byte limit. Empty AcceptedMediaTypes means any inferred media type is accepted, though the file must still be UTF-8 text. When a slot declares accepted media types, Notarius compares the canonical base media type inferred from the file extension, case-insensitively and without parameters.

The resolver materializes reference content for chunk, extractor, and normalizer targets. Runtime delivery uses contracts.ChunkRequest.References, contracts.ExtractionRequest.References, and contracts.NormalizeRequest.References. Reference material is not source evidence and must not be converted into SourceRef values. If a module prompt uses references, pass them as prompt input materials through the structured LLM request. Prompt metadata hashes remain based on prompt asset source, not rendered reference bytes.

LLM-backed modules own Scriptorium prompt definitions and response schemas in their embedded assets. Module-owned prompts live under each module's shallow assets/prompts tree and schemas live under assets/schemas. Generic shared prompt filesystem composition lives under internal/modules/sharedassets. Common D&D prompt fragments, reference slot helpers, prompt input assembly, and reference rendering live under internal/modules/sharedassets/dnd. Module contracts should expose prompt IDs, versions, input material names, and non-secret prompt/schema hashes through manifest metadata; they should not expose Scriptorium public types through chunk, extract, or normalize contracts.

Chunk modules receive the structured LLM client, configured Scriptorium profile ID, prompt session ID, and raw source input material through contracts.ChunkRequest when they need model-backed chunking. The pipeline runner validates generic chunk result invariants before extraction; module-owned policies may be stricter but must stay within the module package.

Normalize modules receive the structured LLM client, configured Scriptorium profile ID, prompt session ID, and reference material through contracts.NormalizeRequest when they need model-backed reconciliation.

seriatim Input

Package: internal/modules/input/seriatim

The seriatim adapter parses Seriatim transcript JSON into a generic source document. It owns transcript JSON details, source ID selection, source digest creation, transcript segment validation, and segment metadata mapping.

Provides:

• source.transcript
• transcript.speaker
• transcript.timestamps

External JSON shape belongs in the Seriatim integration doc.

generic Chunker

Package: internal/modules/chunk/generic

The generic chunker splits source units into ordered chunks. It validates the source document, clones source units, assigns chunk IDs such as chunk-000001, and records chunk metadata for start unit, end unit, and unit count.

The pipeline runner canonicalizes chunk units from the source document by ID before extractors and mergers run. Chunker-owned context should stay in SourceChunk.Metadata.

Options:

• max_units: positive integer, default 50;
• overlap_units: non-negative integer, default 0, and less than max_units.

Provides:

• chunks

dnd/scenes Chunker

Package: internal/modules/chunk/dnd/scenes

The dnd/scenes chunker uses the structured LLM client to divide transcript source units into coherent D&D scenes. It supplies the embedded Scriptorium prompt ID, prompt version, transcript input material, response schema, and session ID to the runtime; validates model-authored source-unit boundaries; and converts each scene into a deterministic source chunk.

Its prompt definition lives under assets/prompts and its schema under assets/schemas. Shared reusable D&D prompt fragments are provided by internal/modules/sharedassets/dnd and referenced from prompt definitions under ./sharedassets/.

Requires:

• source.transcript

Provides:

• chunks
• chunks.scenes

Options: none. Non-empty options are rejected.

The chunker enforces full source-unit coverage from the first source unit to the last, exact source-unit IDs, sequential contiguous scenes, and no overlap. It assigns chunk IDs such as scene-000001 and stores scene metadata including title, primary mode, participants, summary, boundary note, confidence, boundary unit IDs, and unit count. Boundary caveats become warnings with reason code scene_boundary_caveat. Whitespace-only caveats are treated as malformed structured output rather than silently dropped.

Malformed model output fails explicitly rather than falling back to another chunker. The chunker exposes prompt and response-schema provenance through top-level module_metadata.chunker without raw prompts, raw schemas, source text, or secrets.

dnd/spells Extractor

Package: internal/modules/extract/dnd/spells

The dnd/spells extractor owns D&D spell-cast artifact semantics. It supplies the embedded Scriptorium prompt ID, prompt version, transcript and reference input materials, response schema, and session ID to the runtime; converts spell-cast responses into artifact candidates; and supplies deterministic validators.

Its prompt definition lives under assets/prompts and its schema under assets/schemas. Shared reusable D&D prompt fragments are provided by internal/modules/sharedassets/dnd and referenced from prompt definitions under ./sharedassets/.

Requires:

• chunks
• source.transcript

Provides:

• dnd.spell_casts

Artifact type and schema version:

• artifact type: dnd.spell_cast
• schema version: v1

The extractor adds prompt and response-schema provenance to lane manifest metadata under artifact_lanes[].metadata.extractor. Durable artifact payload details belong in the D&D spell artifact contract.

The dnd/scenes chunker and dnd/spells extractor declare optional players, party, and glossary reference slots accepting UTF-8 plain text, Markdown, YAML, or JSON. They also accept roster as a deprecated compatibility alias for party. Their prompts frame references as supporting disambiguation material only; spell-cast artifacts must still be grounded in the source transcript.

D&D Spell Validators

The spell extractor returns two built-in validators:

• dnd/spells/shape: rejects malformed payloads and missing required fields.
• dnd/spells/source_refs: rejects candidates without valid source references. It also emits a warning when the extracted spell name is not found in the cited source text.

Reason codes include:

• invalid_payload
• missing_required_field
• missing_source_ref
• invalid_source_ref
• spell_not_near_source

These validators are supplied by the extractor when no validators are configured for the lane.

appendorder Merger

Package: internal/modules/merge/appendorder

The appendorder merger clones and appends candidates in chunk order. It does not deduplicate or reconcile candidates.

Provides:

• merged

noop Normalizer

Package: internal/modules/normalize/noop

The noop normalizer clones merged candidates and returns them unchanged.

Requires:

• merged

Provides:

• normalized

json Output

Package: internal/modules/output/json

The json output encoder converts approved artifacts, rejected artifacts, warnings, and the run manifest into logical JSON output files. It groups approved artifacts by artifact type and sanitizes artifact-type file names.

Requires:

• normalized

Provides:

• encoded

Durable output file shapes belong in the JSON output contract. Operator behavior belongs in Operations.

Production Registration

Production registration is centralized in internal/cli/catalog.go.

Do not make framework code import production modules. The CLI wires production modules at the application boundary; tests may provide fake registries or fake catalogs directly.

Adding A Module

When adding a module, keep source-format and extraction-domain boundaries clear:

• input modules may know external source formats;
• extract modules may know artifact semantics and prompt/schema assets;
• merge and normalize modules own candidate combination and reconciliation;
• output modules own serialization, not diagnostics or CLI reporting.

Update Development, Configuration, internal docs, integration docs, and examples when the new module becomes implemented production behavior.