notarius/docs/internal/modules.md

Modules

Production modules live under internal/modules. Each module implements one contract from internal/framework/contracts, exposes a ModuleSpec, and registers itself with the matching pipeline registry.

The CLI production catalog currently registers only the modules listed here.

Contract Pattern

A production module package should provide:

• a stable module key;
• a constructor such as New;
• the relevant contract implementation;
• ModuleSpec;
• Register;
• focused tests for registration, options, contract behavior, and errors.

Module specs should describe capabilities accurately. Resolution uses specs to reject incompatible pipelines before execution.

Chunk, extract, merge, and normalize modules that accept auxiliary reference material must declare slots through both ReferenceSlots() and ModuleSpec().ReferenceSlots. The runtime slot list and registry metadata should match so config validation can inspect slots without constructing module instances. A slot declaration names the slot, whether it is required, accepted media types, whether multiple items are allowed, and any byte limit. Empty AcceptedMediaTypes means any inferred media type is accepted, though the file must still be UTF-8 text. When a slot declares accepted media types, Notarius compares the canonical base media type inferred from the file extension, case-insensitively and without parameters.

The resolver materializes reference content for chunk, extractor, merger, and normalizer targets. Runtime delivery uses contracts.ChunkRequest.References, contracts.ExtractionRequest.References, contracts.MergeRequest.References, and contracts.NormalizeRequest.References. Reference material is not source evidence and must not be converted into SourceRef values. If a module prompt uses references, pass them as prompt input materials through the structured LLM request. Prompt metadata hashes remain based on prompt asset source, not rendered reference bytes.

LLM-backed modules own Scriptorium prompt definitions and response schemas in their embedded assets. Module-owned prompts live under each module's shallow assets/prompts tree and schemas live under assets/schemas. Generic shared prompt filesystem composition lives under internal/modules/sharedassets. Common D&D prompt fragments, reference slot helpers, prompt input assembly, and reference rendering live under internal/modules/sharedassets/dnd. Module contracts should expose prompt IDs, versions, input material names, and non-secret prompt/schema hashes through manifest metadata; they should not expose Scriptorium public types through chunk, extract, merge, or normalize contracts.

Chunk modules receive the structured LLM client, configured Scriptorium profile ID, prompt session ID, and raw source input material through contracts.ChunkRequest when they need model-backed chunking. The pipeline runner validates generic chunk result invariants before extraction; module-owned policies may be stricter but must stay within the module package.

Normalize modules receive the structured LLM client, configured Scriptorium profile ID, prompt session ID, and reference material through contracts.NormalizeRequest when they need model-backed reconciliation.

Merge modules receive the structured LLM client, configured Scriptorium profile ID, prompt session ID, raw source input material, and reference material through contracts.MergeRequest when they need model-backed merge behavior.

seriatim Input

Package: internal/modules/input/seriatim

The seriatim adapter parses Seriatim transcript JSON into a generic source document. It owns transcript JSON details, source ID selection, source digest creation, transcript segment validation, and segment metadata mapping.

Provides:

• source.transcript
• transcript.speaker
• transcript.timestamps

External JSON shape belongs in the Seriatim integration doc.

generic Chunker

Package: internal/modules/chunk/generic

The generic chunker splits source units into ordered chunks. It validates the source document, clones source units, assigns chunk IDs such as chunk-000001, and records chunk metadata for start unit, end unit, and unit count.

The pipeline runner canonicalizes chunk units from the source document by integer ID before extractors and mergers run. Chunkers also populate chunk start and end unit IDs, content bytes, and media type. Chunker-owned context should stay in SourceChunk.Metadata.

Options:

• max_units: positive integer, default 50;
• overlap_units: non-negative integer, default 0, and less than max_units.

Provides:

• chunks

dnd/scenes Chunker

Package: internal/modules/chunk/dnd/scenes

The dnd/scenes chunker uses the structured LLM client to divide transcript source units into coherent D&D scenes. It supplies the embedded Scriptorium prompt ID, prompt version, transcript input material, response schema, and session ID to the runtime; validates model-authored source-unit boundaries; and converts each scene into a deterministic source chunk.

Its prompt definition lives under assets/prompts and its schema under assets/schemas. Shared reusable D&D prompt fragments are provided by internal/modules/sharedassets/dnd and referenced from prompt definitions under ./sharedassets/.

Requires:

• source.transcript

Provides:

• chunks
• chunks.scenes

Options: none. Non-empty options are rejected.

The chunker enforces full source-unit coverage from the first source unit to the last, sequential contiguous scenes, and no overlap. Its LLM-facing schema uses integer start_unit_id and end_unit_id values matching source-unit IDs. It assigns chunk IDs such as scene-000001, emits JSON chunk content, and stores scene metadata including title, primary mode, participants, summary, boundary note, confidence, boundary unit IDs, and unit count. Boundary caveats become warnings with reason code scene_boundary_caveat. Whitespace-only caveats are treated as malformed structured output rather than silently dropped.

Malformed model output fails explicitly rather than falling back to another chunker. The chunker exposes prompt and response-schema provenance through top-level module_metadata.chunker without raw prompts, raw schemas, source text, or secrets.

dnd/spells Extractor

Package: internal/modules/extract/dnd/spells

The dnd/spells extractor owns D&D spell-cast extraction semantics. It supplies the embedded Scriptorium prompt ID, prompt version, chunk-scoped transcript input material, reference input materials, response schema, and session ID to the runtime; then returns the structured LLM spell_casts response as raw JSON. Its LLM-facing source-reference schema uses integer start_unit_id and end_unit_id values matching source-unit IDs.

Its prompt definition lives under assets/prompts and its schema under assets/schemas. Shared reusable D&D prompt fragments are provided by internal/modules/sharedassets/dnd and referenced from prompt definitions under ./sharedassets/.

Requires:

• chunks
• source.transcript

Provides:

• dnd.spell_casts

Response schema identity:

• schema ID: notarius.dnd.spells
• schema name: notarius_dnd_spells_v1
• schema version: v1

The extractor adds prompt and response-schema provenance to lane manifest metadata under artifact_lanes[].metadata.extractor. Durable raw output details belong in the D&D spell raw output contract.

The dnd/scenes chunker and dnd/spells extractor declare optional players, party, and glossary reference slots accepting UTF-8 plain text, Markdown, YAML, or JSON. They also accept roster as a deprecated compatibility alias for party. Their prompts frame references as supporting disambiguation material only; spell-cast artifacts must still be grounded in the source transcript.

appendorder Merger

Package: internal/modules/merge/appendorder

The appendorder merger preserves chunk order for raw extract outputs. A single JSON extract output is passed through as the merge output. Multiple JSON object outputs with one common top-level array field are merged by concatenating that array field in chunk order. Other valid JSON shapes are merged as a JSON array of decoded values in chunk order. Non-JSON media types and invalid JSON are rejected.

Provides:

• merged

noop Normalizer

Package: internal/modules/normalize/noop

The noop normalizer clones the raw merge output and returns it unchanged.

Requires:

• merged

Provides:

• normalized

json Output

Package: internal/modules/output/json

The json output encoder converts normalized raw outputs, rejected raw outputs, warnings, and the run manifest into logical JSON output files. It writes one payload file per lane under lanes/ and sanitizes lane IDs for file names. Normalized output payloads must be valid application/json.

Requires:

• normalized

Provides:

• encoded

Durable output file shapes belong in the JSON output contract. Operator behavior belongs in Operations.

Production Registration

Production registration is centralized in internal/cli/catalog.go.

Do not make framework code import production modules. The CLI wires production modules at the application boundary; tests may provide fake registries or fake catalogs directly.

Adding A Module

When adding a module, keep source-format and extraction-domain boundaries clear:

• input modules may know external source formats;
• extract modules may know artifact semantics and prompt/schema assets;
• merge and normalize modules own raw output combination and reconciliation;
• output modules own serialization, not diagnostics or CLI reporting.

Update Development, Configuration, internal docs, integration docs, and examples when the new module becomes implemented production behavior.