236 lines
7.7 KiB
Markdown
236 lines
7.7 KiB
Markdown
# Modules
|
|
|
|
Production modules live under `internal/modules`. Each module implements one
|
|
contract from `internal/framework/contracts`, exposes a `ModuleSpec`, and
|
|
registers itself with the matching pipeline registry.
|
|
|
|
The CLI production catalog currently registers only the modules listed here.
|
|
|
|
## Contract Pattern
|
|
|
|
A production module package should provide:
|
|
|
|
- a stable module key;
|
|
- a constructor such as `New`;
|
|
- the relevant contract implementation;
|
|
- `ModuleSpec`;
|
|
- `Register`;
|
|
- focused tests for registration, options, contract behavior, and errors.
|
|
|
|
Module specs should describe capabilities accurately. Resolution uses specs to
|
|
reject incompatible pipelines before execution.
|
|
|
|
Extractor modules that accept auxiliary reference material must declare slots
|
|
through both `ReferenceSlots()` and `ModuleSpec().ReferenceSlots`. The runtime
|
|
slot list and registry metadata should match so config validation can inspect
|
|
slots without constructing extractor instances. A slot declaration names the
|
|
slot, whether it is required, accepted media types, whether multiple items are
|
|
allowed, and any byte limit. Empty `AcceptedMediaTypes` means any inferred
|
|
media type is accepted, though the file must still be UTF-8 text. When a slot
|
|
declares accepted media types, Notarius compares the canonical base media type
|
|
inferred from the file extension, case-insensitively and without parameters.
|
|
|
|
Reference content is delivered only to the lane extractor through
|
|
`contracts.ExtractionRequest.References`. It is not source evidence and must not
|
|
be converted into `SourceRef` values. If a module prompt uses references, load
|
|
the prompt bundle with the same declared slots and render with
|
|
`RenderUserSystemWithReferences`. Prompt templates may use the `reference`
|
|
function for content and the `hasreference` function for conditional sections.
|
|
Prompt metadata hashes remain based on template source, not rendered reference
|
|
bytes.
|
|
|
|
Chunk modules receive the structured LLM client through `contracts.ChunkRequest`
|
|
when they need model-backed chunking. The pipeline runner validates generic
|
|
chunk result invariants before extraction; module-owned policies may be stricter
|
|
but must stay within the module package.
|
|
|
|
## `seriatim` Input
|
|
|
|
Package: `internal/modules/input/seriatim`
|
|
|
|
The `seriatim` adapter parses Seriatim transcript JSON into a generic source
|
|
document. It owns transcript JSON details, source ID selection, source digest
|
|
creation, transcript segment validation, and segment metadata mapping.
|
|
|
|
Provides:
|
|
|
|
- `source.transcript`
|
|
- `transcript.speaker`
|
|
- `transcript.timestamps`
|
|
|
|
External JSON shape belongs in the Seriatim integration doc.
|
|
|
|
## `generic` Chunker
|
|
|
|
Package: `internal/modules/chunk/generic`
|
|
|
|
The `generic` chunker splits source units into ordered chunks. It validates the
|
|
source document, clones source units, assigns chunk IDs such as `chunk-000001`,
|
|
and records chunk metadata for start unit, end unit, and unit count.
|
|
|
|
The pipeline runner canonicalizes chunk units from the source document by ID
|
|
before extractors and mergers run. Chunker-owned context should stay in
|
|
`SourceChunk.Metadata`.
|
|
|
|
Options:
|
|
|
|
- `max_units`: positive integer, default `50`;
|
|
- `overlap_units`: non-negative integer, default `0`, and less than
|
|
`max_units`.
|
|
|
|
Provides:
|
|
|
|
- `chunks`
|
|
|
|
## `dnd/scenes` Chunker
|
|
|
|
Package: `internal/modules/chunk/dnd/scenes`
|
|
|
|
The `dnd/scenes` chunker uses the structured LLM client to divide transcript
|
|
source units into coherent D&D scenes. It renders embedded prompts, loads the
|
|
embedded structured response schema, validates model-authored source-unit
|
|
boundaries, and converts each scene into a deterministic source chunk.
|
|
|
|
Requires:
|
|
|
|
- `source.transcript`
|
|
|
|
Provides:
|
|
|
|
- `chunks`
|
|
- `chunks.scenes`
|
|
|
|
Options: none. Non-empty options are rejected.
|
|
|
|
The chunker enforces full source-unit coverage from the first source unit to the
|
|
last, exact source-unit IDs, sequential contiguous scenes, and no overlap. It
|
|
assigns chunk IDs such as `scene-000001` and stores scene metadata including
|
|
title, primary mode, participants, summary, boundary note, confidence, boundary
|
|
unit IDs, and unit count. Boundary caveats become warnings with reason code
|
|
`scene_boundary_caveat`. Whitespace-only caveats are treated as malformed
|
|
structured output rather than silently dropped.
|
|
|
|
Malformed model output fails explicitly rather than falling back to another
|
|
chunker. The chunker exposes prompt and response-schema provenance through
|
|
top-level `module_metadata.chunker` without raw prompts, raw schemas, source
|
|
text, or secrets.
|
|
|
|
## `dnd/spells` Extractor
|
|
|
|
Package: `internal/modules/extract/dnd/spells`
|
|
|
|
The `dnd/spells` extractor owns D&D spell-cast artifact semantics. It renders
|
|
embedded prompts, loads the embedded structured response schema, calls the
|
|
structured LLM client, converts spell-cast responses into artifact candidates,
|
|
and supplies deterministic validators.
|
|
|
|
Requires:
|
|
|
|
- `chunks`
|
|
- `source.transcript`
|
|
|
|
Provides:
|
|
|
|
- `dnd.spell_casts`
|
|
|
|
Artifact type and schema version:
|
|
|
|
- artifact type: `dnd.spell_cast`
|
|
- schema version: `v1`
|
|
|
|
The extractor adds prompt and response-schema provenance to lane manifest
|
|
metadata under `artifact_lanes[].metadata.extractor`. Durable artifact payload
|
|
details belong in the
|
|
[D&D spell artifact contract](../integrations/dnd-spell-artifacts.md).
|
|
|
|
The extractor declares optional `roster` and `glossary` reference slots
|
|
accepting UTF-8 text without narrowing accepted media types. Its prompt frames
|
|
references as supporting disambiguation material only; spell-cast artifacts must
|
|
still be grounded in the source transcript.
|
|
|
|
## D&D Spell Validators
|
|
|
|
The spell extractor returns two built-in validators:
|
|
|
|
- `dnd/spells/shape`: rejects malformed payloads and missing required fields.
|
|
- `dnd/spells/source_refs`: rejects candidates without valid source references.
|
|
It also emits a warning when the extracted spell name is not found in the
|
|
cited source text.
|
|
|
|
Reason codes include:
|
|
|
|
- `invalid_payload`
|
|
- `missing_required_field`
|
|
- `missing_source_ref`
|
|
- `invalid_source_ref`
|
|
- `spell_not_near_source`
|
|
|
|
These validators are supplied by the extractor when no validators are configured
|
|
for the lane.
|
|
|
|
## `appendorder` Merger
|
|
|
|
Package: `internal/modules/merge/appendorder`
|
|
|
|
The `appendorder` merger clones and appends candidates in chunk order. It does
|
|
not deduplicate or reconcile candidates.
|
|
|
|
Provides:
|
|
|
|
- `merged`
|
|
|
|
## `noop` Normalizer
|
|
|
|
Package: `internal/modules/normalize/noop`
|
|
|
|
The `noop` normalizer clones merged candidates and returns them unchanged.
|
|
|
|
Requires:
|
|
|
|
- `merged`
|
|
|
|
Provides:
|
|
|
|
- `normalized`
|
|
|
|
## `json` Output
|
|
|
|
Package: `internal/modules/output/json`
|
|
|
|
The `json` output encoder converts approved artifacts, rejected artifacts,
|
|
warnings, and the run manifest into logical JSON output files. It groups
|
|
approved artifacts by artifact type and sanitizes artifact-type file names.
|
|
|
|
Requires:
|
|
|
|
- `normalized`
|
|
|
|
Provides:
|
|
|
|
- `encoded`
|
|
|
|
Durable output file shapes belong in the
|
|
[JSON output contract](../integrations/json-output.md). Operator behavior
|
|
belongs in [Operations](../operations.md).
|
|
|
|
## Production Registration
|
|
|
|
Production registration is centralized in `internal/cli/catalog.go`.
|
|
|
|
Do not make framework code import production modules. The CLI wires production
|
|
modules at the application boundary; tests may provide fake registries or fake
|
|
catalogs directly.
|
|
|
|
## Adding A Module
|
|
|
|
When adding a module, keep source-format and extraction-domain boundaries clear:
|
|
|
|
- input modules may know external source formats;
|
|
- extract modules may know artifact semantics and prompt/schema assets;
|
|
- merge and normalize modules own candidate combination and reconciliation;
|
|
- output modules own serialization, not diagnostics or CLI reporting.
|
|
|
|
Update [Development](../policy/development.md), [Configuration](../config.md),
|
|
internal docs, integration docs, and examples when the new module becomes
|
|
implemented production behavior.
|