260 lines
9.8 KiB
Markdown
260 lines
9.8 KiB
Markdown
# Modules
|
|
|
|
Production modules live under `internal/modules`. Each module implements one
|
|
contract from `internal/framework/contracts`, exposes a `ModuleSpec`, and
|
|
registers itself with the matching pipeline registry.
|
|
|
|
The CLI production catalog currently registers only the modules listed here.
|
|
|
|
## Contract Pattern
|
|
|
|
A production module package should provide:
|
|
|
|
- a stable module key;
|
|
- a constructor such as `New`;
|
|
- the relevant contract implementation;
|
|
- `ModuleSpec`;
|
|
- `Register`;
|
|
- focused tests for registration, options, contract behavior, and errors.
|
|
|
|
Module specs should describe capabilities accurately. Resolution uses specs to
|
|
reject incompatible pipelines before execution.
|
|
|
|
Chunk, extract, merge, and normalize modules that accept auxiliary reference material
|
|
must declare slots through both `ReferenceSlots()` and
|
|
`ModuleSpec().ReferenceSlots`. The runtime slot list and registry metadata
|
|
should match so config validation can inspect slots without constructing module
|
|
instances. A slot declaration names the slot, whether it is required, accepted
|
|
media types, whether multiple items are allowed, and any byte limit. Empty
|
|
`AcceptedMediaTypes` means any inferred media type is accepted, though the file
|
|
must still be UTF-8 text. When a slot declares accepted media types, Notarius
|
|
compares the canonical base media type inferred from the file extension,
|
|
case-insensitively and without parameters.
|
|
|
|
The resolver materializes reference content for chunk, extractor, merger, and
|
|
normalizer targets. Runtime delivery uses `contracts.ChunkRequest.References`,
|
|
`contracts.ExtractionRequest.References`, `contracts.MergeRequest.References`,
|
|
and `contracts.NormalizeRequest.References`. Reference material is not source
|
|
evidence and must not be converted into `SourceRef` values. If a module prompt
|
|
uses references, pass them as prompt input materials through the structured LLM
|
|
request. Prompt metadata hashes remain based on prompt asset source, not
|
|
rendered reference bytes.
|
|
|
|
LLM-backed modules own Scriptorium prompt definitions and response schemas in
|
|
their embedded assets. Module-owned prompts live under each module's shallow
|
|
`assets/prompts` tree and schemas live under `assets/schemas`. Generic shared
|
|
prompt filesystem composition lives under `internal/modules/sharedassets`.
|
|
Common D&D prompt fragments, reference slot helpers, prompt input assembly, and
|
|
reference rendering live under `internal/modules/sharedassets/dnd`. Module
|
|
contracts should expose prompt IDs, versions, input material names, and
|
|
non-secret prompt/schema hashes through manifest metadata; they should not
|
|
expose Scriptorium public types through chunk, extract, merge, or normalize contracts.
|
|
|
|
Chunk modules receive the structured LLM client, configured Scriptorium profile
|
|
ID, prompt session ID, and raw source input material through
|
|
`contracts.ChunkRequest` when they need model-backed chunking. The pipeline
|
|
runner validates generic chunk result invariants before extraction; module-owned
|
|
policies may be stricter but must stay within the module package.
|
|
|
|
Normalize modules receive the structured LLM client, configured Scriptorium
|
|
profile ID, prompt session ID, and reference material through
|
|
`contracts.NormalizeRequest` when they need model-backed reconciliation.
|
|
|
|
Merge modules receive the structured LLM client, configured Scriptorium profile
|
|
ID, prompt session ID, raw source input material, and reference material through
|
|
`contracts.MergeRequest` when they need model-backed merge behavior.
|
|
|
|
## `seriatim` Input
|
|
|
|
Package: `internal/modules/input/seriatim`
|
|
|
|
The `seriatim` adapter parses Seriatim transcript JSON into a generic source
|
|
document. It owns transcript JSON details, source ID selection, source digest
|
|
creation, transcript segment validation, and segment metadata mapping.
|
|
|
|
Provides:
|
|
|
|
- `source.transcript`
|
|
- `transcript.speaker`
|
|
- `transcript.timestamps`
|
|
|
|
External JSON shape belongs in the Seriatim integration doc.
|
|
|
|
## `generic` Chunker
|
|
|
|
Package: `internal/modules/chunk/generic`
|
|
|
|
The `generic` chunker splits source units into ordered chunks. It validates the
|
|
source document, clones source units, assigns chunk IDs such as `chunk-000001`,
|
|
and records chunk metadata for start unit, end unit, and unit count.
|
|
|
|
The pipeline runner canonicalizes chunk units from the source document by
|
|
integer ID before extractors and mergers run. Chunkers also populate chunk
|
|
start and end unit IDs, content bytes, and media type. Chunker-owned context
|
|
should stay in `SourceChunk.Metadata`.
|
|
|
|
Options:
|
|
|
|
- `max_units`: positive integer, default `50`;
|
|
- `overlap_units`: non-negative integer, default `0`, and less than
|
|
`max_units`.
|
|
|
|
Provides:
|
|
|
|
- `chunks`
|
|
|
|
## `dnd/scenes` Chunker
|
|
|
|
Package: `internal/modules/chunk/dnd/scenes`
|
|
|
|
The `dnd/scenes` chunker uses the structured LLM client to divide transcript
|
|
source units into coherent D&D scenes. It supplies the embedded Scriptorium
|
|
prompt ID, prompt version, transcript input material, response schema, and
|
|
session ID to the runtime; validates model-authored source-unit boundaries; and
|
|
converts each scene into a deterministic source chunk.
|
|
|
|
Its prompt definition lives under `assets/prompts` and its schema under
|
|
`assets/schemas`. Shared reusable D&D prompt fragments are provided by
|
|
`internal/modules/sharedassets/dnd` and referenced from prompt definitions under
|
|
`./sharedassets/`.
|
|
|
|
Requires:
|
|
|
|
- `source.transcript`
|
|
|
|
Provides:
|
|
|
|
- `chunks`
|
|
- `chunks.scenes`
|
|
|
|
Options: none. Non-empty options are rejected.
|
|
|
|
The chunker enforces full source-unit coverage from the first source unit to the
|
|
last, sequential contiguous scenes, and no overlap. Its LLM-facing schema uses
|
|
integer `start_unit_id` and `end_unit_id` values matching source-unit IDs. It
|
|
assigns chunk IDs such as `scene-000001`, emits JSON chunk content, and stores
|
|
scene metadata including title, primary mode, participants, summary, boundary
|
|
note, confidence, boundary unit IDs, and unit count. Boundary caveats become
|
|
warnings with reason code
|
|
`scene_boundary_caveat`. Whitespace-only caveats are treated as malformed
|
|
structured output rather than silently dropped.
|
|
|
|
Malformed model output fails explicitly rather than falling back to another
|
|
chunker. The chunker exposes prompt and response-schema provenance through
|
|
top-level `module_metadata.chunker` without raw prompts, raw schemas, source
|
|
text, or secrets.
|
|
|
|
## `dnd/spells` Extractor
|
|
|
|
Package: `internal/modules/extract/dnd/spells`
|
|
|
|
The `dnd/spells` extractor owns D&D spell-cast extraction semantics. It
|
|
supplies the embedded Scriptorium prompt ID, prompt version, chunk-scoped
|
|
transcript input material, reference input materials, response schema, and
|
|
session ID to the runtime; then returns the structured LLM `spell_casts`
|
|
response as raw JSON.
|
|
Its LLM-facing source-reference schema uses integer `start_unit_id` and
|
|
`end_unit_id` values matching source-unit IDs.
|
|
|
|
Its prompt definition lives under `assets/prompts` and its schema under
|
|
`assets/schemas`. Shared reusable D&D prompt fragments are provided by
|
|
`internal/modules/sharedassets/dnd` and referenced from prompt definitions under
|
|
`./sharedassets/`.
|
|
|
|
Requires:
|
|
|
|
- `chunks`
|
|
- `source.transcript`
|
|
|
|
Provides:
|
|
|
|
- `dnd.spell_casts`
|
|
|
|
Response schema identity:
|
|
|
|
- schema ID: `notarius.dnd.spells`
|
|
- schema name: `notarius_dnd_spells_v1`
|
|
- schema version: `v1`
|
|
|
|
The extractor adds prompt and response-schema provenance to lane manifest
|
|
metadata under `artifact_lanes[].metadata.extractor`. Durable raw output
|
|
details belong in the
|
|
[D&D spell raw output contract](../integrations/dnd-spell-artifacts.md).
|
|
|
|
The `dnd/scenes` chunker and `dnd/spells` extractor declare optional `players`,
|
|
`party`, and `glossary` reference slots accepting UTF-8 plain text, Markdown,
|
|
YAML, or JSON. They also accept `roster` as a deprecated compatibility alias for
|
|
`party`. Their prompts frame references as supporting disambiguation material
|
|
only; spell-cast artifacts must still be grounded in the source transcript.
|
|
|
|
## `appendorder` Merger
|
|
|
|
Package: `internal/modules/merge/appendorder`
|
|
|
|
The `appendorder` merger preserves chunk order for raw extract outputs. A
|
|
single JSON extract output is passed through as the merge output. Multiple JSON
|
|
object outputs with one common top-level array field are merged by concatenating
|
|
that array field in chunk order. Other valid JSON shapes are merged as a JSON
|
|
array of decoded values in chunk order. Non-JSON media types and invalid JSON
|
|
are rejected.
|
|
|
|
Provides:
|
|
|
|
- `merged`
|
|
|
|
## `noop` Normalizer
|
|
|
|
Package: `internal/modules/normalize/noop`
|
|
|
|
The `noop` normalizer clones the raw merge output and returns it unchanged.
|
|
|
|
Requires:
|
|
|
|
- `merged`
|
|
|
|
Provides:
|
|
|
|
- `normalized`
|
|
|
|
## `json` Output
|
|
|
|
Package: `internal/modules/output/json`
|
|
|
|
The `json` output encoder converts normalized raw outputs, rejected raw outputs,
|
|
warnings, and the run manifest into logical JSON output files. It writes one
|
|
payload file per lane under `lanes/` and sanitizes lane IDs for file names.
|
|
Normalized output payloads must be valid `application/json`.
|
|
|
|
Requires:
|
|
|
|
- `normalized`
|
|
|
|
Provides:
|
|
|
|
- `encoded`
|
|
|
|
Durable output file shapes belong in the
|
|
[JSON output contract](../integrations/json-output.md). Operator behavior
|
|
belongs in [Operations](../operations.md).
|
|
|
|
## Production Registration
|
|
|
|
Production registration is centralized in `internal/cli/catalog.go`.
|
|
|
|
Do not make framework code import production modules. The CLI wires production
|
|
modules at the application boundary; tests may provide fake registries or fake
|
|
catalogs directly.
|
|
|
|
## Adding A Module
|
|
|
|
When adding a module, keep source-format and extraction-domain boundaries clear:
|
|
|
|
- input modules may know external source formats;
|
|
- extract modules may know artifact semantics and prompt/schema assets;
|
|
- merge and normalize modules own raw output combination and reconciliation;
|
|
- output modules own serialization, not diagnostics or CLI reporting.
|
|
|
|
Update [Development](../policy/development.md), [Configuration](../config.md),
|
|
internal docs, integration docs, and examples when the new module becomes
|
|
implemented production behavior.
|