263 lines
9.2 KiB
Markdown
263 lines
9.2 KiB
Markdown
# Modules
|
|
|
|
Production modules live under `internal/modules`. Each module implements one
|
|
contract from `internal/framework/contracts`, exposes a `ModuleSpec`, and
|
|
registers itself with the matching pipeline registry.
|
|
|
|
The CLI production catalog currently registers only the modules listed here.
|
|
|
|
## Contract Pattern
|
|
|
|
A production module package should provide:
|
|
|
|
- a stable module key;
|
|
- a constructor such as `New`;
|
|
- the relevant contract implementation;
|
|
- `ModuleSpec`;
|
|
- `Register`;
|
|
- focused tests for registration, options, contract behavior, and errors.
|
|
|
|
Module specs should describe capabilities accurately. Resolution uses specs to
|
|
reject incompatible pipelines before execution.
|
|
|
|
Chunk, extract, and normalize modules that accept auxiliary reference material
|
|
must declare slots through both `ReferenceSlots()` and
|
|
`ModuleSpec().ReferenceSlots`. The runtime slot list and registry metadata
|
|
should match so config validation can inspect slots without constructing module
|
|
instances. A slot declaration names the slot, whether it is required, accepted
|
|
media types, whether multiple items are allowed, and any byte limit. Empty
|
|
`AcceptedMediaTypes` means any inferred media type is accepted, though the file
|
|
must still be UTF-8 text. When a slot declares accepted media types, Notarius
|
|
compares the canonical base media type inferred from the file extension,
|
|
case-insensitively and without parameters.
|
|
|
|
The resolver materializes reference content for chunk, extractor, and
|
|
normalizer targets. Runtime delivery uses `contracts.ChunkRequest.References`,
|
|
`contracts.ExtractionRequest.References`, and
|
|
`contracts.NormalizeRequest.References`. Reference material is not source
|
|
evidence and must not be converted into `SourceRef` values. If a module prompt
|
|
uses references, pass them as prompt input materials through the structured LLM
|
|
request. Prompt metadata hashes remain based on prompt asset source, not
|
|
rendered reference bytes.
|
|
|
|
LLM-backed modules own Scriptorium prompt definitions and response schemas in
|
|
their embedded assets. Module-owned prompts live under each module's shallow
|
|
`assets/prompts` tree and schemas live under `assets/schemas`. Shared reusable
|
|
prompt fragments live under `internal/modules/sharedassets`. Module contracts
|
|
should expose prompt IDs, versions, input material names, and non-secret
|
|
prompt/schema hashes through manifest metadata; they should not expose
|
|
Scriptorium public types through chunk, extract, or normalize contracts.
|
|
|
|
Chunk modules receive the structured LLM client, configured Scriptorium profile
|
|
ID, prompt session ID, and raw source input material through
|
|
`contracts.ChunkRequest` when they need model-backed chunking. The pipeline
|
|
runner validates generic chunk result invariants before extraction; module-owned
|
|
policies may be stricter but must stay within the module package.
|
|
|
|
Normalize modules receive the structured LLM client, configured Scriptorium
|
|
profile ID, prompt session ID, and reference material through
|
|
`contracts.NormalizeRequest` when they need model-backed reconciliation.
|
|
|
|
## `seriatim` Input
|
|
|
|
Package: `internal/modules/input/seriatim`
|
|
|
|
The `seriatim` adapter parses Seriatim transcript JSON into a generic source
|
|
document. It owns transcript JSON details, source ID selection, source digest
|
|
creation, transcript segment validation, and segment metadata mapping.
|
|
|
|
Provides:
|
|
|
|
- `source.transcript`
|
|
- `transcript.speaker`
|
|
- `transcript.timestamps`
|
|
|
|
External JSON shape belongs in the Seriatim integration doc.
|
|
|
|
## `generic` Chunker
|
|
|
|
Package: `internal/modules/chunk/generic`
|
|
|
|
The `generic` chunker splits source units into ordered chunks. It validates the
|
|
source document, clones source units, assigns chunk IDs such as `chunk-000001`,
|
|
and records chunk metadata for start unit, end unit, and unit count.
|
|
|
|
The pipeline runner canonicalizes chunk units from the source document by ID
|
|
before extractors and mergers run. Chunker-owned context should stay in
|
|
`SourceChunk.Metadata`.
|
|
|
|
Options:
|
|
|
|
- `max_units`: positive integer, default `50`;
|
|
- `overlap_units`: non-negative integer, default `0`, and less than
|
|
`max_units`.
|
|
|
|
Provides:
|
|
|
|
- `chunks`
|
|
|
|
## `dnd/scenes` Chunker
|
|
|
|
Package: `internal/modules/chunk/dnd/scenes`
|
|
|
|
The `dnd/scenes` chunker uses the structured LLM client to divide transcript
|
|
source units into coherent D&D scenes. It supplies the embedded Scriptorium
|
|
prompt ID, prompt version, transcript input material, response schema, and
|
|
session ID to the runtime; validates model-authored source-unit boundaries; and
|
|
converts each scene into a deterministic source chunk.
|
|
|
|
Its prompt definition lives under `assets/prompts` and its schema under
|
|
`assets/schemas`. Shared reusable prompt fragments are provided by
|
|
`internal/modules/sharedassets` and referenced from prompt definitions under
|
|
`./sharedassets/`.
|
|
|
|
Requires:
|
|
|
|
- `source.transcript`
|
|
|
|
Provides:
|
|
|
|
- `chunks`
|
|
- `chunks.scenes`
|
|
|
|
Options: none. Non-empty options are rejected.
|
|
|
|
The chunker enforces full source-unit coverage from the first source unit to the
|
|
last, exact source-unit IDs, sequential contiguous scenes, and no overlap. It
|
|
assigns chunk IDs such as `scene-000001` and stores scene metadata including
|
|
title, primary mode, participants, summary, boundary note, confidence, boundary
|
|
unit IDs, and unit count. Boundary caveats become warnings with reason code
|
|
`scene_boundary_caveat`. Whitespace-only caveats are treated as malformed
|
|
structured output rather than silently dropped.
|
|
|
|
Malformed model output fails explicitly rather than falling back to another
|
|
chunker. The chunker exposes prompt and response-schema provenance through
|
|
top-level `module_metadata.chunker` without raw prompts, raw schemas, source
|
|
text, or secrets.
|
|
|
|
## `dnd/spells` Extractor
|
|
|
|
Package: `internal/modules/extract/dnd/spells`
|
|
|
|
The `dnd/spells` extractor owns D&D spell-cast artifact semantics. It supplies
|
|
the embedded Scriptorium prompt ID, prompt version, transcript and reference
|
|
input materials, response schema, and session ID to the runtime; converts
|
|
spell-cast responses into artifact candidates; and supplies deterministic
|
|
validators.
|
|
|
|
Its prompt definition lives under `assets/prompts` and its schema under
|
|
`assets/schemas`. Shared reusable prompt fragments are provided by
|
|
`internal/modules/sharedassets` and referenced from prompt definitions under
|
|
`./sharedassets/`.
|
|
|
|
Requires:
|
|
|
|
- `chunks`
|
|
- `source.transcript`
|
|
|
|
Provides:
|
|
|
|
- `dnd.spell_casts`
|
|
|
|
Artifact type and schema version:
|
|
|
|
- artifact type: `dnd.spell_cast`
|
|
- schema version: `v1`
|
|
|
|
The extractor adds prompt and response-schema provenance to lane manifest
|
|
metadata under `artifact_lanes[].metadata.extractor`. Durable artifact payload
|
|
details belong in the
|
|
[D&D spell artifact contract](../integrations/dnd-spell-artifacts.md).
|
|
|
|
The `dnd/scenes` chunker and `dnd/spells` extractor declare optional `players`,
|
|
`party`, and `glossary` reference slots accepting UTF-8 plain text, Markdown,
|
|
YAML, or JSON. They also accept `roster` as a deprecated compatibility alias for
|
|
`party`. Their prompts frame references as supporting disambiguation material
|
|
only; spell-cast artifacts must still be grounded in the source transcript.
|
|
|
|
## D&D Spell Validators
|
|
|
|
The spell extractor returns two built-in validators:
|
|
|
|
- `dnd/spells/shape`: rejects malformed payloads and missing required fields.
|
|
- `dnd/spells/source_refs`: rejects candidates without valid source references.
|
|
It also emits a warning when the extracted spell name is not found in the
|
|
cited source text.
|
|
|
|
Reason codes include:
|
|
|
|
- `invalid_payload`
|
|
- `missing_required_field`
|
|
- `missing_source_ref`
|
|
- `invalid_source_ref`
|
|
- `spell_not_near_source`
|
|
|
|
These validators are supplied by the extractor when no validators are configured
|
|
for the lane.
|
|
|
|
## `appendorder` Merger
|
|
|
|
Package: `internal/modules/merge/appendorder`
|
|
|
|
The `appendorder` merger clones and appends candidates in chunk order. It does
|
|
not deduplicate or reconcile candidates.
|
|
|
|
Provides:
|
|
|
|
- `merged`
|
|
|
|
## `noop` Normalizer
|
|
|
|
Package: `internal/modules/normalize/noop`
|
|
|
|
The `noop` normalizer clones merged candidates and returns them unchanged.
|
|
|
|
Requires:
|
|
|
|
- `merged`
|
|
|
|
Provides:
|
|
|
|
- `normalized`
|
|
|
|
## `json` Output
|
|
|
|
Package: `internal/modules/output/json`
|
|
|
|
The `json` output encoder converts approved artifacts, rejected artifacts,
|
|
warnings, and the run manifest into logical JSON output files. It groups
|
|
approved artifacts by artifact type and sanitizes artifact-type file names.
|
|
|
|
Requires:
|
|
|
|
- `normalized`
|
|
|
|
Provides:
|
|
|
|
- `encoded`
|
|
|
|
Durable output file shapes belong in the
|
|
[JSON output contract](../integrations/json-output.md). Operator behavior
|
|
belongs in [Operations](../operations.md).
|
|
|
|
## Production Registration
|
|
|
|
Production registration is centralized in `internal/cli/catalog.go`.
|
|
|
|
Do not make framework code import production modules. The CLI wires production
|
|
modules at the application boundary; tests may provide fake registries or fake
|
|
catalogs directly.
|
|
|
|
## Adding A Module
|
|
|
|
When adding a module, keep source-format and extraction-domain boundaries clear:
|
|
|
|
- input modules may know external source formats;
|
|
- extract modules may know artifact semantics and prompt/schema assets;
|
|
- merge and normalize modules own candidate combination and reconciliation;
|
|
- output modules own serialization, not diagnostics or CLI reporting.
|
|
|
|
Update [Development](../policy/development.md), [Configuration](../config.md),
|
|
internal docs, integration docs, and examples when the new module becomes
|
|
implemented production behavior.
|