217 lines
10 KiB
Markdown
217 lines
10 KiB
Markdown
# Module And Validator Internals
|
|
|
|
Production module and validator implementations live under their domain-first
|
|
trees in `internal/modules`.
|
|
The selectable keys, configuration options, reference slots, and default
|
|
validator chain are canonical in the
|
|
[module](../config.md#implemented-production-modules) and
|
|
[validator](../config.md#implemented-production-validators) catalogs in
|
|
Configuration.
|
|
|
|
## Extension Pattern
|
|
|
|
A stage module package provides a stable key, constructor, contract
|
|
implementation, `ModuleSpec`, `Register`, and focused behavior and registration
|
|
tests. A validator package follows the same pattern with `ValidatorSpec` and the
|
|
validator registry. Package-family registrars compose those leaf registrations
|
|
into the production catalog and own family-level policy such as default
|
|
validator chains and prompt asset collection.
|
|
|
|
Production extract, merge, normalize, and validator packages currently use the
|
|
explicit legacy raw registration APIs. Typed registration is framework-ready,
|
|
but no production artifact kind or codec is registered yet. All selected
|
|
production implementations are constructed during pipeline preparation through
|
|
temporary adapters around their existing zero-argument constructors. Their raw
|
|
option maps and LLM clients remain operation inputs until each implementation
|
|
migrates to its construction-owned decoder and injected dependencies.
|
|
|
|
Specs expose capability and execution metadata without constructing an
|
|
implementation. Registry entries separately expose option validation and
|
|
run-local construction. Chunk, extract, merge, and normalize modules that accept
|
|
auxiliary material declare identical reference slots from both
|
|
`ReferenceSlots()` and `ModuleSpec().ReferenceSlots`; registration tests enforce
|
|
that agreement. Runtime delivery uses the corresponding stage request's
|
|
`References` field.
|
|
|
|
LLM-backed extensions own their prompt definitions and response schemas under
|
|
package-local embedded assets. Shared filesystem composition belongs in
|
|
`internal/framework/promptfs`; reusable D&D prompt fragments, reference
|
|
declarations, prompt-input assembly, and source-unit helpers belong in
|
|
`internal/modules/dnd/shared`. Stage contracts expose only Notarius structured-
|
|
completion types, not Scriptorium public types.
|
|
|
|
Reference material may inform a module or prompt but must not become source
|
|
evidence. The resolver and materializer behavior is described in
|
|
[Pipeline Internals](pipeline.md#reference-materialization).
|
|
|
|
## Input Adapter
|
|
|
|
### `internal/modules/seriatim/input/transcript`
|
|
|
|
The adapter decodes the supported transcript JSON, selects the source identity,
|
|
computes canonical source provenance, validates segments, and maps each segment
|
|
into a generic source unit with a self-reference plus speaker and timestamp
|
|
metadata. Its spec advertises the transcript capabilities consumed by D&D
|
|
modules.
|
|
|
|
Parsing is strict about required values and duplicate unit IDs but deliberately
|
|
ignores unrelated Seriatim fields. The external format and derived-identity
|
|
rules are defined in the
|
|
[Seriatim contract](../integrations/seriatim.md).
|
|
|
|
## Chunkers
|
|
|
|
### `internal/modules/generic/chunk/units`
|
|
|
|
The generic chunker validates the source document, walks units in configured
|
|
windows, clones each selected unit, and emits deterministic ordered chunk IDs.
|
|
Overlap changes the next window start but never reorders units. It records the
|
|
first and last unit and unit count in chunk metadata, and derives the chunk's
|
|
canonical source reference from those unit references.
|
|
|
|
The accepted options and defaults are defined in
|
|
[Configuration](../config.md#implemented-production-modules). Generic
|
|
framework validation canonicalizes the returned unit slices before extraction.
|
|
|
|
### `internal/modules/dnd/chunk/scenes`
|
|
|
|
The scene chunker prepares a structured Scriptorium request from the full
|
|
transcript, session, and optional D&D reference inputs. It validates the model's
|
|
scene boundaries against source-unit IDs and converts them into deterministic
|
|
chunks with canonical source references spanning each scene's units.
|
|
|
|
Scene validation requires sequential, contiguous, non-overlapping coverage from
|
|
the first source unit through the last. Each chunk contains JSON scene content
|
|
and module-owned metadata for the scene description, boundaries, confidence,
|
|
participants, and unit count. Boundary caveats become warnings. Malformed
|
|
structured output is returned as an error; there is no fallback chunker.
|
|
|
|
The package embeds its prompt and response schema and reports their non-secret
|
|
identity and hashes through singleton module metadata. Shared D&D assets supply
|
|
reference declarations and prompt inputs; their user-facing keys and accepted
|
|
file types remain canonical in [Configuration](../config.md).
|
|
|
|
## Extractor
|
|
|
|
### `internal/modules/dnd/extract/spells`
|
|
|
|
The spell extractor prepares a structured request from one chunk, the
|
|
chunk-scoped source input, the session, and optional D&D reference inputs. It
|
|
decodes the model response, assigns the generic source identity to every source
|
|
reference, canonicalizes duplicate references, orders spell casts by their
|
|
earliest cited unit, and returns raw JSON plus response-schema provenance.
|
|
|
|
The package owns its embedded prompt, response schemas, and prompt/schema
|
|
manifest metadata. Shared D&D helpers keep prompt input names and source-unit
|
|
reference conversion consistent with the scene chunker. The extractor produces
|
|
raw output; production validators own approval policy.
|
|
|
|
The durable payload and manifest metadata shapes are defined in the
|
|
[D&D spell artifact contract](../integrations/dnd-spell-artifacts.md).
|
|
|
|
## Merger And Normalizer
|
|
|
|
### `internal/modules/generic/merge/appendorder`
|
|
|
|
The merger preserves extract-result order. It passes through one JSON result,
|
|
concatenates a common top-level array field across multiple JSON objects, and
|
|
otherwise emits an array of the decoded values. It rejects invalid JSON and
|
|
non-JSON media types, and it preserves compatible schema provenance.
|
|
|
|
### `internal/modules/generic/normalize/noop`
|
|
|
|
The normalizer defensively clones the accepted merge result, including payload
|
|
bytes, metadata, warnings, and schema provenance, without changing its logical
|
|
content.
|
|
|
|
## Output Encoder
|
|
|
|
### `internal/modules/generic/output/json`
|
|
|
|
The JSON encoder sorts normalized results by lane, derives collision-checked
|
|
safe logical names, pretty-prints JSON payloads, and assembles the logical index,
|
|
manifest, rejected-result, warning, and lane files. Invalid JSON, unsupported
|
|
media types, unsafe names, and sanitized-name collisions are errors.
|
|
|
|
The encoder returns logical files only. The CLI places them on disk, and the
|
|
[JSON output contract](../integrations/json-output.md) defines their external
|
|
paths and schemas.
|
|
|
|
## Generic Validators
|
|
|
|
The generic validator implementations live under
|
|
`internal/modules/generic/validate`.
|
|
|
|
The unconditional accept and reject validators provide deterministic production
|
|
registrations used primarily for controlled composition and tests.
|
|
|
|
The JSON syntax validator uses `encoding/json` to reject malformed payloads. The
|
|
JSON Schema validator requires schema bytes on the validation request, parses
|
|
the instance and schema with `jsonschema`, and distinguishes payload rejection
|
|
from schema loading or compilation errors. Neither validator calls the LLM.
|
|
|
|
## D&D Spell Validators
|
|
|
|
`internal/modules/dnd/validate/spells/spellpayload` provides strict decoding,
|
|
shape checks, source-reference candidates, and cited-text lookup shared by the
|
|
three validators.
|
|
|
|
The shape validator rejects malformed JSON, unknown fields, missing or empty
|
|
spell fields, and empty reference lists. The source-reference validator applies
|
|
generic source-reference validation to every cited range. The relatedness
|
|
validator approves structurally valid payloads but warns when a case-insensitive
|
|
spell name is absent from all cited source text. It leaves malformed payloads to
|
|
the earlier validators in the configured chain.
|
|
|
|
These validators are deterministic. Their selectable keys and production order
|
|
are defined in
|
|
[Configuration](../config.md#implemented-production-validators); their durable
|
|
payload rules are defined in the
|
|
[artifact contract](../integrations/dnd-spell-artifacts.md).
|
|
|
|
## Production Registration
|
|
|
|
The CLI allocates one complete framework registry set and one LLM asset
|
|
registry. It invokes `internal/modules/generic/register`,
|
|
`internal/modules/seriatim/register`, and `internal/modules/dnd/register` in
|
|
that order, then exposes the matching catalog for resolution. The generic and
|
|
Seriatim registrars own their production leaf registrations. The D&D registrar
|
|
owns D&D leaf registrations, the spell default-validator chain, and D&D
|
|
prompt/schema asset collection.
|
|
|
|
Framework packages must not import production extensions. Tests may compose
|
|
registries and catalogs directly with fakes.
|
|
|
|
## Adding An Extension
|
|
|
|
When adding a production module or validator:
|
|
|
|
1. implement the stage or validator contract and package-local key;
|
|
2. expose and test its spec, constructor, and registration function;
|
|
3. keep format or domain parsing inside the concrete package;
|
|
4. add package-owned prompt/schema assets when the extension is LLM-backed;
|
|
5. register it through its package-family registrar and add a default chain
|
|
there only when production policy requires one;
|
|
6. add resolution and composition coverage for capabilities, options,
|
|
references, and validation behavior;
|
|
7. update the selectable-key catalog in [Configuration](../config.md), the
|
|
relevant external contract, this inventory, and maintained examples when
|
|
user-visible behavior changes.
|
|
|
|
Do not add the extension to `docs/development.md`; that file routes by task and
|
|
does not inventory implementations.
|
|
|
|
## Tests To Inspect
|
|
|
|
- Package-local `*_test.go` files under the module or validator being changed.
|
|
- `internal/framework/pipeline/registry_integration_test.go`: registry and spec
|
|
composition.
|
|
- `internal/framework/pipeline/default_modules_test.go`: framework binding
|
|
defaults.
|
|
- `internal/cli/run_test.go`: production catalog, config resolution, and
|
|
end-to-end CLI composition.
|
|
- `internal/framework/promptfs/*_test.go` and
|
|
`internal/modules/dnd/shared/*_test.go`: shared prompt and reference assembly.
|
|
- `internal/modules/integration/*_test.go`: black-box composition across
|
|
production extension domains.
|