195 lines
8.7 KiB
Markdown
195 lines
8.7 KiB
Markdown
# Module And Validator Internals
|
|
|
|
Production stage implementations live under `internal/modules`; production
|
|
validators live under `internal/validators`. The selectable keys, configuration
|
|
options, reference slots, and default validator chain are canonical in the
|
|
[module](../config.md#implemented-production-modules) and
|
|
[validator](../config.md#implemented-production-validators) catalogs in
|
|
Configuration.
|
|
|
|
## Extension Pattern
|
|
|
|
A stage module package provides a stable key, constructor, contract
|
|
implementation, `ModuleSpec`, `Register`, and focused behavior and registration
|
|
tests. A validator package follows the same pattern with `ValidatorSpec` and the
|
|
validator registry.
|
|
|
|
Specs expose capability and execution metadata without constructing an
|
|
implementation. Chunk, extract, merge, and normalize modules that accept
|
|
auxiliary material declare identical reference slots from both
|
|
`ReferenceSlots()` and `ModuleSpec().ReferenceSlots`; registration tests enforce
|
|
that agreement. Runtime delivery uses the corresponding stage request's
|
|
`References` field.
|
|
|
|
LLM-backed extensions own their prompt definitions and response schemas under
|
|
package-local embedded assets. Shared filesystem composition belongs in
|
|
`internal/modules/sharedassets`; reusable D&D prompt fragments, reference
|
|
declarations, prompt-input assembly, and source-unit helpers belong in
|
|
`internal/modules/sharedassets/dnd`. Stage contracts expose only Notarius
|
|
structured-completion types, not Scriptorium public types.
|
|
|
|
Reference material may inform a module or prompt but must not become source
|
|
evidence. The resolver and materializer behavior is described in
|
|
[Pipeline Internals](pipeline.md#reference-materialization).
|
|
|
|
## Input Adapter
|
|
|
|
### `internal/modules/input/seriatim`
|
|
|
|
The adapter decodes the supported transcript JSON, selects the source identity,
|
|
computes the raw-input digest, validates segments, and maps each segment into a
|
|
generic source unit with speaker and timestamp metadata. Its spec advertises the
|
|
transcript capabilities consumed by D&D modules.
|
|
|
|
Parsing is strict about required values and duplicate unit IDs but deliberately
|
|
ignores unrelated Seriatim fields. The external format and derived-identity
|
|
rules are defined in the
|
|
[Seriatim contract](../integrations/seriatim.md).
|
|
|
|
## Chunkers
|
|
|
|
### `internal/modules/chunk/generic`
|
|
|
|
The generic chunker validates the source document, walks units in configured
|
|
windows, clones each selected unit, and emits deterministic ordered chunk IDs.
|
|
Overlap changes the next window start but never reorders units. It records the
|
|
first and last unit and unit count in chunk metadata.
|
|
|
|
The accepted options and defaults are defined in
|
|
[Configuration](../config.md#implemented-production-modules). Generic
|
|
framework validation canonicalizes the returned unit slices before extraction.
|
|
|
|
### `internal/modules/chunk/dnd/scenes`
|
|
|
|
The scene chunker prepares a structured Scriptorium request from the full
|
|
transcript, session, and optional D&D reference inputs. It validates the model's
|
|
scene boundaries against source-unit IDs and converts them into deterministic
|
|
chunks.
|
|
|
|
Scene validation requires sequential, contiguous, non-overlapping coverage from
|
|
the first source unit through the last. Each chunk contains JSON scene content
|
|
and module-owned metadata for the scene description, boundaries, confidence,
|
|
participants, and unit count. Boundary caveats become warnings. Malformed
|
|
structured output is returned as an error; there is no fallback chunker.
|
|
|
|
The package embeds its prompt and response schema and reports their non-secret
|
|
identity and hashes through singleton module metadata. Shared D&D assets supply
|
|
reference declarations and prompt inputs; their user-facing keys and accepted
|
|
file types remain canonical in [Configuration](../config.md).
|
|
|
|
## Extractor
|
|
|
|
### `internal/modules/extract/dnd/spells`
|
|
|
|
The spell extractor prepares a structured request from one chunk, the
|
|
chunk-scoped source input, the session, and optional D&D reference inputs. It
|
|
decodes the model response, assigns the generic source identity to every source
|
|
reference, canonicalizes duplicate references, orders spell casts by their
|
|
earliest cited unit, and returns raw JSON plus response-schema provenance.
|
|
|
|
The package owns its embedded prompt, response schemas, and prompt/schema
|
|
manifest metadata. Shared D&D helpers keep prompt input names and source-unit
|
|
reference conversion consistent with the scene chunker. The extractor produces
|
|
raw output; production validators own approval policy.
|
|
|
|
The durable payload and manifest metadata shapes are defined in the
|
|
[D&D spell artifact contract](../integrations/dnd-spell-artifacts.md).
|
|
|
|
## Merger And Normalizer
|
|
|
|
### `internal/modules/merge/appendorder`
|
|
|
|
The merger preserves extract-result order. It passes through one JSON result,
|
|
concatenates a common top-level array field across multiple JSON objects, and
|
|
otherwise emits an array of the decoded values. It rejects invalid JSON and
|
|
non-JSON media types, and it preserves compatible schema provenance.
|
|
|
|
### `internal/modules/normalize/noop`
|
|
|
|
The normalizer defensively clones the accepted merge result, including payload
|
|
bytes, metadata, warnings, and schema provenance, without changing its logical
|
|
content.
|
|
|
|
## Output Encoder
|
|
|
|
### `internal/modules/output/json`
|
|
|
|
The JSON encoder sorts normalized results by lane, derives collision-checked
|
|
safe logical names, pretty-prints JSON payloads, and assembles the logical index,
|
|
manifest, rejected-result, warning, and lane files. Invalid JSON, unsupported
|
|
media types, unsafe names, and sanitized-name collisions are errors.
|
|
|
|
The encoder returns logical files only. The CLI places them on disk, and the
|
|
[JSON output contract](../integrations/json-output.md) defines their external
|
|
paths and schemas.
|
|
|
|
## Generic Validators
|
|
|
|
The unconditional accept and reject validators provide deterministic production
|
|
registrations used primarily for controlled composition and tests.
|
|
|
|
The JSON syntax validator uses `encoding/json` to reject malformed payloads. The
|
|
JSON Schema validator requires schema bytes on the validation request, parses
|
|
the instance and schema with `jsonschema`, and distinguishes payload rejection
|
|
from schema loading or compilation errors. Neither validator calls the LLM.
|
|
|
|
## D&D Spell Validators
|
|
|
|
`internal/validators/extract/dnd/spells/spellpayload` provides strict decoding,
|
|
shape checks, source-reference candidates, and cited-text lookup shared by the
|
|
three validators.
|
|
|
|
The shape validator rejects malformed JSON, unknown fields, missing or empty
|
|
spell fields, and empty reference lists. The source-reference validator applies
|
|
generic source-reference validation to every cited range. The relatedness
|
|
validator approves structurally valid payloads but warns when a case-insensitive
|
|
spell name is absent from all cited source text. It leaves malformed payloads to
|
|
the earlier validators in the configured chain.
|
|
|
|
These validators are deterministic. Their selectable keys and production order
|
|
are defined in
|
|
[Configuration](../config.md#implemented-production-validators); their durable
|
|
payload rules are defined in the
|
|
[artifact contract](../integrations/dnd-spell-artifacts.md).
|
|
|
|
## Production Registration
|
|
|
|
`internal/cli/catalog.go` builds the production registries, registers module and
|
|
validator constructors, installs default validator-chain mappings, and exposes
|
|
the matching catalog for resolution. It also collects prompt assets from
|
|
LLM-backed packages before constructing the production client.
|
|
|
|
Framework packages must not import production extensions. Tests may compose
|
|
registries and catalogs directly with fakes.
|
|
|
|
## Adding An Extension
|
|
|
|
When adding a production module or validator:
|
|
|
|
1. implement the stage or validator contract and package-local key;
|
|
2. expose and test its spec, constructor, and registration function;
|
|
3. keep format or domain parsing inside the concrete package;
|
|
4. add package-owned prompt/schema assets when the extension is LLM-backed;
|
|
5. register it in `internal/cli/catalog.go` and add a default chain only when
|
|
production policy requires one;
|
|
6. add resolution and composition coverage for capabilities, options,
|
|
references, and validation behavior;
|
|
7. update the selectable-key catalog in [Configuration](../config.md), the
|
|
relevant external contract, this inventory, and maintained examples when
|
|
user-visible behavior changes.
|
|
|
|
Do not add the extension to `docs/development.md`; that file routes by task and
|
|
does not inventory implementations.
|
|
|
|
## Tests To Inspect
|
|
|
|
- Package-local `*_test.go` files under the module or validator being changed.
|
|
- `internal/framework/pipeline/registry_integration_test.go`: registry and spec
|
|
composition.
|
|
- `internal/framework/pipeline/default_modules_test.go`: framework binding
|
|
defaults.
|
|
- `internal/cli/run_test.go`: production catalog, config resolution, and
|
|
end-to-end CLI composition.
|
|
- `internal/modules/sharedassets/**/*_test.go`: shared prompt and reference
|
|
assembly.
|