241 lines
12 KiB
Markdown
241 lines
12 KiB
Markdown
# Module And Validator Internals
|
|
|
|
Production module and validator implementations live under their domain-first
|
|
trees in `internal/modules`.
|
|
The selectable keys, configuration options, reference slots, and default
|
|
validator chain are canonical in the
|
|
[module](../config.md#implemented-production-modules) and
|
|
[validator](../config.md#implemented-production-validators) catalogs in
|
|
Configuration.
|
|
|
|
## Extension Pattern
|
|
|
|
A stage module package provides a stable key, constructor, contract
|
|
implementation, `ModuleSpec`, `Register`, and focused behavior and registration
|
|
tests. A validator package follows the same pattern with `ValidatorSpec` and the
|
|
validator registry. Package-family registrars compose those leaf registrations
|
|
into the production catalog and own family-level policy such as default
|
|
validator chains and prompt asset collection.
|
|
|
|
Production input, chunk, output, and D&D spell-extract packages register strict
|
|
option decoders and run-local builders. Preparation decodes their options into
|
|
implementation-owned values and injects dependencies. The spell extractor is
|
|
typed over the canonical D&D model. D&D validators, merge, and normalize use
|
|
typed variants; JSON representation validators use serialized requests; and
|
|
unconditional validators expose separate chunk and typed variants. The D&D
|
|
production registrar registers only the canonical typed spell implementations.
|
|
|
|
Prepared extractors, extract validators, and codecs may be reused concurrently
|
|
by the run-wide extract pool. Production implementations are immutable after
|
|
construction: they retain only typed options, immutable assets, or the shared
|
|
concurrency-safe LLM client. Implementations that introduce mutable state must
|
|
synchronize that state without creating a separate provider scheduler.
|
|
|
|
Specs expose capability and execution metadata without constructing an
|
|
implementation. Registry entries separately expose option validation and
|
|
run-local construction. Chunk, extract, merge, and normalize modules that accept
|
|
auxiliary material declare identical reference slots from both
|
|
`ReferenceSlots()` and `ModuleSpec().ReferenceSlots`; registration tests enforce
|
|
that agreement. Runtime delivery uses the corresponding stage request's
|
|
`References` field.
|
|
|
|
LLM-backed extensions own their prompt definitions and response schemas under
|
|
package-local embedded assets. Shared filesystem composition belongs in
|
|
`internal/framework/promptfs`; reusable D&D prompt fragments, reference
|
|
declarations, prompt-input assembly, and source-unit helpers belong in
|
|
`internal/modules/dnd/shared`. Stage contracts expose only Notarius structured-
|
|
completion types, not Scriptorium public types.
|
|
|
|
Reference material may inform a module or prompt but must not become source
|
|
evidence. The resolver and materializer behavior is described in
|
|
[Pipeline Internals](pipeline.md#reference-materialization).
|
|
|
|
## Input Adapter
|
|
|
|
### `internal/modules/seriatim/input/transcript`
|
|
|
|
The adapter decodes the supported transcript JSON, selects the source identity,
|
|
computes canonical source provenance, validates segments, and maps each segment
|
|
into a generic source unit with a self-reference plus speaker and timestamp
|
|
metadata. It accepts no module options. Its spec advertises the transcript
|
|
capabilities consumed by D&D modules.
|
|
|
|
Parsing is strict about required values and duplicate unit IDs but deliberately
|
|
ignores unrelated Seriatim fields. The external format and derived-identity
|
|
rules are defined in the
|
|
[Seriatim contract](../integrations/seriatim.md).
|
|
|
|
## Chunkers
|
|
|
|
### `internal/modules/generic/chunk/units`
|
|
|
|
The generic chunker validates the source document, walks units in configured
|
|
windows, clones each selected unit, and emits deterministic ordered chunk IDs.
|
|
Overlap changes the next window start but never reorders units. It records the
|
|
first and last unit and unit count in chunk metadata, and derives the chunk's
|
|
canonical source reference from those unit references.
|
|
|
|
The accepted options and defaults are defined in
|
|
[Configuration](../config.md#implemented-production-modules). Generic
|
|
framework validation canonicalizes the returned unit slices before extraction.
|
|
The chunker decodes its options during construction and retains only the typed
|
|
window settings used by `Chunk`.
|
|
|
|
### `internal/modules/dnd/chunk/scenes`
|
|
|
|
The scene chunker prepares a structured Scriptorium request from the full
|
|
transcript, session, and optional D&D reference inputs. It validates the model's
|
|
scene boundaries against source-unit IDs and converts them into deterministic
|
|
chunks with canonical source references spanning each scene's units.
|
|
Preparation injects the shared structured LLM client into the chunker; `Chunk`
|
|
supplies only the run-specific profile, session, source, references, and
|
|
metadata.
|
|
|
|
Scene validation requires sequential, contiguous, non-overlapping coverage from
|
|
the first source unit through the last. Each chunk contains JSON scene content
|
|
and module-owned metadata for the scene description, boundaries, confidence,
|
|
participants, and unit count. Boundary caveats become warnings. Malformed
|
|
structured output is returned as an error; there is no fallback chunker.
|
|
|
|
The package embeds its prompt and response schema and reports their non-secret
|
|
identity and hashes through singleton module metadata. Shared D&D assets supply
|
|
reference declarations and prompt inputs; their user-facing keys and accepted
|
|
file types remain canonical in [Configuration](../config.md).
|
|
|
|
## Extractor
|
|
|
|
### `internal/modules/dnd/extract/spells`
|
|
|
|
The spell extractor prepares a structured request from one chunk, the
|
|
chunk-scoped source input, the session, and optional D&D reference inputs. It
|
|
decodes the model response, assigns the generic source identity to every source
|
|
reference, canonicalizes duplicate references, orders spell casts by their
|
|
earliest cited unit, and returns `dnd.SpellList`.
|
|
|
|
The extractor owns its private model-response DTO, embedded prompt, LLM response
|
|
schema, strict option decoder, injected shared LLM client, and prompt/schema
|
|
manifest metadata. The separate `internal/modules/dnd/codec/spells` package
|
|
owns the durable schema and stable JSON representation for artifact kind
|
|
`dnd/spell-list`. The runner keeps the result typed through validators and later
|
|
stages, using the codec only for checkpoint, debug, and output boundaries.
|
|
Shared D&D helpers keep prompt input
|
|
names and source-unit reference conversion consistent with the scene chunker.
|
|
|
|
The durable payload and manifest metadata shapes are defined in the
|
|
[D&D spell artifact contract](../integrations/dnd-spell-artifacts.md).
|
|
|
|
## Merger And Normalizer
|
|
|
|
### `internal/modules/generic/merge/appendorder`
|
|
|
|
The merger passes typed values to an injected combine function in framework
|
|
source-chunk order. The D&D registrar specializes it with a spell-list append
|
|
function.
|
|
|
|
### `internal/modules/generic/normalize/noop`
|
|
|
|
The normalizer returns the merged domain value unchanged and is reusable for
|
|
any registered artifact type.
|
|
|
|
## Output Encoder
|
|
|
|
### `internal/modules/generic/output/json`
|
|
|
|
The JSON encoder sorts normalized results by lane, derives collision-checked
|
|
safe logical names, pretty-prints JSON payloads, and assembles the logical index,
|
|
manifest, rejected-result, warning, and lane files. Invalid JSON, unsupported
|
|
media types, unsafe names, and sanitized-name collisions are errors.
|
|
|
|
The encoder returns logical files only. The CLI places them on disk, and the
|
|
[JSON output contract](../integrations/json-output.md) defines their external
|
|
paths and schemas.
|
|
|
|
## Generic Validators
|
|
|
|
The generic validator implementations live under
|
|
`internal/modules/generic/validate`.
|
|
|
|
The unconditional accept and reject validators provide explicit chunk and
|
|
typed-artifact variants used primarily for controlled composition and tests.
|
|
|
|
The serialized JSON syntax validator uses `encoding/json` to reject malformed
|
|
representation bytes. The serialized JSON Schema validator requires schema
|
|
bytes, parses the instance and schema with `jsonschema`, and distinguishes
|
|
payload rejection from schema loading or compilation errors. The framework
|
|
serialized-validation request carries either canonical chunk bytes or artifact
|
|
codec bytes according to its target context. Neither validator calls the LLM.
|
|
|
|
## D&D Spell Validators
|
|
|
|
All three validators receive `dnd.SpellList` directly. The shape validator
|
|
rejects missing or empty spell fields and empty reference lists. The
|
|
source-reference validator applies generic source-reference validation to every
|
|
cited range. The relatedness validator warns when a case-insensitive spell name
|
|
is absent from all cited source text.
|
|
|
|
These validators are deterministic. Their selectable keys and production order
|
|
are defined in
|
|
[Configuration](../config.md#implemented-production-validators); their durable
|
|
payload rules are defined in the
|
|
[artifact contract](../integrations/dnd-spell-artifacts.md).
|
|
|
|
## Production Registration
|
|
|
|
Production composition occurs through family registrars. The CLI allocates one
|
|
complete framework registry set and one LLM asset registry. It invokes
|
|
`internal/modules/generic/register`,
|
|
`internal/modules/seriatim/register`, and `internal/modules/dnd/register` in
|
|
that order, then exposes the matching catalog for resolution. The generic and
|
|
Seriatim registrars own their production leaf registrations. The D&D registrar
|
|
owns D&D leaf registrations, the spell default-validator chain, and D&D
|
|
prompt/schema asset collection.
|
|
|
|
Concrete implementation packages do not import generic implementation
|
|
packages directly. A concrete family's `register` package is its composition
|
|
point for specializing reusable generic implementations, while the generic
|
|
registrar composes only generic children.
|
|
|
|
Core and framework production packages do not import production extensions.
|
|
CLI production code is the sole application composition root for extensions
|
|
and imports only exact family registrar packages. Other production packages,
|
|
including commands and newly introduced package trees, do not import module
|
|
packages directly. Compatibility tests in the CLI, core, and framework trees
|
|
may import roots and implementation leaves directly. Other non-module tests do
|
|
not receive that exemption. White-box tests within module families retain the
|
|
production family boundaries. `internal/modules/integration` is test
|
|
infrastructure: its black-box tests may compose multiple families, but it is
|
|
not a production module family or production dependency target.
|
|
|
|
## Adding An Extension
|
|
|
|
When adding a production module or validator:
|
|
|
|
1. implement the stage or validator contract and package-local key;
|
|
2. expose and test its spec, constructor, and registration function;
|
|
3. keep format or domain parsing inside the concrete package;
|
|
4. add package-owned prompt/schema assets when the extension is LLM-backed;
|
|
5. register it through its package-family registrar and add a default chain
|
|
there only when production policy requires one;
|
|
6. add resolution and composition coverage for capabilities, options,
|
|
references, and validation behavior;
|
|
7. update the selectable-key catalog in [Configuration](../config.md), the
|
|
relevant external contract, this inventory, and maintained examples when
|
|
user-visible behavior changes.
|
|
|
|
Do not add the extension to `docs/development.md`; that file routes by task and
|
|
does not inventory implementations.
|
|
|
|
## Tests To Inspect
|
|
|
|
- Package-local `*_test.go` files under the module or validator being changed.
|
|
- `internal/framework/pipeline/typed_resolution_test.go`: typed registry, spec,
|
|
and heterogeneous artifact composition.
|
|
- `internal/framework/pipeline/default_modules_test.go`: framework binding
|
|
defaults.
|
|
- `internal/cli/run_test.go`: production catalog, config resolution, and
|
|
end-to-end CLI composition.
|
|
- `internal/framework/promptfs/*_test.go` and
|
|
`internal/modules/dnd/shared/*_test.go`: shared prompt and reference assembly.
|
|
- `internal/modules/integration/*_test.go`: black-box composition across
|
|
production extension domains.
|