Compose production modules through family registrars
This commit is contained in:
@@ -28,8 +28,8 @@ without exposing Scriptorium types through stage contracts.
|
||||
|
||||
`internal/cli` constructs the production runtime by:
|
||||
|
||||
1. collecting embedded prompt and response-schema assets from production module
|
||||
packages;
|
||||
1. allocating the asset registry populated by the generic, Seriatim, and D&D
|
||||
package-family registrars;
|
||||
2. creating a `ScriptoriumClient` from the effective profile source;
|
||||
3. attaching an `LLMProfileRecorder`;
|
||||
4. creating a scheduler from the effective concurrency limit;
|
||||
|
||||
@@ -12,7 +12,9 @@ Configuration.
|
||||
A stage module package provides a stable key, constructor, contract
|
||||
implementation, `ModuleSpec`, `Register`, and focused behavior and registration
|
||||
tests. A validator package follows the same pattern with `ValidatorSpec` and the
|
||||
validator registry.
|
||||
validator registry. Package-family registrars compose those leaf registrations
|
||||
into the production catalog and own family-level policy such as default
|
||||
validator chains and prompt asset collection.
|
||||
|
||||
Specs expose capability and execution metadata without constructing an
|
||||
implementation. Chunk, extract, merge, and normalize modules that accept
|
||||
@@ -154,10 +156,13 @@ payload rules are defined in the
|
||||
|
||||
## Production Registration
|
||||
|
||||
`internal/cli/catalog.go` builds the production registries, registers module and
|
||||
validator constructors, installs default validator-chain mappings, and exposes
|
||||
the matching catalog for resolution. It also collects prompt assets from
|
||||
LLM-backed packages before constructing the production client.
|
||||
The CLI allocates one complete framework registry set and one LLM asset
|
||||
registry. It invokes `internal/modules/generic/register`,
|
||||
`internal/modules/seriatim/register`, and `internal/modules/dnd/register` in
|
||||
that order, then exposes the matching catalog for resolution. The generic and
|
||||
Seriatim registrars own their production leaf registrations. The D&D registrar
|
||||
owns D&D leaf registrations, the spell default-validator chain, and D&D
|
||||
prompt/schema asset collection.
|
||||
|
||||
Framework packages must not import production extensions. Tests may compose
|
||||
registries and catalogs directly with fakes.
|
||||
@@ -170,8 +175,8 @@ When adding a production module or validator:
|
||||
2. expose and test its spec, constructor, and registration function;
|
||||
3. keep format or domain parsing inside the concrete package;
|
||||
4. add package-owned prompt/schema assets when the extension is LLM-backed;
|
||||
5. register it in `internal/cli/catalog.go` and add a default chain only when
|
||||
production policy requires one;
|
||||
5. register it through its package-family registrar and add a default chain
|
||||
there only when production policy requires one;
|
||||
6. add resolution and composition coverage for capabilities, options,
|
||||
references, and validation behavior;
|
||||
7. update the selectable-key catalog in [Configuration](../config.md), the
|
||||
|
||||
@@ -22,7 +22,7 @@ a sorted set of artifact lanes before the runner constructs any stage module.
|
||||
| Package | Implemented responsibility |
|
||||
| --- | --- |
|
||||
| `cmd/notarius` | Executable entry point and process exit delegation. |
|
||||
| `internal/cli` | Command parsing, config discovery, production registration, prompt asset collection, LLM client construction, reference materialization, workspace collaborator setup, durable writes, and user-facing results. |
|
||||
| `internal/cli` | Command parsing, config discovery, package-family registrar invocation, LLM client construction, reference materialization, workspace collaborator setup, durable writes, and user-facing results. |
|
||||
|
||||
## Core Packages
|
||||
|
||||
@@ -75,7 +75,19 @@ Concrete validators live under `internal/validators`. Generic packages provide
|
||||
unconditional test decisions, JSON syntax validation, and JSON Schema
|
||||
validation. D&D spell packages provide shape, source-reference, and
|
||||
source-relatedness decisions, with `spellpayload` holding their shared parser
|
||||
and lookup helpers. Production chain composition is owned by `internal/cli`.
|
||||
and lookup helpers.
|
||||
|
||||
Production composition is grouped behind package-family registrars while the
|
||||
implementations remain in their current stage-oriented packages:
|
||||
|
||||
| Package | Implemented responsibility |
|
||||
| --- | --- |
|
||||
| `internal/modules/generic/register` | Registers domain-neutral chunk, merge, normalize, output, and validator implementations. |
|
||||
| `internal/modules/seriatim/register` | Registers the Seriatim input adapter. |
|
||||
| `internal/modules/dnd/register` | Registers D&D modules, validators, default validator policy, and prompt/schema assets. |
|
||||
|
||||
The CLI allocates the framework registries and asset registry, then invokes
|
||||
these registrars in generic, Seriatim, and D&D order.
|
||||
|
||||
Implementation details for all production extensions are in
|
||||
[Module Internals](modules.md).
|
||||
|
||||
@@ -9,23 +9,17 @@ import (
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/framework/llm"
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/framework/pipeline"
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/modules/chunk/dnd/scenes"
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/modules/chunk/generic"
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/modules/extract/dnd/spells"
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/modules/input/seriatim"
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/modules/merge/appendorder"
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/modules/normalize/noop"
|
||||
jsonoutput "gitea.maximumdirect.net/eric/notarius/internal/modules/output/json"
|
||||
spellshape "gitea.maximumdirect.net/eric/notarius/internal/validators/extract/dnd/spells/shape"
|
||||
spellsourcerefs "gitea.maximumdirect.net/eric/notarius/internal/validators/extract/dnd/spells/source_refs"
|
||||
spellrelatedness "gitea.maximumdirect.net/eric/notarius/internal/validators/extract/dnd/spells/source_relatedness"
|
||||
alwaysaccept "gitea.maximumdirect.net/eric/notarius/internal/validators/generic/always_accept"
|
||||
alwaysreject "gitea.maximumdirect.net/eric/notarius/internal/validators/generic/always_reject"
|
||||
validjson "gitea.maximumdirect.net/eric/notarius/internal/validators/generic/valid_json"
|
||||
validjsonschema "gitea.maximumdirect.net/eric/notarius/internal/validators/generic/valid_json_schema"
|
||||
dndregister "gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/register"
|
||||
genericregister "gitea.maximumdirect.net/eric/notarius/internal/modules/generic/register"
|
||||
seriatimregister "gitea.maximumdirect.net/eric/notarius/internal/modules/seriatim/register"
|
||||
)
|
||||
|
||||
func productionRegistries() (pipeline.Registries, error) {
|
||||
type productionComponents struct {
|
||||
registries pipeline.Registries
|
||||
assets *llm.AssetRegistry
|
||||
}
|
||||
|
||||
func newProductionComponents() (productionComponents, error) {
|
||||
registries := pipeline.Registries{
|
||||
Inputs: pipeline.NewInputAdapterRegistry(),
|
||||
Chunkers: pipeline.NewChunkerRegistry(),
|
||||
@@ -36,72 +30,26 @@ func productionRegistries() (pipeline.Registries, error) {
|
||||
ValidatorChains: pipeline.NewValidatorChainRegistry(),
|
||||
Outputs: pipeline.NewOutputEncoderRegistry(),
|
||||
}
|
||||
if err := seriatim.Register(registries.Inputs); err != nil {
|
||||
return pipeline.Registries{}, fmt.Errorf("register seriatim input: %w", err)
|
||||
}
|
||||
if err := generic.Register(registries.Chunkers); err != nil {
|
||||
return pipeline.Registries{}, fmt.Errorf("register generic chunker: %w", err)
|
||||
}
|
||||
if err := scenes.Register(registries.Chunkers); err != nil {
|
||||
return pipeline.Registries{}, fmt.Errorf("register dnd scenes chunker: %w", err)
|
||||
}
|
||||
if err := spells.Register(registries.Extractors); err != nil {
|
||||
return pipeline.Registries{}, fmt.Errorf("register dnd spells extractor: %w", err)
|
||||
}
|
||||
if err := appendorder.Register(registries.Mergers); err != nil {
|
||||
return pipeline.Registries{}, fmt.Errorf("register appendorder merger: %w", err)
|
||||
}
|
||||
if err := noop.Register(registries.Normalizers); err != nil {
|
||||
return pipeline.Registries{}, fmt.Errorf("register noop normalizer: %w", err)
|
||||
}
|
||||
if err := registerProductionValidators(registries.Validators); err != nil {
|
||||
return pipeline.Registries{}, err
|
||||
}
|
||||
if err := registerProductionValidatorChains(registries.ValidatorChains); err != nil {
|
||||
return pipeline.Registries{}, err
|
||||
}
|
||||
if err := jsonoutput.Register(registries.Outputs); err != nil {
|
||||
return pipeline.Registries{}, fmt.Errorf("register json output encoder: %w", err)
|
||||
}
|
||||
return registries, nil
|
||||
}
|
||||
|
||||
func registerProductionValidators(registry *pipeline.ValidatorRegistry) error {
|
||||
registrations := []struct {
|
||||
assets := llm.NewAssetRegistry()
|
||||
registrars := []struct {
|
||||
name string
|
||||
register func(*pipeline.ValidatorRegistry) error
|
||||
register func(pipeline.Registries, *llm.AssetRegistry) error
|
||||
}{
|
||||
{name: "generic always accept validator", register: alwaysaccept.Register},
|
||||
{name: "generic always reject validator", register: alwaysreject.Register},
|
||||
{name: "generic valid json validator", register: validjson.Register},
|
||||
{name: "generic valid json schema validator", register: validjsonschema.Register},
|
||||
{name: "dnd spell shape validator", register: spellshape.Register},
|
||||
{name: "dnd spell source references validator", register: spellsourcerefs.Register},
|
||||
{name: "dnd spell source relatedness validator", register: spellrelatedness.Register},
|
||||
{name: "generic", register: genericregister.Register},
|
||||
{name: "seriatim", register: seriatimregister.Register},
|
||||
{name: "dnd", register: dndregister.Register},
|
||||
}
|
||||
for _, registration := range registrations {
|
||||
if err := registration.register(registry); err != nil {
|
||||
return fmt.Errorf("register %s: %w", registration.name, err)
|
||||
for _, registrar := range registrars {
|
||||
if err := registrar.register(registries, assets); err != nil {
|
||||
return productionComponents{}, fmt.Errorf("register %s module family: %w", registrar.name, err)
|
||||
}
|
||||
}
|
||||
return nil
|
||||
return productionComponents{registries: registries, assets: assets}, nil
|
||||
}
|
||||
|
||||
func registerProductionValidatorChains(registry *pipeline.ValidatorChainRegistry) error {
|
||||
if err := registry.Register(pipeline.ValidatorChainMapping{
|
||||
Stage: pipeline.StageExtract,
|
||||
Module: spells.Key,
|
||||
Validators: []pipeline.ModuleBinding{
|
||||
pipeline.Binding(validjson.Key),
|
||||
pipeline.Binding(validjsonschema.Key),
|
||||
pipeline.Binding(spellshape.Key),
|
||||
pipeline.Binding(spellsourcerefs.Key),
|
||||
pipeline.Binding(spellrelatedness.Key),
|
||||
},
|
||||
}); err != nil {
|
||||
return fmt.Errorf("register dnd spells validator chain: %w", err)
|
||||
}
|
||||
return nil
|
||||
func productionRegistries() (pipeline.Registries, error) {
|
||||
components, err := newProductionComponents()
|
||||
return components.registries, err
|
||||
}
|
||||
|
||||
func productionCatalog() (pipeline.ModuleCatalog, error) {
|
||||
@@ -113,14 +61,8 @@ func productionCatalog() (pipeline.ModuleCatalog, error) {
|
||||
}
|
||||
|
||||
func productionPromptAssets() (*llm.AssetRegistry, error) {
|
||||
registry := llm.NewAssetRegistry()
|
||||
if err := scenes.RegisterPromptAssets(registry); err != nil {
|
||||
return nil, fmt.Errorf("register dnd scenes prompt assets: %w", err)
|
||||
}
|
||||
if err := spells.RegisterPromptAssets(registry); err != nil {
|
||||
return nil, fmt.Errorf("register dnd spells prompt assets: %w", err)
|
||||
}
|
||||
return registry, nil
|
||||
components, err := newProductionComponents()
|
||||
return components.assets, err
|
||||
}
|
||||
|
||||
func effectiveCatalog(opts Options) (pipeline.ModuleCatalog, error) {
|
||||
@@ -199,6 +141,22 @@ func productionLLMClientFactory(ctx context.Context, cfg config.Config, profileI
|
||||
if err != nil {
|
||||
return nil, nil, err
|
||||
}
|
||||
return buildProductionLLMClient(ctx, cfg, profileID, assets)
|
||||
}
|
||||
|
||||
func productionLLMClientFactoryWithAssets(assets *llm.AssetRegistry) LLMClientFactory {
|
||||
return func(ctx context.Context, cfg config.Config, profileID string) (contracts.StructuredLLMClient, []artifacts.LLMProfileManifest, error) {
|
||||
return buildProductionLLMClient(ctx, cfg, profileID, assets)
|
||||
}
|
||||
}
|
||||
|
||||
func buildProductionLLMClient(ctx context.Context, cfg config.Config, profileID string, assets *llm.AssetRegistry) (contracts.StructuredLLMClient, []artifacts.LLMProfileManifest, error) {
|
||||
if err := ctx.Err(); err != nil {
|
||||
return nil, nil, err
|
||||
}
|
||||
if assets == nil {
|
||||
return nil, nil, fmt.Errorf("production asset registry must not be nil")
|
||||
}
|
||||
recorder := llm.NewLLMProfileRecorder()
|
||||
client, err := llm.NewScriptoriumClient(llm.ScriptoriumClientConfig{
|
||||
ProfileDir: cfg.Scriptorium.ProfileDir,
|
||||
|
||||
@@ -28,6 +28,24 @@ import (
|
||||
)
|
||||
|
||||
func TestProductionCompatibilitySnapshot(t *testing.T) {
|
||||
normalizedOptions, err := normalizeOptions(Options{})
|
||||
if err != nil {
|
||||
t.Fatalf("normalizeOptions() error = %v, want nil", err)
|
||||
}
|
||||
if normalizedOptions.Catalog.Inputs != normalizedOptions.Registries.Inputs ||
|
||||
normalizedOptions.Catalog.Chunkers != normalizedOptions.Registries.Chunkers ||
|
||||
normalizedOptions.Catalog.Extractors != normalizedOptions.Registries.Extractors ||
|
||||
normalizedOptions.Catalog.Mergers != normalizedOptions.Registries.Mergers ||
|
||||
normalizedOptions.Catalog.Normalizers != normalizedOptions.Registries.Normalizers ||
|
||||
normalizedOptions.Catalog.Validators != normalizedOptions.Registries.Validators ||
|
||||
normalizedOptions.Catalog.ValidatorChains != normalizedOptions.Registries.ValidatorChains ||
|
||||
normalizedOptions.Catalog.Outputs != normalizedOptions.Registries.Outputs {
|
||||
t.Fatal("production catalog and execution registries do not share one composition")
|
||||
}
|
||||
if normalizedOptions.LLMClientFactory == nil {
|
||||
t.Fatal("production LLM client factory is nil")
|
||||
}
|
||||
|
||||
registries, err := productionRegistries()
|
||||
if err != nil {
|
||||
t.Fatalf("productionRegistries() error = %v, want nil", err)
|
||||
|
||||
@@ -51,7 +51,12 @@ func Run(args []string, stdout, stderr io.Writer) int {
|
||||
}
|
||||
|
||||
func RunWithOptions(args []string, stdout, stderr io.Writer, opts Options) int {
|
||||
opts = normalizeOptions(opts)
|
||||
var err error
|
||||
opts, err = normalizeOptions(opts)
|
||||
if err != nil {
|
||||
fmt.Fprintf(stderr, "notarius: %v\n", err)
|
||||
return 1
|
||||
}
|
||||
if len(args) == 0 {
|
||||
writeUsage(stdout)
|
||||
return 0
|
||||
@@ -78,17 +83,28 @@ func writeUsage(w io.Writer) {
|
||||
fmt.Fprint(w, usage)
|
||||
}
|
||||
|
||||
func normalizeOptions(opts Options) Options {
|
||||
func normalizeOptions(opts Options) (Options, error) {
|
||||
if opts.LookupEnv == nil {
|
||||
opts.LookupEnv = os.LookupEnv
|
||||
}
|
||||
if opts.Now == nil {
|
||||
opts.Now = time.Now
|
||||
}
|
||||
if isEmptyCatalog(opts.Catalog) && isEmptyRegistries(opts.Registries) {
|
||||
components, err := newProductionComponents()
|
||||
if err != nil {
|
||||
return Options{}, err
|
||||
}
|
||||
opts.Registries = components.registries
|
||||
opts.Catalog = catalogFromRegistries(components.registries)
|
||||
if opts.LLMClientFactory == nil {
|
||||
opts.LLMClientFactory = productionLLMClientFactoryWithAssets(components.assets)
|
||||
}
|
||||
}
|
||||
if opts.LLMClientFactory == nil {
|
||||
opts.LLMClientFactory = productionLLMClientFactory
|
||||
}
|
||||
return opts
|
||||
return opts, nil
|
||||
}
|
||||
|
||||
func runPipelineCommand(args []string, stdout, stderr io.Writer, opts Options) int {
|
||||
|
||||
71
internal/modules/dnd/register/register.go
Normal file
71
internal/modules/dnd/register/register.go
Normal file
@@ -0,0 +1,71 @@
|
||||
// Package register composes the production D&D module family.
|
||||
package register
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/framework/llm"
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/framework/pipeline"
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/modules/chunk/dnd/scenes"
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/modules/extract/dnd/spells"
|
||||
spellshape "gitea.maximumdirect.net/eric/notarius/internal/validators/extract/dnd/spells/shape"
|
||||
spellsourcerefs "gitea.maximumdirect.net/eric/notarius/internal/validators/extract/dnd/spells/source_refs"
|
||||
spellrelatedness "gitea.maximumdirect.net/eric/notarius/internal/validators/extract/dnd/spells/source_relatedness"
|
||||
validjson "gitea.maximumdirect.net/eric/notarius/internal/validators/generic/valid_json"
|
||||
validjsonschema "gitea.maximumdirect.net/eric/notarius/internal/validators/generic/valid_json_schema"
|
||||
)
|
||||
|
||||
// Register adds all production D&D modules, validators, policy, and assets.
|
||||
func Register(registries pipeline.Registries, assets *llm.AssetRegistry) error {
|
||||
if err := validateRegistries(registries, assets); err != nil {
|
||||
return err
|
||||
}
|
||||
registrations := []struct {
|
||||
name string
|
||||
register func() error
|
||||
}{
|
||||
{name: "scenes chunker", register: func() error { return scenes.Register(registries.Chunkers) }},
|
||||
{name: "spells extractor", register: func() error { return spells.Register(registries.Extractors) }},
|
||||
{name: "spell shape validator", register: func() error { return spellshape.Register(registries.Validators) }},
|
||||
{name: "spell source references validator", register: func() error { return spellsourcerefs.Register(registries.Validators) }},
|
||||
{name: "spell source relatedness validator", register: func() error { return spellrelatedness.Register(registries.Validators) }},
|
||||
{name: "scenes prompt assets", register: func() error { return scenes.RegisterPromptAssets(assets) }},
|
||||
{name: "spells prompt assets", register: func() error { return spells.RegisterPromptAssets(assets) }},
|
||||
}
|
||||
for _, registration := range registrations {
|
||||
if err := registration.register(); err != nil {
|
||||
return fmt.Errorf("register dnd %s: %w", registration.name, err)
|
||||
}
|
||||
}
|
||||
if err := registries.ValidatorChains.Register(pipeline.ValidatorChainMapping{
|
||||
Stage: pipeline.StageExtract,
|
||||
Module: spells.Key,
|
||||
Validators: []pipeline.ModuleBinding{
|
||||
pipeline.Binding(validjson.Key),
|
||||
pipeline.Binding(validjsonschema.Key),
|
||||
pipeline.Binding(spellshape.Key),
|
||||
pipeline.Binding(spellsourcerefs.Key),
|
||||
pipeline.Binding(spellrelatedness.Key),
|
||||
},
|
||||
}); err != nil {
|
||||
return fmt.Errorf("register dnd spells validator chain: %w", err)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func validateRegistries(registries pipeline.Registries, assets *llm.AssetRegistry) error {
|
||||
switch {
|
||||
case registries.Chunkers == nil:
|
||||
return fmt.Errorf("dnd registrar: chunker registry must not be nil")
|
||||
case registries.Extractors == nil:
|
||||
return fmt.Errorf("dnd registrar: extractor registry must not be nil")
|
||||
case registries.Validators == nil:
|
||||
return fmt.Errorf("dnd registrar: validator registry must not be nil")
|
||||
case registries.ValidatorChains == nil:
|
||||
return fmt.Errorf("dnd registrar: validator chain registry must not be nil")
|
||||
case assets == nil:
|
||||
return fmt.Errorf("dnd registrar: asset registry must not be nil")
|
||||
default:
|
||||
return nil
|
||||
}
|
||||
}
|
||||
138
internal/modules/dnd/register/register_test.go
Normal file
138
internal/modules/dnd/register/register_test.go
Normal file
@@ -0,0 +1,138 @@
|
||||
package register
|
||||
|
||||
import (
|
||||
"io/fs"
|
||||
"reflect"
|
||||
"sort"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/framework/llm"
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/framework/pipeline"
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/modules/extract/dnd/spells"
|
||||
)
|
||||
|
||||
func TestRegisterAddsDNDFamily(t *testing.T) {
|
||||
registries := completeRegistries()
|
||||
assets := llm.NewAssetRegistry()
|
||||
if err := Register(registries, assets); err != nil {
|
||||
t.Fatalf("Register() error = %v, want nil", err)
|
||||
}
|
||||
assertKeys(t, "chunkers", registries.Chunkers.RegisteredKeys(), []string{"dnd/scenes"})
|
||||
assertKeys(t, "extractors", registries.Extractors.RegisteredKeys(), []string{"dnd/spells"})
|
||||
assertKeys(t, "validators", registries.Validators.RegisteredKeys(), []string{
|
||||
"extract/dnd/spells/shape",
|
||||
"extract/dnd/spells/source_refs",
|
||||
"extract/dnd/spells/source_relatedness",
|
||||
})
|
||||
wantChain := []pipeline.ModuleBinding{
|
||||
pipeline.Binding("generic/valid_json"),
|
||||
pipeline.Binding("generic/valid_json_schema"),
|
||||
pipeline.Binding("extract/dnd/spells/shape"),
|
||||
pipeline.Binding("extract/dnd/spells/source_refs"),
|
||||
pipeline.Binding("extract/dnd/spells/source_relatedness"),
|
||||
}
|
||||
if got := registries.ValidatorChains.Validators(pipeline.StageExtract, spells.Key); !reflect.DeepEqual(got, wantChain) {
|
||||
t.Fatalf("spell validator chain = %#v, want %#v", got, wantChain)
|
||||
}
|
||||
assertAssetNames(t, assets.PromptFS, []string{
|
||||
"dnd.scenes/dnd.scenes.yaml",
|
||||
"dnd.scenes/instructions.md",
|
||||
"dnd.scenes/sharedassets/common-dnd-references.md",
|
||||
"dnd.scenes/sharedassets/common-dnd-system.md",
|
||||
"dnd.scenes/sharedassets/common-dnd-transcript.md",
|
||||
"dnd.scenes/task.md",
|
||||
"dnd.spells/dnd.spells.yaml",
|
||||
"dnd.spells/instructions.md",
|
||||
"dnd.spells/sharedassets/common-dnd-references.md",
|
||||
"dnd.spells/sharedassets/common-dnd-system.md",
|
||||
"dnd.spells/sharedassets/common-dnd-transcript.md",
|
||||
"dnd.spells/task.md",
|
||||
})
|
||||
assertAssetNames(t, assets.SchemaFS, []string{
|
||||
"dnd_scenes.v1.json",
|
||||
"dnd_spells.v1.json",
|
||||
"dnd_spells_llm.v1.json",
|
||||
})
|
||||
}
|
||||
|
||||
func TestRegisterRejectsMissingDNDDependenciesBeforeMutation(t *testing.T) {
|
||||
tests := []struct {
|
||||
name string
|
||||
remove func(*pipeline.Registries, **llm.AssetRegistry)
|
||||
wantErr string
|
||||
}{
|
||||
{name: "chunkers", remove: func(r *pipeline.Registries, _ **llm.AssetRegistry) { r.Chunkers = nil }, wantErr: "chunker registry"},
|
||||
{name: "extractors", remove: func(r *pipeline.Registries, _ **llm.AssetRegistry) { r.Extractors = nil }, wantErr: "extractor registry"},
|
||||
{name: "validators", remove: func(r *pipeline.Registries, _ **llm.AssetRegistry) { r.Validators = nil }, wantErr: "validator registry"},
|
||||
{name: "validator chains", remove: func(r *pipeline.Registries, _ **llm.AssetRegistry) { r.ValidatorChains = nil }, wantErr: "validator chain registry"},
|
||||
{name: "assets", remove: func(_ *pipeline.Registries, assets **llm.AssetRegistry) { *assets = nil }, wantErr: "asset registry"},
|
||||
}
|
||||
for _, test := range tests {
|
||||
t.Run(test.name, func(t *testing.T) {
|
||||
registries := completeRegistries()
|
||||
assets := llm.NewAssetRegistry()
|
||||
test.remove(®istries, &assets)
|
||||
err := Register(registries, assets)
|
||||
if err == nil || !strings.Contains(err.Error(), test.wantErr) {
|
||||
t.Fatalf("Register() error = %v, want %q", err, test.wantErr)
|
||||
}
|
||||
if got := registries.Chunkers; got != nil && len(got.RegisteredKeys()) != 0 {
|
||||
t.Fatalf("chunker keys = %#v, want validation before mutation", got.RegisteredKeys())
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestRegisterReportsDuplicateDNDRegistration(t *testing.T) {
|
||||
registries := completeRegistries()
|
||||
assets := llm.NewAssetRegistry()
|
||||
if err := Register(registries, assets); err != nil {
|
||||
t.Fatalf("first Register() error = %v, want nil", err)
|
||||
}
|
||||
err := Register(registries, assets)
|
||||
if err == nil || !strings.Contains(err.Error(), "register dnd scenes chunker") || !strings.Contains(err.Error(), "already registered") {
|
||||
t.Fatalf("second Register() error = %v, want contextual duplicate error", err)
|
||||
}
|
||||
}
|
||||
|
||||
func completeRegistries() pipeline.Registries {
|
||||
return pipeline.Registries{
|
||||
Inputs: pipeline.NewInputAdapterRegistry(),
|
||||
Chunkers: pipeline.NewChunkerRegistry(),
|
||||
Extractors: pipeline.NewExtractorRegistry(),
|
||||
Mergers: pipeline.NewMergerRegistry(),
|
||||
Normalizers: pipeline.NewNormalizerRegistry(),
|
||||
Validators: pipeline.NewValidatorRegistry(),
|
||||
ValidatorChains: pipeline.NewValidatorChainRegistry(),
|
||||
Outputs: pipeline.NewOutputEncoderRegistry(),
|
||||
}
|
||||
}
|
||||
|
||||
func assertKeys(t *testing.T, name string, got, want []string) {
|
||||
t.Helper()
|
||||
if !reflect.DeepEqual(got, want) {
|
||||
t.Fatalf("%s keys = %#v, want %#v", name, got, want)
|
||||
}
|
||||
}
|
||||
|
||||
func assertAssetNames(t *testing.T, getFS func() (fs.FS, error), want []string) {
|
||||
t.Helper()
|
||||
fSys, err := getFS()
|
||||
if err != nil {
|
||||
t.Fatalf("asset filesystem error = %v, want nil", err)
|
||||
}
|
||||
var got []string
|
||||
if err := fs.WalkDir(fSys, ".", func(path string, entry fs.DirEntry, err error) error {
|
||||
if err == nil && !entry.IsDir() {
|
||||
got = append(got, path)
|
||||
}
|
||||
return err
|
||||
}); err != nil {
|
||||
t.Fatalf("walk assets: %v", err)
|
||||
}
|
||||
sort.Strings(got)
|
||||
if !reflect.DeepEqual(got, want) {
|
||||
t.Fatalf("asset names = %#v, want %#v", got, want)
|
||||
}
|
||||
}
|
||||
61
internal/modules/generic/register/register.go
Normal file
61
internal/modules/generic/register/register.go
Normal file
@@ -0,0 +1,61 @@
|
||||
// Package register composes the production domain-neutral module family.
|
||||
package register
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/framework/llm"
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/framework/pipeline"
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/modules/chunk/generic"
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/modules/merge/appendorder"
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/modules/normalize/noop"
|
||||
jsonoutput "gitea.maximumdirect.net/eric/notarius/internal/modules/output/json"
|
||||
alwaysaccept "gitea.maximumdirect.net/eric/notarius/internal/validators/generic/always_accept"
|
||||
alwaysreject "gitea.maximumdirect.net/eric/notarius/internal/validators/generic/always_reject"
|
||||
validjson "gitea.maximumdirect.net/eric/notarius/internal/validators/generic/valid_json"
|
||||
validjsonschema "gitea.maximumdirect.net/eric/notarius/internal/validators/generic/valid_json_schema"
|
||||
)
|
||||
|
||||
// Register adds all production domain-neutral modules and validators.
|
||||
func Register(registries pipeline.Registries, assets *llm.AssetRegistry) error {
|
||||
_ = assets
|
||||
if err := validateRegistries(registries); err != nil {
|
||||
return err
|
||||
}
|
||||
registrations := []struct {
|
||||
name string
|
||||
register func() error
|
||||
}{
|
||||
{name: "generic chunker", register: func() error { return generic.Register(registries.Chunkers) }},
|
||||
{name: "appendorder merger", register: func() error { return appendorder.Register(registries.Mergers) }},
|
||||
{name: "noop normalizer", register: func() error { return noop.Register(registries.Normalizers) }},
|
||||
{name: "always accept validator", register: func() error { return alwaysaccept.Register(registries.Validators) }},
|
||||
{name: "always reject validator", register: func() error { return alwaysreject.Register(registries.Validators) }},
|
||||
{name: "valid json validator", register: func() error { return validjson.Register(registries.Validators) }},
|
||||
{name: "valid json schema validator", register: func() error { return validjsonschema.Register(registries.Validators) }},
|
||||
{name: "json output encoder", register: func() error { return jsonoutput.Register(registries.Outputs) }},
|
||||
}
|
||||
for _, registration := range registrations {
|
||||
if err := registration.register(); err != nil {
|
||||
return fmt.Errorf("register %s: %w", registration.name, err)
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func validateRegistries(registries pipeline.Registries) error {
|
||||
switch {
|
||||
case registries.Chunkers == nil:
|
||||
return fmt.Errorf("generic registrar: chunker registry must not be nil")
|
||||
case registries.Mergers == nil:
|
||||
return fmt.Errorf("generic registrar: merger registry must not be nil")
|
||||
case registries.Normalizers == nil:
|
||||
return fmt.Errorf("generic registrar: normalizer registry must not be nil")
|
||||
case registries.Validators == nil:
|
||||
return fmt.Errorf("generic registrar: validator registry must not be nil")
|
||||
case registries.Outputs == nil:
|
||||
return fmt.Errorf("generic registrar: output registry must not be nil")
|
||||
default:
|
||||
return nil
|
||||
}
|
||||
}
|
||||
90
internal/modules/generic/register/register_test.go
Normal file
90
internal/modules/generic/register/register_test.go
Normal file
@@ -0,0 +1,90 @@
|
||||
package register
|
||||
|
||||
import (
|
||||
"reflect"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/framework/pipeline"
|
||||
)
|
||||
|
||||
func TestRegisterAddsGenericFamily(t *testing.T) {
|
||||
registries := completeRegistries()
|
||||
if err := Register(registries, nil); err != nil {
|
||||
t.Fatalf("Register() error = %v, want nil", err)
|
||||
}
|
||||
assertKeys(t, "chunkers", registries.Chunkers.RegisteredKeys(), []string{"generic"})
|
||||
assertKeys(t, "mergers", registries.Mergers.RegisteredKeys(), []string{"appendorder"})
|
||||
assertKeys(t, "normalizers", registries.Normalizers.RegisteredKeys(), []string{"noop"})
|
||||
assertKeys(t, "validators", registries.Validators.RegisteredKeys(), []string{
|
||||
"generic/always_accept",
|
||||
"generic/always_reject",
|
||||
"generic/valid_json",
|
||||
"generic/valid_json_schema",
|
||||
})
|
||||
assertKeys(t, "outputs", registries.Outputs.RegisteredKeys(), []string{"json"})
|
||||
if got := registries.Inputs.RegisteredKeys(); len(got) != 0 {
|
||||
t.Fatalf("input keys = %#v, want generic registrar to leave inputs unchanged", got)
|
||||
}
|
||||
if got := registries.Extractors.RegisteredKeys(); len(got) != 0 {
|
||||
t.Fatalf("extractor keys = %#v, want generic registrar to leave extractors unchanged", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestRegisterRejectsMissingGenericRegistriesBeforeMutation(t *testing.T) {
|
||||
tests := []struct {
|
||||
name string
|
||||
remove func(*pipeline.Registries)
|
||||
wantErr string
|
||||
}{
|
||||
{name: "chunkers", remove: func(r *pipeline.Registries) { r.Chunkers = nil }, wantErr: "chunker registry"},
|
||||
{name: "mergers", remove: func(r *pipeline.Registries) { r.Mergers = nil }, wantErr: "merger registry"},
|
||||
{name: "normalizers", remove: func(r *pipeline.Registries) { r.Normalizers = nil }, wantErr: "normalizer registry"},
|
||||
{name: "validators", remove: func(r *pipeline.Registries) { r.Validators = nil }, wantErr: "validator registry"},
|
||||
{name: "outputs", remove: func(r *pipeline.Registries) { r.Outputs = nil }, wantErr: "output registry"},
|
||||
}
|
||||
for _, test := range tests {
|
||||
t.Run(test.name, func(t *testing.T) {
|
||||
registries := completeRegistries()
|
||||
test.remove(®istries)
|
||||
err := Register(registries, nil)
|
||||
if err == nil || !strings.Contains(err.Error(), test.wantErr) {
|
||||
t.Fatalf("Register() error = %v, want %q", err, test.wantErr)
|
||||
}
|
||||
if got := registries.Chunkers; got != nil && len(got.RegisteredKeys()) != 0 {
|
||||
t.Fatalf("chunker keys = %#v, want validation before mutation", got.RegisteredKeys())
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestRegisterReportsDuplicateGenericRegistration(t *testing.T) {
|
||||
registries := completeRegistries()
|
||||
if err := Register(registries, nil); err != nil {
|
||||
t.Fatalf("first Register() error = %v, want nil", err)
|
||||
}
|
||||
err := Register(registries, nil)
|
||||
if err == nil || !strings.Contains(err.Error(), "register generic chunker") || !strings.Contains(err.Error(), "already registered") {
|
||||
t.Fatalf("second Register() error = %v, want contextual duplicate error", err)
|
||||
}
|
||||
}
|
||||
|
||||
func completeRegistries() pipeline.Registries {
|
||||
return pipeline.Registries{
|
||||
Inputs: pipeline.NewInputAdapterRegistry(),
|
||||
Chunkers: pipeline.NewChunkerRegistry(),
|
||||
Extractors: pipeline.NewExtractorRegistry(),
|
||||
Mergers: pipeline.NewMergerRegistry(),
|
||||
Normalizers: pipeline.NewNormalizerRegistry(),
|
||||
Validators: pipeline.NewValidatorRegistry(),
|
||||
ValidatorChains: pipeline.NewValidatorChainRegistry(),
|
||||
Outputs: pipeline.NewOutputEncoderRegistry(),
|
||||
}
|
||||
}
|
||||
|
||||
func assertKeys(t *testing.T, name string, got, want []string) {
|
||||
t.Helper()
|
||||
if !reflect.DeepEqual(got, want) {
|
||||
t.Fatalf("%s keys = %#v, want %#v", name, got, want)
|
||||
}
|
||||
}
|
||||
22
internal/modules/seriatim/register/register.go
Normal file
22
internal/modules/seriatim/register/register.go
Normal file
@@ -0,0 +1,22 @@
|
||||
// Package register composes the production Seriatim module family.
|
||||
package register
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/framework/llm"
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/framework/pipeline"
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/modules/input/seriatim"
|
||||
)
|
||||
|
||||
// Register adds all production Seriatim modules.
|
||||
func Register(registries pipeline.Registries, assets *llm.AssetRegistry) error {
|
||||
_ = assets
|
||||
if registries.Inputs == nil {
|
||||
return fmt.Errorf("seriatim registrar: input registry must not be nil")
|
||||
}
|
||||
if err := seriatim.Register(registries.Inputs); err != nil {
|
||||
return fmt.Errorf("register transcript input: %w", err)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
37
internal/modules/seriatim/register/register_test.go
Normal file
37
internal/modules/seriatim/register/register_test.go
Normal file
@@ -0,0 +1,37 @@
|
||||
package register
|
||||
|
||||
import (
|
||||
"reflect"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/framework/pipeline"
|
||||
)
|
||||
|
||||
func TestRegisterAddsSeriatimFamily(t *testing.T) {
|
||||
registries := pipeline.Registries{Inputs: pipeline.NewInputAdapterRegistry()}
|
||||
if err := Register(registries, nil); err != nil {
|
||||
t.Fatalf("Register() error = %v, want nil", err)
|
||||
}
|
||||
if got, want := registries.Inputs.RegisteredKeys(), []string{"seriatim"}; !reflect.DeepEqual(got, want) {
|
||||
t.Fatalf("input keys = %#v, want %#v", got, want)
|
||||
}
|
||||
}
|
||||
|
||||
func TestRegisterRejectsNilInputRegistry(t *testing.T) {
|
||||
err := Register(pipeline.Registries{}, nil)
|
||||
if err == nil || !strings.Contains(err.Error(), "input registry must not be nil") {
|
||||
t.Fatalf("Register() error = %v, want nil input registry error", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestRegisterReportsDuplicateSeriatimRegistration(t *testing.T) {
|
||||
registries := pipeline.Registries{Inputs: pipeline.NewInputAdapterRegistry()}
|
||||
if err := Register(registries, nil); err != nil {
|
||||
t.Fatalf("first Register() error = %v, want nil", err)
|
||||
}
|
||||
err := Register(registries, nil)
|
||||
if err == nil || !strings.Contains(err.Error(), "register transcript input") || !strings.Contains(err.Error(), "already registered") {
|
||||
t.Fatalf("second Register() error = %v, want contextual duplicate error", err)
|
||||
}
|
||||
}
|
||||
Reference in New Issue
Block a user