diff --git a/docs/consumers/pkg-scriptorium.md b/docs/consumers/pkg-scriptorium.md index 3b09d84..1cbc174 100644 --- a/docs/consumers/pkg-scriptorium.md +++ b/docs/consumers/pkg-scriptorium.md @@ -21,7 +21,17 @@ if err != nil { } ``` -`PromptDir` is required. `ProfileDir` is optional; omit it to use built-in profiles only, or set it to overlay custom profiles above built-ins. `SchemaDir` defaults to the built-in schema directory. `Timeout` and `HTTPClient` configure the default OpenAI-compatible client used by `Run` when no custom LLM client is supplied. +`PromptDir` is required unless an explicit prompt source option is supplied. `ProfileDir` is optional; omit it to use built-in profiles only, or set it to overlay custom profiles above built-ins. `SchemaDir` defaults to the built-in schema directory. `Timeout` and `HTTPClient` configure the default OpenAI-compatible client used by `Run` when no custom LLM client is supplied. + +## Asset Sources + +Directory fields on `Config` remain the compatibility path. Explicit source options override the matching directory field: + +- `WithPromptFS(fsys, root)` and `WithPromptFile(path)` +- `WithProfileFS(fsys, root)` and `WithProfileFile(path)` +- `WithSchemaFS(fsys, root)` and `WithSchemaFile(path)` + +Prompt and profile sources load standard Scriptorium YAML with the same strict validation as directory sources. Prompt `content_file` paths resolve relative to the prompt file in the same source. Profile options overlay custom profiles above built-ins. Schema `fs.FS` sources preserve prompt `schema_path` semantics; schema file options expose the file by its base name. ## Prepare A Prompt diff --git a/docs/internal/adapters.md b/docs/internal/adapters.md index cd15c4e..f9f8100 100644 --- a/docs/internal/adapters.md +++ b/docs/internal/adapters.md @@ -9,13 +9,13 @@ This document describes implemented adapter/repository boundaries and their curr - `internal/adapter/cli`: CLI command parsing, app wiring, stdout/stderr handling, exit codes. - `internal/adapter/http`: HTTP request/response mapping for `POST /v1/runs`. - root package `scriptorium`: public Go library facade for preparing and running prompt requests. -- `internal/promptdef`: filesystem prompt-definition repository. +- `internal/promptdef`: filesystem and `fs.FS` prompt-definition repositories. - `internal/profile`: filesystem, `fs.FS`, and overlay execution-profile repositories. - `internal/profile/builtin`: embedded built-in execution-profile repository. - `internal/artifact`: input artifact reader. - `internal/prompt`: Go-template renderer. - `internal/llm`: OpenAI-compatible LLM client implementation. -- `internal/validate`: output validator. +- `internal/validate`: filesystem and `fs.FS` output validators. - `internal/format`: prepared-run formatters for `render` output. ## Inputs And Outputs @@ -38,18 +38,20 @@ Public library facade: - Output: typed `PreparedRun` and `RunResult` values plus public sentinel errors. - Custom LLM behavior is injected with `WithLLMClient`; otherwise the default OpenAI-compatible client is used. - `RunRequest.APIKey` is a request-scoped Go value only; it is converted into internal execution state for LLM generation and stripped from public result types. +- Prompt, profile, and schema source options can use directories, single files, or `fs.FS` roots. Explicit source options override the matching `Config` directory field. - Public types are facade types converted at the package boundary; internal domain types remain internal. -Filesystem repositories: +Prompt/profile repositories: -- Input: prompt/profile YAML files under configured directories. +- Input: prompt/profile YAML files under configured directories or `fs.FS` roots. - Output: normalized domain definitions/profiles or typed errors. +- Single-file public sources are represented as `fs.FS` roots containing one YAML file; lookup still uses YAML `id` values. Profile repository composition: - Built-in profiles are embedded and loaded through the same profile validation rules as filesystem profiles. - When no custom profile directory is configured, the runner receives the built-in profile repository. -- When a custom profile directory is configured, the runner receives an overlay repository with custom profiles as primary and built-ins as fallback. +- When a custom profile directory/file/`fs.FS` source is configured, the runner receives an overlay repository with custom profiles as primary and built-ins as fallback. - Overlay lookup falls back only after custom profile-not-found errors; custom load/validation/raw-key errors are returned directly. Artifact reader: @@ -67,6 +69,7 @@ Validator: - Input: artifact body + output contract. - Output: validation result or runtime validation error. +- Schema documents may be loaded from a directory, single file, or `fs.FS` root in the public package. CLI and HTTP continue to use directory-backed schema loading. ## Boundaries @@ -103,6 +106,7 @@ Strict decoding and input checks: - config/prompt/profile loaders reject unknown YAML fields. - prompt/profile repositories scan nested subdirectories recursively. - prompt/profile lookup uses YAML `id` values; subdirectory paths are organizational only. +- prompt `content_file` paths resolve relative to the prompt YAML file within the same source. - duplicate prompt/profile IDs are invalid and fail instead of using first-match behavior. - duplicate profile IDs across custom and built-in sources are allowed; the custom source overrides the built-in profile. - HTTP DTO decoder rejects unknown JSON fields. diff --git a/engine.go b/engine.go index 7e073b0..1d00c72 100644 --- a/engine.go +++ b/engine.go @@ -4,13 +4,17 @@ import ( "context" "errors" "fmt" + "io/fs" "net/http" + "os" + "path/filepath" "strings" "time" artifactadapter "gitea.maximumdirect.net/eric/scriptorium/internal/artifact" "gitea.maximumdirect.net/eric/scriptorium/internal/defaults" "gitea.maximumdirect.net/eric/scriptorium/internal/llm" + "gitea.maximumdirect.net/eric/scriptorium/internal/profile" "gitea.maximumdirect.net/eric/scriptorium/internal/profile/builtin" "gitea.maximumdirect.net/eric/scriptorium/internal/prompt" "gitea.maximumdirect.net/eric/scriptorium/internal/promptdef" @@ -51,7 +55,13 @@ type Config struct { type Option func(*engineOptions) error type engineOptions struct { - llmClient llm.Client + llmClient llm.Client + promptDefs promptdef.Repository + profiles profile.Repository + validator validate.Validator + promptSource bool + profileSource bool + validatorSource bool } // WithLLMClient injects a custom LLM client for execution. @@ -65,13 +75,87 @@ func WithLLMClient(client LLMClient) Option { } } +func WithPromptFS(fsys fs.FS, root string) Option { + return func(options *engineOptions) error { + if fsys == nil { + return ErrInvalidConfig + } + if strings.TrimSpace(root) == "" { + return ErrInvalidConfig + } + options.promptDefs = promptdef.NewFSRepository(fsys, root) + options.promptSource = true + return nil + } +} + +func WithPromptFile(path string) Option { + return func(options *engineOptions) error { + fsys, root, err := fileSource(path) + if err != nil { + return err + } + options.promptDefs = promptdef.NewFSRepository(fsys, root) + options.promptSource = true + return nil + } +} + +func WithProfileFS(fsys fs.FS, root string) Option { + return func(options *engineOptions) error { + if fsys == nil { + return ErrInvalidConfig + } + if strings.TrimSpace(root) == "" { + return ErrInvalidConfig + } + options.profiles = profile.NewFSRepository(fsys, root) + options.profileSource = true + return nil + } +} + +func WithProfileFile(path string) Option { + return func(options *engineOptions) error { + fsys, root, err := fileSource(path) + if err != nil { + return err + } + options.profiles = profile.NewFSRepository(fsys, root) + options.profileSource = true + return nil + } +} + +func WithSchemaFS(fsys fs.FS, root string) Option { + return func(options *engineOptions) error { + if fsys == nil { + return ErrInvalidConfig + } + if strings.TrimSpace(root) == "" { + return ErrInvalidConfig + } + options.validator = validate.NewFSValidator(fsys, root) + options.validatorSource = true + return nil + } +} + +func WithSchemaFile(path string) Option { + return func(options *engineOptions) error { + fsys, root, err := fileSource(path) + if err != nil { + return err + } + options.validator = validate.NewFSValidator(fsys, root) + options.validatorSource = true + return nil + } +} + // NewEngine constructs an Engine using the same default internal components as // the CLI and HTTP adapters. func NewEngine(cfg Config, opts ...Option) (*Engine, error) { - if strings.TrimSpace(cfg.PromptDir) == "" { - return nil, fmt.Errorf("%w: prompt directory is required", ErrInvalidConfig) - } - var options engineOptions for _, opt := range opts { if opt == nil { @@ -82,9 +166,26 @@ func NewEngine(cfg Config, opts ...Option) (*Engine, error) { } } - schemaDir := cfg.SchemaDir - if strings.TrimSpace(schemaDir) == "" { - schemaDir = defaults.SchemaDirDefault + promptDefs := options.promptDefs + if !options.promptSource { + if strings.TrimSpace(cfg.PromptDir) == "" { + return nil, fmt.Errorf("%w: prompt directory is required", ErrInvalidConfig) + } + promptDefs = promptdef.NewFilesystemRepository(cfg.PromptDir) + } + + profiles := builtin.NewRepositoryWithDirectory(cfg.ProfileDir) + if options.profileSource { + profiles = builtin.NewRepositoryWithPrimary(options.profiles) + } + + validator := options.validator + if !options.validatorSource { + schemaDir := cfg.SchemaDir + if strings.TrimSpace(schemaDir) == "" { + schemaDir = defaults.SchemaDirDefault + } + validator = validate.NewStandardValidator(schemaDir) } llmClient := options.llmClient @@ -101,16 +202,36 @@ func NewEngine(cfg Config, opts ...Option) (*Engine, error) { return &Engine{ runner: usecase.NewRunner( - promptdef.NewFilesystemRepository(cfg.PromptDir), - builtin.NewRepositoryWithDirectory(cfg.ProfileDir), + promptDefs, + profiles, artifactadapter.NewCompositeReader(), prompt.NewGoRenderer(), llmClient, - validate.NewStandardValidator(schemaDir), + validator, ), }, nil } +func fileSource(name string) (fs.FS, string, error) { + cleanName := strings.TrimSpace(name) + if cleanName == "" { + return nil, "", ErrInvalidConfig + } + dir := filepath.Dir(cleanName) + base := filepath.Base(cleanName) + if base == "." || base == string(filepath.Separator) || strings.TrimSpace(base) == "" { + return nil, "", ErrInvalidConfig + } + info, err := os.Stat(cleanName) + if err != nil { + return nil, "", fmt.Errorf("%w: failed to access source file %q: %v", ErrInvalidConfig, cleanName, err) + } + if info.IsDir() { + return nil, "", fmt.Errorf("%w: source path %q must be a file", ErrInvalidConfig, cleanName) + } + return os.DirFS(dir), filepath.ToSlash(base), nil +} + // Prepare resolves a prompt request without calling an LLM. func (e *Engine) Prepare(ctx context.Context, req RunRequest) (*PreparedRun, error) { if e == nil || e.runner == nil { diff --git a/engine_test.go b/engine_test.go index fe5eba3..8fc52c5 100644 --- a/engine_test.go +++ b/engine_test.go @@ -11,6 +11,7 @@ import ( "reflect" "strings" "testing" + "testing/fstest" "gitea.maximumdirect.net/eric/scriptorium" ) @@ -709,6 +710,256 @@ unexpected: true } } +func TestPrepareWorksWithPromptFSAndRelativeContentFile(t *testing.T) { + promptFS := fstest.MapFS{ + "assets/prompts/fs-summary.yaml": &fstest.MapFile{Data: []byte(` +id: fs.summary +version: "1.0.0" +default_profile: local-fast +inputs: + - name: transcript + required: true +messages: + - role: user + content_file: ./messages/summary.tmpl +output: + format: text + validation_mode: none + repair_attempts: 0 +`)}, + "assets/prompts/messages/summary.tmpl": &fstest.MapFile{Data: []byte(`Summarize {{input "transcript"}} from prompt fs.`)}, + } + + engine, err := scriptorium.NewEngine(scriptorium.Config{ + PromptDir: t.TempDir(), + ProfileDir: "./examples/profiles", + SchemaDir: "./examples/schemas", + }, scriptorium.WithPromptFS(promptFS, "assets/prompts")) + if err != nil { + t.Fatalf("expected engine construction to succeed, got %v", err) + } + + prepared, err := engine.Prepare(context.Background(), scriptorium.RunRequest{ + PromptID: "fs.summary", + Inputs: map[string]scriptorium.ArtifactRef{ + "transcript": scriptorium.Inline("Rin opens the gate."), + }, + }) + if err != nil { + t.Fatalf("expected prepare to succeed, got %v", err) + } + if len(prepared.Messages) != 1 || !strings.Contains(prepared.Messages[0].Content, "prompt fs") { + t.Fatalf("expected content_file body from prompt fs, got %+v", prepared.Messages) + } +} + +func TestPrepareWorksWithPromptFile(t *testing.T) { + promptDir := t.TempDir() + promptPath := filepath.Join(promptDir, "single.yaml") + if err := os.WriteFile(promptPath, []byte(` +id: single.file.prompt +version: "1.0.0" +default_profile: local-fast +inputs: + - name: transcript + required: true +messages: + - role: user + content: "Summarize {{input \"transcript\"}} from file." +output: + format: text + validation_mode: none + repair_attempts: 0 +`), 0o644); err != nil { + t.Fatal(err) + } + + engine, err := scriptorium.NewEngine(scriptorium.Config{ + ProfileDir: "./examples/profiles", + SchemaDir: "./examples/schemas", + }, scriptorium.WithPromptFile(promptPath)) + if err != nil { + t.Fatalf("expected engine construction to succeed, got %v", err) + } + + prepared, err := engine.Prepare(context.Background(), scriptorium.RunRequest{ + PromptID: "single.file.prompt", + Inputs: map[string]scriptorium.ArtifactRef{ + "transcript": scriptorium.Inline("Rin opens the gate."), + }, + }) + if err != nil { + t.Fatalf("expected prepare to succeed, got %v", err) + } + if prepared.PromptID != "single.file.prompt" { + t.Fatalf("unexpected prompt id: %q", prepared.PromptID) + } +} + +func TestPrepareWorksWithProfileFSOverBuiltIns(t *testing.T) { + profileFS := fstest.MapFS{ + "profiles/mistral-small-3.yaml": &fstest.MapFile{Data: []byte(` +id: mistral-small-3 +endpoint: http://profile-fs/v1 +model: profile-fs-model +`)}, + } + + engine, err := scriptorium.NewEngine(scriptorium.Config{ + PromptDir: "./examples/prompts", + SchemaDir: "./examples/schemas", + }, scriptorium.WithProfileFS(profileFS, "profiles")) + if err != nil { + t.Fatalf("expected engine construction to succeed, got %v", err) + } + + prepared, err := engine.Prepare(context.Background(), scriptorium.RunRequest{ + PromptID: "generic.markdown_summary", + ProfileID: "mistral-small-3", + Inputs: map[string]scriptorium.ArtifactRef{ + "transcript": scriptorium.Inline("Rin opens the gate."), + "glossary": scriptorium.Inline("gate: A guarded passage."), + }, + }) + if err != nil { + t.Fatalf("expected prepare to succeed, got %v", err) + } + if prepared.EffectiveModelParams.Model != "profile-fs-model" { + t.Fatalf("expected profile fs to override built-in, got %q", prepared.EffectiveModelParams.Model) + } +} + +func TestPrepareWorksWithProfileFileOverBuiltIns(t *testing.T) { + profileDir := t.TempDir() + profilePath := filepath.Join(profileDir, "mistral-small-3.yaml") + if err := os.WriteFile(profilePath, []byte(` +id: mistral-small-3 +endpoint: http://profile-file/v1 +model: profile-file-model +`), 0o644); err != nil { + t.Fatal(err) + } + + engine, err := scriptorium.NewEngine(scriptorium.Config{ + PromptDir: "./examples/prompts", + SchemaDir: "./examples/schemas", + }, scriptorium.WithProfileFile(profilePath)) + if err != nil { + t.Fatalf("expected engine construction to succeed, got %v", err) + } + + prepared, err := engine.Prepare(context.Background(), scriptorium.RunRequest{ + PromptID: "generic.markdown_summary", + ProfileID: "mistral-small-3", + Inputs: map[string]scriptorium.ArtifactRef{ + "transcript": scriptorium.Inline("Rin opens the gate."), + "glossary": scriptorium.Inline("gate: A guarded passage."), + }, + }) + if err != nil { + t.Fatalf("expected prepare to succeed, got %v", err) + } + if prepared.EffectiveModelParams.Model != "profile-file-model" { + t.Fatalf("expected profile file to override built-in, got %q", prepared.EffectiveModelParams.Model) + } +} + +func TestRunStructuredOutputWorksWithSchemaFS(t *testing.T) { + fake := &fakeLLMClient{response: &scriptorium.GenerateResponse{Content: `{"events":[]}`}} + engine, err := scriptorium.NewEngine(scriptorium.Config{ + PromptDir: t.TempDir(), + ProfileDir: "./examples/profiles", + SchemaDir: t.TempDir(), + }, + scriptorium.WithPromptFS(publicStructuredPromptFS("schema.fs.prompt", "events.schema.json"), "prompts"), + scriptorium.WithSchemaFS(publicSchemaFS(), "schemas"), + scriptorium.WithLLMClient(fake), + ) + if err != nil { + t.Fatalf("expected engine construction to succeed, got %v", err) + } + + result, err := engine.Run(context.Background(), scriptorium.RunRequest{ + PromptID: "schema.fs.prompt", + Inputs: map[string]scriptorium.ArtifactRef{ + "transcript": scriptorium.Inline("Rin opens the gate."), + }, + }) + if err != nil { + t.Fatalf("expected run to succeed, got %v", err) + } + if result.Validation.Status != scriptorium.ValidationPassed || !result.Validation.IsValid { + t.Fatalf("expected schema validation to pass, got %+v", result.Validation) + } + if len(fake.requests) != 1 || fake.requests[0].StructuredOutput == nil { + t.Fatalf("expected structured output request, got %+v", fake.requests) + } +} + +func TestRunStructuredOutputWorksWithSchemaFile(t *testing.T) { + schemaDir := t.TempDir() + schemaPath := filepath.Join(schemaDir, "events.schema.json") + if err := os.WriteFile(schemaPath, []byte(publicSchemaJSON()), 0o644); err != nil { + t.Fatal(err) + } + + fake := &fakeLLMClient{response: &scriptorium.GenerateResponse{Content: `{"events":[]}`}} + engine, err := scriptorium.NewEngine(scriptorium.Config{ + ProfileDir: "./examples/profiles", + }, + scriptorium.WithPromptFS(publicStructuredPromptFS("schema.file.prompt", "events.schema.json"), "prompts"), + scriptorium.WithSchemaFile(schemaPath), + scriptorium.WithLLMClient(fake), + ) + if err != nil { + t.Fatalf("expected engine construction to succeed, got %v", err) + } + + result, err := engine.Run(context.Background(), scriptorium.RunRequest{ + PromptID: "schema.file.prompt", + Inputs: map[string]scriptorium.ArtifactRef{ + "transcript": scriptorium.Inline("Rin opens the gate."), + }, + }) + if err != nil { + t.Fatalf("expected run to succeed, got %v", err) + } + if result.Validation.Status != scriptorium.ValidationPassed || !result.Validation.IsValid { + t.Fatalf("expected schema validation to pass, got %+v", result.Validation) + } +} + +func TestSourceOptionsRejectInvalidInputs(t *testing.T) { + missingFile := filepath.Join(t.TempDir(), "missing.yaml") + directoryPath := t.TempDir() + + tests := []struct { + name string + opt scriptorium.Option + }{ + {name: "prompt fs nil", opt: scriptorium.WithPromptFS(nil, "prompts")}, + {name: "prompt fs empty root", opt: scriptorium.WithPromptFS(fstest.MapFS{}, "")}, + {name: "prompt file empty", opt: scriptorium.WithPromptFile("")}, + {name: "prompt file missing", opt: scriptorium.WithPromptFile(missingFile)}, + {name: "prompt file directory", opt: scriptorium.WithPromptFile(directoryPath)}, + {name: "profile fs nil", opt: scriptorium.WithProfileFS(nil, "profiles")}, + {name: "profile fs empty root", opt: scriptorium.WithProfileFS(fstest.MapFS{}, "")}, + {name: "profile file empty", opt: scriptorium.WithProfileFile("")}, + {name: "schema fs nil", opt: scriptorium.WithSchemaFS(nil, "schemas")}, + {name: "schema fs empty root", opt: scriptorium.WithSchemaFS(fstest.MapFS{}, "")}, + {name: "schema file empty", opt: scriptorium.WithSchemaFile("")}, + } + + for _, tc := range tests { + t.Run(tc.name, func(t *testing.T) { + _, err := scriptorium.NewEngine(scriptorium.Config{PromptDir: "./examples/prompts"}, tc.opt) + if !errors.Is(err, scriptorium.ErrInvalidConfig) { + t.Fatalf("expected ErrInvalidConfig, got %v", err) + } + }) + } +} + func TestExtraParamsTypedNestedValuesAreCopiedAcrossPublicBoundary(t *testing.T) { fake := &fakeLLMClient{response: &scriptorium.GenerateResponse{Content: "ok"}} engine := newExampleEngineWithOptions(t, "./examples/schemas", scriptorium.WithLLMClient(fake)) @@ -862,6 +1113,43 @@ api_key_env: ` + apiKeyEnv + ` } } +func publicStructuredPromptFS(id string, schemaPath string) fstest.MapFS { + return fstest.MapFS{ + "prompts/prompt.yaml": &fstest.MapFile{Data: []byte(`id: ` + id + ` +version: "1.0.0" +default_profile: local-fast +inputs: + - name: transcript + required: true +messages: + - role: user + content: "Extract events from {{input \"transcript\"}}." +output: + format: json + validation_mode: json_schema + schema_path: ` + schemaPath + ` + repair_attempts: 0 +`)}, + } +} + +func publicSchemaFS() fstest.MapFS { + return fstest.MapFS{ + "schemas/events.schema.json": &fstest.MapFile{Data: []byte(publicSchemaJSON())}, + } +} + +func publicSchemaJSON() string { + return `{ + "$schema": "https://json-schema.org/draft/2020-12/schema", + "type": "object", + "required": ["events"], + "properties": { + "events": {"type": "array"} + } +}` +} + type fakeLLMClient struct { response *scriptorium.GenerateResponse err error diff --git a/internal/promptdef/filesystem_repository.go b/internal/promptdef/filesystem_repository.go index 737f90d..f4cf6e3 100644 --- a/internal/promptdef/filesystem_repository.go +++ b/internal/promptdef/filesystem_repository.go @@ -5,7 +5,9 @@ import ( "context" "errors" "fmt" + "io/fs" "os" + "path" "path/filepath" "strings" @@ -24,6 +26,11 @@ type filesystemRepository struct { dir string } +type fsRepository struct { + fsys fs.FS + root string +} + type promptDefinitionFile struct { ID string `yaml:"id"` Version string `yaml:"version"` @@ -65,6 +72,10 @@ func NewFilesystemRepository(dir string) Repository { return &filesystemRepository{dir: dir} } +func NewFSRepository(fsys fs.FS, root string) Repository { + return &fsRepository{fsys: fsys, root: root} +} + func (r *filesystemRepository) GetPromptDefinition(ctx context.Context, id string, version string) (*domain.PromptDefinition, error) { if strings.TrimSpace(id) == "" { return nil, fmt.Errorf("%w: prompt id is required", ErrInvalidPromptDefinition) @@ -132,6 +143,10 @@ func (r *filesystemRepository) GetPromptDefinition(ctx context.Context, id strin return nil, ErrPromptDefinitionNotFound } +func (r *fsRepository) GetPromptDefinition(ctx context.Context, id string, version string) (*domain.PromptDefinition, error) { + return loadPromptDefinition(ctx, r.fsys, r.root, id, version) +} + type promptDefinitionMatch struct { def *domain.PromptDefinition path string @@ -166,7 +181,187 @@ func promptDefinitionFileHasID(path string, id string) bool { return strings.TrimSpace(raw.ID) == id } +func loadPromptDefinition(ctx context.Context, fsys fs.FS, root string, id string, version string) (*domain.PromptDefinition, error) { + if strings.TrimSpace(id) == "" { + return nil, fmt.Errorf("%w: prompt id is required", ErrInvalidPromptDefinition) + } + if fsys == nil { + return nil, fmt.Errorf("failed to read prompt definition directory: filesystem is nil") + } + + files, err := findPromptDefinitionYAMLFiles(ctx, fsys, root) + if err != nil { + return nil, fmt.Errorf("failed to read prompt definition directory: %w", err) + } + + var matches []promptDefinitionMatch + for _, fullPath := range files { + select { + case <-ctx.Done(): + return nil, ctx.Err() + default: + } + + relPath := displayPath(root, fullPath) + fileMatch := filecatalog.Stem(path.Base(fullPath)) == id + data, err := fs.ReadFile(fsys, fullPath) + if err != nil { + if fileMatch { + return nil, fmt.Errorf("%w: %s: failed to read prompt definition file: %v", ErrInvalidYAML, relPath, err) + } + continue + } + + raw, err := decodePromptDefinition(data) + if err != nil { + if fileMatch || promptDefinitionDataHasID(data, id) { + return nil, fmt.Errorf("%w: %s: %v", ErrInvalidYAML, relPath, err) + } + continue + } + + def, err := normalizePromptDefinitionFromFS(raw, fsys, fullPath) + if err != nil { + if fileMatch || strings.TrimSpace(raw.ID) == id { + return nil, fmt.Errorf("%w: %s: %v", ErrInvalidPromptDefinition, relPath, err) + } + continue + } + + if def.ID != id { + continue + } + if version != "" && def.Version != version { + continue + } + matches = append(matches, promptDefinitionMatch{ + def: def, + path: relPath, + }) + } + + if len(matches) > 1 { + paths := make([]string, 0, len(matches)) + for _, match := range matches { + paths = append(paths, match.path) + } + if version != "" { + return nil, fmt.Errorf("%w: duplicate prompt definition id %q version %q found in: %s", ErrInvalidPromptDefinition, id, version, strings.Join(paths, ", ")) + } + return nil, fmt.Errorf("%w: duplicate prompt definition id %q found in: %s", ErrInvalidPromptDefinition, id, strings.Join(paths, ", ")) + } + + if len(matches) == 1 { + return matches[0].def, nil + } + + return nil, ErrPromptDefinitionNotFound +} + +func findPromptDefinitionYAMLFiles(ctx context.Context, fsys fs.FS, root string) ([]string, error) { + cleanRoot := cleanFSRoot(root) + var files []string + err := fs.WalkDir(fsys, cleanRoot, func(name string, d fs.DirEntry, err error) error { + if err != nil { + return err + } + select { + case <-ctx.Done(): + return ctx.Err() + default: + } + if d.IsDir() { + return nil + } + if !isPromptDefinitionYAMLFile(d.Name()) { + return nil + } + files = append(files, name) + return nil + }) + return files, err +} + +func cleanFSRoot(root string) string { + root = strings.TrimSpace(root) + if root == "" || root == "." { + return "." + } + return path.Clean(root) +} + +func displayPath(root string, name string) string { + cleanRoot := cleanFSRoot(root) + cleanName := path.Clean(name) + if cleanRoot == "." { + return cleanName + } + prefix := strings.TrimSuffix(cleanRoot, "/") + "/" + if strings.HasPrefix(cleanName, prefix) { + return strings.TrimPrefix(cleanName, prefix) + } + return cleanName +} + +func isPromptDefinitionYAMLFile(name string) bool { + return strings.HasSuffix(name, ".yaml") || strings.HasSuffix(name, ".yml") +} + +func decodePromptDefinition(data []byte) (*promptDefinitionFile, error) { + var raw promptDefinitionFile + decoder := yaml.NewDecoder(bytes.NewReader(data)) + decoder.KnownFields(true) + if err := decoder.Decode(&raw); err != nil { + return nil, err + } + return &raw, nil +} + +func promptDefinitionDataHasID(data []byte, id string) bool { + var raw struct { + ID string `yaml:"id"` + } + if err := yaml.NewDecoder(bytes.NewReader(data)).Decode(&raw); err != nil { + return false + } + return strings.TrimSpace(raw.ID) == id +} + func normalizePromptDefinition(raw *promptDefinitionFile, sourcePath string) (*domain.PromptDefinition, error) { + promptDir := filepath.Dir(sourcePath) + return normalizePromptDefinitionWithContent(raw, func(contentFile string) (string, string, error) { + resolvedPath := strings.TrimSpace(contentFile) + if !filepath.IsAbs(resolvedPath) { + resolvedPath = filepath.Join(promptDir, resolvedPath) + } + resolvedPath = filepath.Clean(resolvedPath) + + body, err := os.ReadFile(resolvedPath) + if err != nil { + return "", "", err + } + return string(body), resolvedPath, nil + }) +} + +func normalizePromptDefinitionFromFS(raw *promptDefinitionFile, fsys fs.FS, sourcePath string) (*domain.PromptDefinition, error) { + promptDir := path.Dir(sourcePath) + return normalizePromptDefinitionWithContent(raw, func(contentFile string) (string, string, error) { + resolvedPath := strings.TrimSpace(contentFile) + if !path.IsAbs(resolvedPath) { + resolvedPath = path.Join(promptDir, resolvedPath) + } + resolvedPath = strings.TrimPrefix(path.Clean(resolvedPath), "/") + + body, err := fs.ReadFile(fsys, resolvedPath) + if err != nil { + return "", "", err + } + return string(body), resolvedPath, nil + }) +} + +func normalizePromptDefinitionWithContent(raw *promptDefinitionFile, readContentFile func(string) (string, string, error)) (*domain.PromptDefinition, error) { if raw == nil { return nil, errors.New("prompt definition is nil") } @@ -206,7 +401,6 @@ func normalizePromptDefinition(raw *promptDefinitionFile, sourcePath string) (*d } templates := make([]domain.PromptMessageTemplate, 0, len(raw.Messages)) - promptDir := filepath.Dir(sourcePath) for i, msg := range raw.Messages { role := strings.TrimSpace(msg.Role) if role == "" { @@ -227,17 +421,11 @@ func normalizePromptDefinition(raw *promptDefinitionFile, sourcePath string) (*d templateContent := msg.Content resolvedContentFile := "" if hasContentFile { - resolvedPath := strings.TrimSpace(msg.ContentFile) - if !filepath.IsAbs(resolvedPath) { - resolvedPath = filepath.Join(promptDir, resolvedPath) - } - resolvedPath = filepath.Clean(resolvedPath) - - body, err := os.ReadFile(resolvedPath) + body, resolvedPath, err := readContentFile(msg.ContentFile) if err != nil { return nil, fmt.Errorf("prompt %q message %d (%s): failed to read content_file %q: %w", id, i, role, msg.ContentFile, err) } - templateContent = string(body) + templateContent = body resolvedContentFile = resolvedPath } diff --git a/internal/promptdef/repository_test.go b/internal/promptdef/repository_test.go index b5899d7..9551104 100644 --- a/internal/promptdef/repository_test.go +++ b/internal/promptdef/repository_test.go @@ -8,6 +8,7 @@ import ( "path/filepath" "strings" "testing" + "testing/fstest" "gitea.maximumdirect.net/eric/scriptorium/internal/domain" ) @@ -324,6 +325,97 @@ output: }) } +func TestFSRepositoryGetPromptDefinition(t *testing.T) { + repo := NewFSRepository(fstest.MapFS{ + "prompts/nested/prompt.yaml": &fstest.MapFile{Data: []byte(` +id: fs-prompt +version: "1.0.0" +inputs: + - name: transcript + required: true +messages: + - role: user + content_file: ./messages/user.tmpl +output: + format: markdown + validation_mode: basic + repair_attempts: 0 +`)}, + "prompts/nested/messages/user.tmpl": &fstest.MapFile{Data: []byte(`Summarize {{input "transcript"}}.`)}, + }, "prompts") + + got, err := repo.GetPromptDefinition(context.Background(), "fs-prompt", "") + if err != nil { + t.Fatalf("expected no error, got %v", err) + } + if got.ID != "fs-prompt" { + t.Fatalf("unexpected prompt id: %q", got.ID) + } + if len(got.Templates) != 1 || !strings.Contains(got.Templates[0].Content, `{{input "transcript"}}`) { + t.Fatalf("expected content_file body to be loaded, got %+v", got.Templates) + } + if got.Templates[0].ContentFile != "prompts/nested/messages/user.tmpl" { + t.Fatalf("unexpected content file path: %q", got.Templates[0].ContentFile) + } +} + +func TestFSRepositoryRejectsDuplicatePromptIDs(t *testing.T) { + repo := NewFSRepository(fstest.MapFS{ + "one.yaml": &fstest.MapFile{Data: []byte(` +id: duplicate-fs-prompt +version: "1.0.0" +messages: + - role: user + content: First. +output: + format: text + validation_mode: none + repair_attempts: 0 +`)}, + "nested/two.yaml": &fstest.MapFile{Data: []byte(` +id: duplicate-fs-prompt +version: "1.0.0" +messages: + - role: user + content: Second. +output: + format: text + validation_mode: none + repair_attempts: 0 +`)}, + }, ".") + + _, err := repo.GetPromptDefinition(context.Background(), "duplicate-fs-prompt", "") + if !errors.Is(err, ErrInvalidPromptDefinition) { + t.Fatalf("expected ErrInvalidPromptDefinition, got %v", err) + } + if !strings.Contains(err.Error(), "one.yaml") || !strings.Contains(err.Error(), "nested/two.yaml") { + t.Fatalf("expected duplicate paths in error, got %v", err) + } +} + +func TestFSRepositoryRejectsUnknownYAMLFields(t *testing.T) { + repo := NewFSRepository(fstest.MapFS{ + "not_named_like_id.yaml": &fstest.MapFile{Data: []byte(` +id: strict-fs-prompt +version: "1.0.0" +unknown: true +messages: + - role: user + content: Invalid. +output: + format: text + validation_mode: none + repair_attempts: 0 +`)}, + }, ".") + + _, err := repo.GetPromptDefinition(context.Background(), "strict-fs-prompt", "") + if !errors.Is(err, ErrInvalidYAML) { + t.Fatalf("expected ErrInvalidYAML, got %v", err) + } +} + func assertCacheControl(t *testing.T, got *domain.CacheControl, wantType domain.CacheControlType, wantTTL string) { t.Helper() if got == nil { diff --git a/internal/validate/standard_validator.go b/internal/validate/standard_validator.go index 2cfeb22..8ad6e2a 100644 --- a/internal/validate/standard_validator.go +++ b/internal/validate/standard_validator.go @@ -5,7 +5,9 @@ import ( "encoding/json" "errors" "fmt" + "io/fs" "os" + "path" "path/filepath" "strings" @@ -18,10 +20,19 @@ type StandardValidator struct { schemaBaseDir string } +type FSValidator struct { + fsys fs.FS + root string +} + func NewStandardValidator(schemaBaseDir string) Validator { return &StandardValidator{schemaBaseDir: schemaBaseDir} } +func NewFSValidator(fsys fs.FS, root string) Validator { + return &FSValidator{fsys: fsys, root: root} +} + func (v *StandardValidator) Validate(ctx context.Context, artifact *domain.Artifact, contract domain.OutputContract) (domain.ValidationResult, error) { select { case <-ctx.Done(): @@ -100,6 +111,88 @@ func (v *StandardValidator) Validate(ctx context.Context, artifact *domain.Artif } } +func (v *FSValidator) Validate(ctx context.Context, artifact *domain.Artifact, contract domain.OutputContract) (domain.ValidationResult, error) { + select { + case <-ctx.Done(): + return domain.ValidationResult{}, ctx.Err() + default: + } + + res := domain.ValidationResult{ + Mode: contract.ValidationMode, + SchemaPath: contract.SchemaPath, + RepairAttempts: contract.RepairAttempts, + } + + if artifact == nil { + return domain.ValidationResult{}, errors.New("artifact is required for validation") + } + + switch contract.ValidationMode { + case domain.ValidationNone: + res.Status = domain.ValidationSkipped + res.IsValid = true + return res, nil + case domain.ValidationBasic: + if strings.TrimSpace(string(artifact.Body)) == "" { + res.Status = domain.ValidationFailed + res.IsValid = false + res.Errors = []string{"output is empty"} + return res, nil + } + res.Status = domain.ValidationPassed + res.IsValid = true + return res, nil + case domain.ValidationJSON: + _, jsonErr := parseJSON(artifact.Body) + if jsonErr != nil { + res.Status = domain.ValidationFailed + res.IsValid = false + res.Errors = []string{fmt.Sprintf("invalid JSON: %v", jsonErr)} + return res, nil + } + res.Status = domain.ValidationPassed + res.IsValid = true + return res, nil + case domain.ValidationJSONSchema: + instance, jsonErr := parseJSON(artifact.Body) + if jsonErr != nil { + res.Status = domain.ValidationFailed + res.IsValid = false + res.Errors = []string{fmt.Sprintf("invalid JSON: %v", jsonErr)} + return res, nil + } + + schemaName, schemaDoc, err := v.loadSchemaDocument(contract.SchemaPath) + if err != nil { + return domain.ValidationResult{}, err + } + + resourceURL := fsSchemaResourceURL(schemaName) + compiler := jsonschema.NewCompiler() + if err := compiler.AddResource(resourceURL, schemaDoc); err != nil { + return domain.ValidationResult{}, fmt.Errorf("failed to register JSON schema %q: %w", schemaName, err) + } + schema, err := compiler.Compile(resourceURL) + if err != nil { + return domain.ValidationResult{}, fmt.Errorf("failed to compile JSON schema %q: %w", schemaName, err) + } + + if err := schema.Validate(instance); err != nil { + res.Status = domain.ValidationFailed + res.IsValid = false + res.Errors = []string{fmt.Sprintf("json schema validation failed: %v", err)} + return res, nil + } + + res.Status = domain.ValidationPassed + res.IsValid = true + return res, nil + default: + return domain.ValidationResult{}, fmt.Errorf("unsupported validation mode: %q", contract.ValidationMode) + } +} + func parseJSON(body []byte) (any, error) { var v any if err := json.Unmarshal(body, &v); err != nil { @@ -132,6 +225,20 @@ func (v *StandardValidator) LoadSchemaDocument(ctx context.Context, schemaPath s return doc, nil } +func (v *FSValidator) LoadSchemaDocument(ctx context.Context, schemaPath string) (any, error) { + select { + case <-ctx.Done(): + return nil, ctx.Err() + default: + } + + _, doc, err := v.loadSchemaDocument(schemaPath) + if err != nil { + return nil, err + } + return doc, nil +} + func (v *StandardValidator) resolveSchemaPath(schemaPath string) (string, error) { if strings.TrimSpace(schemaPath) == "" { return "", errors.New("schema path is required for json_schema validation") @@ -149,3 +256,70 @@ func (v *StandardValidator) resolveSchemaPath(schemaPath string) (string, error) return resolved, nil } + +func (v *FSValidator) loadSchemaDocument(schemaPath string) (string, any, error) { + resolved, err := v.resolveSchemaPath(schemaPath) + if err != nil { + return "", nil, err + } + + raw, err := fs.ReadFile(v.fsys, resolved) + if err != nil { + return "", nil, fmt.Errorf("failed to read schema file %q: %w", resolved, err) + } + + var doc any + if err := json.Unmarshal(raw, &doc); err != nil { + return "", nil, fmt.Errorf("failed to decode JSON schema %q: %w", resolved, err) + } + return resolved, doc, nil +} + +func (v *FSValidator) resolveSchemaPath(schemaPath string) (string, error) { + if strings.TrimSpace(schemaPath) == "" { + return "", errors.New("schema path is required for json_schema validation") + } + if v.fsys == nil { + return "", errors.New("schema filesystem is nil") + } + + cleanRoot := cleanFSRoot(v.root) + rootInfo, err := fs.Stat(v.fsys, cleanRoot) + if err != nil { + return "", fmt.Errorf("failed to access schema source %q: %w", cleanRoot, err) + } + + cleanSchemaPath := cleanSchemaFSPath(schemaPath) + var resolved string + if rootInfo.IsDir() { + resolved = path.Join(cleanRoot, cleanSchemaPath) + } else { + if cleanSchemaPath != path.Base(cleanRoot) { + return "", fmt.Errorf("schema path %q does not match schema file %q", cleanSchemaPath, path.Base(cleanRoot)) + } + resolved = cleanRoot + } + + if _, err := fs.Stat(v.fsys, resolved); err != nil { + return "", fmt.Errorf("failed to access schema file %q: %w", resolved, err) + } + return resolved, nil +} + +func cleanSchemaFSPath(schemaPath string) string { + cleaned := strings.TrimSpace(schemaPath) + cleaned = strings.TrimPrefix(path.Clean(cleaned), "/") + return cleaned +} + +func cleanFSRoot(root string) string { + root = strings.TrimSpace(root) + if root == "" || root == "." { + return "." + } + return strings.TrimPrefix(path.Clean(root), "/") +} + +func fsSchemaResourceURL(schemaName string) string { + return "scriptorium-schema:///" + strings.TrimPrefix(path.Clean(schemaName), "/") +} diff --git a/internal/validate/standard_validator_test.go b/internal/validate/standard_validator_test.go index 30d910b..b68a971 100644 --- a/internal/validate/standard_validator_test.go +++ b/internal/validate/standard_validator_test.go @@ -5,6 +5,7 @@ import ( "os" "path/filepath" "testing" + "testing/fstest" "gitea.maximumdirect.net/eric/scriptorium/internal/domain" ) @@ -250,3 +251,77 @@ func TestStandardValidatorLoadSchemaDocumentInvalidJSON(t *testing.T) { t.Fatal("expected decode error") } } + +func TestFSValidatorJSONSchemaSuccess(t *testing.T) { + v := NewFSValidator(fstest.MapFS{ + "schemas/events.schema.json": &fstest.MapFile{Data: []byte(`{ + "$schema": "https://json-schema.org/draft/2020-12/schema", + "type": "object", + "required": ["events"], + "properties": { + "events": {"type": "array"} + } +}`)}, + }, "schemas") + + res, err := v.Validate(context.Background(), &domain.Artifact{Body: []byte(`{"events":[]}`)}, domain.OutputContract{ + ValidationMode: domain.ValidationJSONSchema, + SchemaPath: "events.schema.json", + }) + if err != nil { + t.Fatalf("expected no error, got %v", err) + } + if res.Status != domain.ValidationPassed || !res.IsValid { + t.Fatalf("expected passed/valid, got status=%q valid=%v", res.Status, res.IsValid) + } +} + +func TestFSValidatorSingleSchemaFileUsesBaseName(t *testing.T) { + v := NewFSValidator(fstest.MapFS{ + "events.schema.json": &fstest.MapFile{Data: []byte(`{ + "type": "object", + "required": ["events"], + "properties": { + "events": {"type": "array"} + } +}`)}, + }, "events.schema.json") + + res, err := v.Validate(context.Background(), &domain.Artifact{Body: []byte(`{"events":[]}`)}, domain.OutputContract{ + ValidationMode: domain.ValidationJSONSchema, + SchemaPath: "events.schema.json", + }) + if err != nil { + t.Fatalf("expected no error, got %v", err) + } + if res.Status != domain.ValidationPassed || !res.IsValid { + t.Fatalf("expected passed/valid, got status=%q valid=%v", res.Status, res.IsValid) + } + + _, err = v.Validate(context.Background(), &domain.Artifact{Body: []byte(`{"events":[]}`)}, domain.OutputContract{ + ValidationMode: domain.ValidationJSONSchema, + SchemaPath: "other.schema.json", + }) + if err == nil { + t.Fatal("expected schema path mismatch error") + } +} + +func TestFSValidatorLoadSchemaDocument(t *testing.T) { + v := NewFSValidator(fstest.MapFS{ + "schemas/schema.json": &fstest.MapFile{Data: []byte(`{"type":"object"}`)}, + }, "schemas") + loader, ok := v.(SchemaDocumentLoader) + if !ok { + t.Fatal("fs validator must implement SchemaDocumentLoader") + } + + doc, err := loader.LoadSchemaDocument(context.Background(), "schema.json") + if err != nil { + t.Fatalf("expected no error, got %v", err) + } + obj, ok := doc.(map[string]any) + if !ok || obj["type"] != "object" { + t.Fatalf("unexpected schema document: %#v", doc) + } +}