Add public asset source options

This commit is contained in:
2026-07-04 17:02:09 +00:00
parent 3ad247039b
commit 6f91603168
8 changed files with 978 additions and 26 deletions

View File

@@ -21,7 +21,17 @@ if err != nil {
}
```
`PromptDir` is required. `ProfileDir` is optional; omit it to use built-in profiles only, or set it to overlay custom profiles above built-ins. `SchemaDir` defaults to the built-in schema directory. `Timeout` and `HTTPClient` configure the default OpenAI-compatible client used by `Run` when no custom LLM client is supplied.
`PromptDir` is required unless an explicit prompt source option is supplied. `ProfileDir` is optional; omit it to use built-in profiles only, or set it to overlay custom profiles above built-ins. `SchemaDir` defaults to the built-in schema directory. `Timeout` and `HTTPClient` configure the default OpenAI-compatible client used by `Run` when no custom LLM client is supplied.
## Asset Sources
Directory fields on `Config` remain the compatibility path. Explicit source options override the matching directory field:
- `WithPromptFS(fsys, root)` and `WithPromptFile(path)`
- `WithProfileFS(fsys, root)` and `WithProfileFile(path)`
- `WithSchemaFS(fsys, root)` and `WithSchemaFile(path)`
Prompt and profile sources load standard Scriptorium YAML with the same strict validation as directory sources. Prompt `content_file` paths resolve relative to the prompt file in the same source. Profile options overlay custom profiles above built-ins. Schema `fs.FS` sources preserve prompt `schema_path` semantics; schema file options expose the file by its base name.
## Prepare A Prompt

View File

@@ -9,13 +9,13 @@ This document describes implemented adapter/repository boundaries and their curr
- `internal/adapter/cli`: CLI command parsing, app wiring, stdout/stderr handling, exit codes.
- `internal/adapter/http`: HTTP request/response mapping for `POST /v1/runs`.
- root package `scriptorium`: public Go library facade for preparing and running prompt requests.
- `internal/promptdef`: filesystem prompt-definition repository.
- `internal/promptdef`: filesystem and `fs.FS` prompt-definition repositories.
- `internal/profile`: filesystem, `fs.FS`, and overlay execution-profile repositories.
- `internal/profile/builtin`: embedded built-in execution-profile repository.
- `internal/artifact`: input artifact reader.
- `internal/prompt`: Go-template renderer.
- `internal/llm`: OpenAI-compatible LLM client implementation.
- `internal/validate`: output validator.
- `internal/validate`: filesystem and `fs.FS` output validators.
- `internal/format`: prepared-run formatters for `render` output.
## Inputs And Outputs
@@ -38,18 +38,20 @@ Public library facade:
- Output: typed `PreparedRun` and `RunResult` values plus public sentinel errors.
- Custom LLM behavior is injected with `WithLLMClient`; otherwise the default OpenAI-compatible client is used.
- `RunRequest.APIKey` is a request-scoped Go value only; it is converted into internal execution state for LLM generation and stripped from public result types.
- Prompt, profile, and schema source options can use directories, single files, or `fs.FS` roots. Explicit source options override the matching `Config` directory field.
- Public types are facade types converted at the package boundary; internal domain types remain internal.
Filesystem repositories:
Prompt/profile repositories:
- Input: prompt/profile YAML files under configured directories.
- Input: prompt/profile YAML files under configured directories or `fs.FS` roots.
- Output: normalized domain definitions/profiles or typed errors.
- Single-file public sources are represented as `fs.FS` roots containing one YAML file; lookup still uses YAML `id` values.
Profile repository composition:
- Built-in profiles are embedded and loaded through the same profile validation rules as filesystem profiles.
- When no custom profile directory is configured, the runner receives the built-in profile repository.
- When a custom profile directory is configured, the runner receives an overlay repository with custom profiles as primary and built-ins as fallback.
- When a custom profile directory/file/`fs.FS` source is configured, the runner receives an overlay repository with custom profiles as primary and built-ins as fallback.
- Overlay lookup falls back only after custom profile-not-found errors; custom load/validation/raw-key errors are returned directly.
Artifact reader:
@@ -67,6 +69,7 @@ Validator:
- Input: artifact body + output contract.
- Output: validation result or runtime validation error.
- Schema documents may be loaded from a directory, single file, or `fs.FS` root in the public package. CLI and HTTP continue to use directory-backed schema loading.
## Boundaries
@@ -103,6 +106,7 @@ Strict decoding and input checks:
- config/prompt/profile loaders reject unknown YAML fields.
- prompt/profile repositories scan nested subdirectories recursively.
- prompt/profile lookup uses YAML `id` values; subdirectory paths are organizational only.
- prompt `content_file` paths resolve relative to the prompt YAML file within the same source.
- duplicate prompt/profile IDs are invalid and fail instead of using first-match behavior.
- duplicate profile IDs across custom and built-in sources are allowed; the custom source overrides the built-in profile.
- HTTP DTO decoder rejects unknown JSON fields.

143
engine.go
View File

@@ -4,13 +4,17 @@ import (
"context"
"errors"
"fmt"
"io/fs"
"net/http"
"os"
"path/filepath"
"strings"
"time"
artifactadapter "gitea.maximumdirect.net/eric/scriptorium/internal/artifact"
"gitea.maximumdirect.net/eric/scriptorium/internal/defaults"
"gitea.maximumdirect.net/eric/scriptorium/internal/llm"
"gitea.maximumdirect.net/eric/scriptorium/internal/profile"
"gitea.maximumdirect.net/eric/scriptorium/internal/profile/builtin"
"gitea.maximumdirect.net/eric/scriptorium/internal/prompt"
"gitea.maximumdirect.net/eric/scriptorium/internal/promptdef"
@@ -51,7 +55,13 @@ type Config struct {
type Option func(*engineOptions) error
type engineOptions struct {
llmClient llm.Client
llmClient llm.Client
promptDefs promptdef.Repository
profiles profile.Repository
validator validate.Validator
promptSource bool
profileSource bool
validatorSource bool
}
// WithLLMClient injects a custom LLM client for execution.
@@ -65,13 +75,87 @@ func WithLLMClient(client LLMClient) Option {
}
}
func WithPromptFS(fsys fs.FS, root string) Option {
return func(options *engineOptions) error {
if fsys == nil {
return ErrInvalidConfig
}
if strings.TrimSpace(root) == "" {
return ErrInvalidConfig
}
options.promptDefs = promptdef.NewFSRepository(fsys, root)
options.promptSource = true
return nil
}
}
func WithPromptFile(path string) Option {
return func(options *engineOptions) error {
fsys, root, err := fileSource(path)
if err != nil {
return err
}
options.promptDefs = promptdef.NewFSRepository(fsys, root)
options.promptSource = true
return nil
}
}
func WithProfileFS(fsys fs.FS, root string) Option {
return func(options *engineOptions) error {
if fsys == nil {
return ErrInvalidConfig
}
if strings.TrimSpace(root) == "" {
return ErrInvalidConfig
}
options.profiles = profile.NewFSRepository(fsys, root)
options.profileSource = true
return nil
}
}
func WithProfileFile(path string) Option {
return func(options *engineOptions) error {
fsys, root, err := fileSource(path)
if err != nil {
return err
}
options.profiles = profile.NewFSRepository(fsys, root)
options.profileSource = true
return nil
}
}
func WithSchemaFS(fsys fs.FS, root string) Option {
return func(options *engineOptions) error {
if fsys == nil {
return ErrInvalidConfig
}
if strings.TrimSpace(root) == "" {
return ErrInvalidConfig
}
options.validator = validate.NewFSValidator(fsys, root)
options.validatorSource = true
return nil
}
}
func WithSchemaFile(path string) Option {
return func(options *engineOptions) error {
fsys, root, err := fileSource(path)
if err != nil {
return err
}
options.validator = validate.NewFSValidator(fsys, root)
options.validatorSource = true
return nil
}
}
// NewEngine constructs an Engine using the same default internal components as
// the CLI and HTTP adapters.
func NewEngine(cfg Config, opts ...Option) (*Engine, error) {
if strings.TrimSpace(cfg.PromptDir) == "" {
return nil, fmt.Errorf("%w: prompt directory is required", ErrInvalidConfig)
}
var options engineOptions
for _, opt := range opts {
if opt == nil {
@@ -82,9 +166,26 @@ func NewEngine(cfg Config, opts ...Option) (*Engine, error) {
}
}
schemaDir := cfg.SchemaDir
if strings.TrimSpace(schemaDir) == "" {
schemaDir = defaults.SchemaDirDefault
promptDefs := options.promptDefs
if !options.promptSource {
if strings.TrimSpace(cfg.PromptDir) == "" {
return nil, fmt.Errorf("%w: prompt directory is required", ErrInvalidConfig)
}
promptDefs = promptdef.NewFilesystemRepository(cfg.PromptDir)
}
profiles := builtin.NewRepositoryWithDirectory(cfg.ProfileDir)
if options.profileSource {
profiles = builtin.NewRepositoryWithPrimary(options.profiles)
}
validator := options.validator
if !options.validatorSource {
schemaDir := cfg.SchemaDir
if strings.TrimSpace(schemaDir) == "" {
schemaDir = defaults.SchemaDirDefault
}
validator = validate.NewStandardValidator(schemaDir)
}
llmClient := options.llmClient
@@ -101,16 +202,36 @@ func NewEngine(cfg Config, opts ...Option) (*Engine, error) {
return &Engine{
runner: usecase.NewRunner(
promptdef.NewFilesystemRepository(cfg.PromptDir),
builtin.NewRepositoryWithDirectory(cfg.ProfileDir),
promptDefs,
profiles,
artifactadapter.NewCompositeReader(),
prompt.NewGoRenderer(),
llmClient,
validate.NewStandardValidator(schemaDir),
validator,
),
}, nil
}
func fileSource(name string) (fs.FS, string, error) {
cleanName := strings.TrimSpace(name)
if cleanName == "" {
return nil, "", ErrInvalidConfig
}
dir := filepath.Dir(cleanName)
base := filepath.Base(cleanName)
if base == "." || base == string(filepath.Separator) || strings.TrimSpace(base) == "" {
return nil, "", ErrInvalidConfig
}
info, err := os.Stat(cleanName)
if err != nil {
return nil, "", fmt.Errorf("%w: failed to access source file %q: %v", ErrInvalidConfig, cleanName, err)
}
if info.IsDir() {
return nil, "", fmt.Errorf("%w: source path %q must be a file", ErrInvalidConfig, cleanName)
}
return os.DirFS(dir), filepath.ToSlash(base), nil
}
// Prepare resolves a prompt request without calling an LLM.
func (e *Engine) Prepare(ctx context.Context, req RunRequest) (*PreparedRun, error) {
if e == nil || e.runner == nil {

View File

@@ -11,6 +11,7 @@ import (
"reflect"
"strings"
"testing"
"testing/fstest"
"gitea.maximumdirect.net/eric/scriptorium"
)
@@ -709,6 +710,256 @@ unexpected: true
}
}
func TestPrepareWorksWithPromptFSAndRelativeContentFile(t *testing.T) {
promptFS := fstest.MapFS{
"assets/prompts/fs-summary.yaml": &fstest.MapFile{Data: []byte(`
id: fs.summary
version: "1.0.0"
default_profile: local-fast
inputs:
- name: transcript
required: true
messages:
- role: user
content_file: ./messages/summary.tmpl
output:
format: text
validation_mode: none
repair_attempts: 0
`)},
"assets/prompts/messages/summary.tmpl": &fstest.MapFile{Data: []byte(`Summarize {{input "transcript"}} from prompt fs.`)},
}
engine, err := scriptorium.NewEngine(scriptorium.Config{
PromptDir: t.TempDir(),
ProfileDir: "./examples/profiles",
SchemaDir: "./examples/schemas",
}, scriptorium.WithPromptFS(promptFS, "assets/prompts"))
if err != nil {
t.Fatalf("expected engine construction to succeed, got %v", err)
}
prepared, err := engine.Prepare(context.Background(), scriptorium.RunRequest{
PromptID: "fs.summary",
Inputs: map[string]scriptorium.ArtifactRef{
"transcript": scriptorium.Inline("Rin opens the gate."),
},
})
if err != nil {
t.Fatalf("expected prepare to succeed, got %v", err)
}
if len(prepared.Messages) != 1 || !strings.Contains(prepared.Messages[0].Content, "prompt fs") {
t.Fatalf("expected content_file body from prompt fs, got %+v", prepared.Messages)
}
}
func TestPrepareWorksWithPromptFile(t *testing.T) {
promptDir := t.TempDir()
promptPath := filepath.Join(promptDir, "single.yaml")
if err := os.WriteFile(promptPath, []byte(`
id: single.file.prompt
version: "1.0.0"
default_profile: local-fast
inputs:
- name: transcript
required: true
messages:
- role: user
content: "Summarize {{input \"transcript\"}} from file."
output:
format: text
validation_mode: none
repair_attempts: 0
`), 0o644); err != nil {
t.Fatal(err)
}
engine, err := scriptorium.NewEngine(scriptorium.Config{
ProfileDir: "./examples/profiles",
SchemaDir: "./examples/schemas",
}, scriptorium.WithPromptFile(promptPath))
if err != nil {
t.Fatalf("expected engine construction to succeed, got %v", err)
}
prepared, err := engine.Prepare(context.Background(), scriptorium.RunRequest{
PromptID: "single.file.prompt",
Inputs: map[string]scriptorium.ArtifactRef{
"transcript": scriptorium.Inline("Rin opens the gate."),
},
})
if err != nil {
t.Fatalf("expected prepare to succeed, got %v", err)
}
if prepared.PromptID != "single.file.prompt" {
t.Fatalf("unexpected prompt id: %q", prepared.PromptID)
}
}
func TestPrepareWorksWithProfileFSOverBuiltIns(t *testing.T) {
profileFS := fstest.MapFS{
"profiles/mistral-small-3.yaml": &fstest.MapFile{Data: []byte(`
id: mistral-small-3
endpoint: http://profile-fs/v1
model: profile-fs-model
`)},
}
engine, err := scriptorium.NewEngine(scriptorium.Config{
PromptDir: "./examples/prompts",
SchemaDir: "./examples/schemas",
}, scriptorium.WithProfileFS(profileFS, "profiles"))
if err != nil {
t.Fatalf("expected engine construction to succeed, got %v", err)
}
prepared, err := engine.Prepare(context.Background(), scriptorium.RunRequest{
PromptID: "generic.markdown_summary",
ProfileID: "mistral-small-3",
Inputs: map[string]scriptorium.ArtifactRef{
"transcript": scriptorium.Inline("Rin opens the gate."),
"glossary": scriptorium.Inline("gate: A guarded passage."),
},
})
if err != nil {
t.Fatalf("expected prepare to succeed, got %v", err)
}
if prepared.EffectiveModelParams.Model != "profile-fs-model" {
t.Fatalf("expected profile fs to override built-in, got %q", prepared.EffectiveModelParams.Model)
}
}
func TestPrepareWorksWithProfileFileOverBuiltIns(t *testing.T) {
profileDir := t.TempDir()
profilePath := filepath.Join(profileDir, "mistral-small-3.yaml")
if err := os.WriteFile(profilePath, []byte(`
id: mistral-small-3
endpoint: http://profile-file/v1
model: profile-file-model
`), 0o644); err != nil {
t.Fatal(err)
}
engine, err := scriptorium.NewEngine(scriptorium.Config{
PromptDir: "./examples/prompts",
SchemaDir: "./examples/schemas",
}, scriptorium.WithProfileFile(profilePath))
if err != nil {
t.Fatalf("expected engine construction to succeed, got %v", err)
}
prepared, err := engine.Prepare(context.Background(), scriptorium.RunRequest{
PromptID: "generic.markdown_summary",
ProfileID: "mistral-small-3",
Inputs: map[string]scriptorium.ArtifactRef{
"transcript": scriptorium.Inline("Rin opens the gate."),
"glossary": scriptorium.Inline("gate: A guarded passage."),
},
})
if err != nil {
t.Fatalf("expected prepare to succeed, got %v", err)
}
if prepared.EffectiveModelParams.Model != "profile-file-model" {
t.Fatalf("expected profile file to override built-in, got %q", prepared.EffectiveModelParams.Model)
}
}
func TestRunStructuredOutputWorksWithSchemaFS(t *testing.T) {
fake := &fakeLLMClient{response: &scriptorium.GenerateResponse{Content: `{"events":[]}`}}
engine, err := scriptorium.NewEngine(scriptorium.Config{
PromptDir: t.TempDir(),
ProfileDir: "./examples/profiles",
SchemaDir: t.TempDir(),
},
scriptorium.WithPromptFS(publicStructuredPromptFS("schema.fs.prompt", "events.schema.json"), "prompts"),
scriptorium.WithSchemaFS(publicSchemaFS(), "schemas"),
scriptorium.WithLLMClient(fake),
)
if err != nil {
t.Fatalf("expected engine construction to succeed, got %v", err)
}
result, err := engine.Run(context.Background(), scriptorium.RunRequest{
PromptID: "schema.fs.prompt",
Inputs: map[string]scriptorium.ArtifactRef{
"transcript": scriptorium.Inline("Rin opens the gate."),
},
})
if err != nil {
t.Fatalf("expected run to succeed, got %v", err)
}
if result.Validation.Status != scriptorium.ValidationPassed || !result.Validation.IsValid {
t.Fatalf("expected schema validation to pass, got %+v", result.Validation)
}
if len(fake.requests) != 1 || fake.requests[0].StructuredOutput == nil {
t.Fatalf("expected structured output request, got %+v", fake.requests)
}
}
func TestRunStructuredOutputWorksWithSchemaFile(t *testing.T) {
schemaDir := t.TempDir()
schemaPath := filepath.Join(schemaDir, "events.schema.json")
if err := os.WriteFile(schemaPath, []byte(publicSchemaJSON()), 0o644); err != nil {
t.Fatal(err)
}
fake := &fakeLLMClient{response: &scriptorium.GenerateResponse{Content: `{"events":[]}`}}
engine, err := scriptorium.NewEngine(scriptorium.Config{
ProfileDir: "./examples/profiles",
},
scriptorium.WithPromptFS(publicStructuredPromptFS("schema.file.prompt", "events.schema.json"), "prompts"),
scriptorium.WithSchemaFile(schemaPath),
scriptorium.WithLLMClient(fake),
)
if err != nil {
t.Fatalf("expected engine construction to succeed, got %v", err)
}
result, err := engine.Run(context.Background(), scriptorium.RunRequest{
PromptID: "schema.file.prompt",
Inputs: map[string]scriptorium.ArtifactRef{
"transcript": scriptorium.Inline("Rin opens the gate."),
},
})
if err != nil {
t.Fatalf("expected run to succeed, got %v", err)
}
if result.Validation.Status != scriptorium.ValidationPassed || !result.Validation.IsValid {
t.Fatalf("expected schema validation to pass, got %+v", result.Validation)
}
}
func TestSourceOptionsRejectInvalidInputs(t *testing.T) {
missingFile := filepath.Join(t.TempDir(), "missing.yaml")
directoryPath := t.TempDir()
tests := []struct {
name string
opt scriptorium.Option
}{
{name: "prompt fs nil", opt: scriptorium.WithPromptFS(nil, "prompts")},
{name: "prompt fs empty root", opt: scriptorium.WithPromptFS(fstest.MapFS{}, "")},
{name: "prompt file empty", opt: scriptorium.WithPromptFile("")},
{name: "prompt file missing", opt: scriptorium.WithPromptFile(missingFile)},
{name: "prompt file directory", opt: scriptorium.WithPromptFile(directoryPath)},
{name: "profile fs nil", opt: scriptorium.WithProfileFS(nil, "profiles")},
{name: "profile fs empty root", opt: scriptorium.WithProfileFS(fstest.MapFS{}, "")},
{name: "profile file empty", opt: scriptorium.WithProfileFile("")},
{name: "schema fs nil", opt: scriptorium.WithSchemaFS(nil, "schemas")},
{name: "schema fs empty root", opt: scriptorium.WithSchemaFS(fstest.MapFS{}, "")},
{name: "schema file empty", opt: scriptorium.WithSchemaFile("")},
}
for _, tc := range tests {
t.Run(tc.name, func(t *testing.T) {
_, err := scriptorium.NewEngine(scriptorium.Config{PromptDir: "./examples/prompts"}, tc.opt)
if !errors.Is(err, scriptorium.ErrInvalidConfig) {
t.Fatalf("expected ErrInvalidConfig, got %v", err)
}
})
}
}
func TestExtraParamsTypedNestedValuesAreCopiedAcrossPublicBoundary(t *testing.T) {
fake := &fakeLLMClient{response: &scriptorium.GenerateResponse{Content: "ok"}}
engine := newExampleEngineWithOptions(t, "./examples/schemas", scriptorium.WithLLMClient(fake))
@@ -862,6 +1113,43 @@ api_key_env: ` + apiKeyEnv + `
}
}
func publicStructuredPromptFS(id string, schemaPath string) fstest.MapFS {
return fstest.MapFS{
"prompts/prompt.yaml": &fstest.MapFile{Data: []byte(`id: ` + id + `
version: "1.0.0"
default_profile: local-fast
inputs:
- name: transcript
required: true
messages:
- role: user
content: "Extract events from {{input \"transcript\"}}."
output:
format: json
validation_mode: json_schema
schema_path: ` + schemaPath + `
repair_attempts: 0
`)},
}
}
func publicSchemaFS() fstest.MapFS {
return fstest.MapFS{
"schemas/events.schema.json": &fstest.MapFile{Data: []byte(publicSchemaJSON())},
}
}
func publicSchemaJSON() string {
return `{
"$schema": "https://json-schema.org/draft/2020-12/schema",
"type": "object",
"required": ["events"],
"properties": {
"events": {"type": "array"}
}
}`
}
type fakeLLMClient struct {
response *scriptorium.GenerateResponse
err error

View File

@@ -5,7 +5,9 @@ import (
"context"
"errors"
"fmt"
"io/fs"
"os"
"path"
"path/filepath"
"strings"
@@ -24,6 +26,11 @@ type filesystemRepository struct {
dir string
}
type fsRepository struct {
fsys fs.FS
root string
}
type promptDefinitionFile struct {
ID string `yaml:"id"`
Version string `yaml:"version"`
@@ -65,6 +72,10 @@ func NewFilesystemRepository(dir string) Repository {
return &filesystemRepository{dir: dir}
}
func NewFSRepository(fsys fs.FS, root string) Repository {
return &fsRepository{fsys: fsys, root: root}
}
func (r *filesystemRepository) GetPromptDefinition(ctx context.Context, id string, version string) (*domain.PromptDefinition, error) {
if strings.TrimSpace(id) == "" {
return nil, fmt.Errorf("%w: prompt id is required", ErrInvalidPromptDefinition)
@@ -132,6 +143,10 @@ func (r *filesystemRepository) GetPromptDefinition(ctx context.Context, id strin
return nil, ErrPromptDefinitionNotFound
}
func (r *fsRepository) GetPromptDefinition(ctx context.Context, id string, version string) (*domain.PromptDefinition, error) {
return loadPromptDefinition(ctx, r.fsys, r.root, id, version)
}
type promptDefinitionMatch struct {
def *domain.PromptDefinition
path string
@@ -166,7 +181,187 @@ func promptDefinitionFileHasID(path string, id string) bool {
return strings.TrimSpace(raw.ID) == id
}
func loadPromptDefinition(ctx context.Context, fsys fs.FS, root string, id string, version string) (*domain.PromptDefinition, error) {
if strings.TrimSpace(id) == "" {
return nil, fmt.Errorf("%w: prompt id is required", ErrInvalidPromptDefinition)
}
if fsys == nil {
return nil, fmt.Errorf("failed to read prompt definition directory: filesystem is nil")
}
files, err := findPromptDefinitionYAMLFiles(ctx, fsys, root)
if err != nil {
return nil, fmt.Errorf("failed to read prompt definition directory: %w", err)
}
var matches []promptDefinitionMatch
for _, fullPath := range files {
select {
case <-ctx.Done():
return nil, ctx.Err()
default:
}
relPath := displayPath(root, fullPath)
fileMatch := filecatalog.Stem(path.Base(fullPath)) == id
data, err := fs.ReadFile(fsys, fullPath)
if err != nil {
if fileMatch {
return nil, fmt.Errorf("%w: %s: failed to read prompt definition file: %v", ErrInvalidYAML, relPath, err)
}
continue
}
raw, err := decodePromptDefinition(data)
if err != nil {
if fileMatch || promptDefinitionDataHasID(data, id) {
return nil, fmt.Errorf("%w: %s: %v", ErrInvalidYAML, relPath, err)
}
continue
}
def, err := normalizePromptDefinitionFromFS(raw, fsys, fullPath)
if err != nil {
if fileMatch || strings.TrimSpace(raw.ID) == id {
return nil, fmt.Errorf("%w: %s: %v", ErrInvalidPromptDefinition, relPath, err)
}
continue
}
if def.ID != id {
continue
}
if version != "" && def.Version != version {
continue
}
matches = append(matches, promptDefinitionMatch{
def: def,
path: relPath,
})
}
if len(matches) > 1 {
paths := make([]string, 0, len(matches))
for _, match := range matches {
paths = append(paths, match.path)
}
if version != "" {
return nil, fmt.Errorf("%w: duplicate prompt definition id %q version %q found in: %s", ErrInvalidPromptDefinition, id, version, strings.Join(paths, ", "))
}
return nil, fmt.Errorf("%w: duplicate prompt definition id %q found in: %s", ErrInvalidPromptDefinition, id, strings.Join(paths, ", "))
}
if len(matches) == 1 {
return matches[0].def, nil
}
return nil, ErrPromptDefinitionNotFound
}
func findPromptDefinitionYAMLFiles(ctx context.Context, fsys fs.FS, root string) ([]string, error) {
cleanRoot := cleanFSRoot(root)
var files []string
err := fs.WalkDir(fsys, cleanRoot, func(name string, d fs.DirEntry, err error) error {
if err != nil {
return err
}
select {
case <-ctx.Done():
return ctx.Err()
default:
}
if d.IsDir() {
return nil
}
if !isPromptDefinitionYAMLFile(d.Name()) {
return nil
}
files = append(files, name)
return nil
})
return files, err
}
func cleanFSRoot(root string) string {
root = strings.TrimSpace(root)
if root == "" || root == "." {
return "."
}
return path.Clean(root)
}
func displayPath(root string, name string) string {
cleanRoot := cleanFSRoot(root)
cleanName := path.Clean(name)
if cleanRoot == "." {
return cleanName
}
prefix := strings.TrimSuffix(cleanRoot, "/") + "/"
if strings.HasPrefix(cleanName, prefix) {
return strings.TrimPrefix(cleanName, prefix)
}
return cleanName
}
func isPromptDefinitionYAMLFile(name string) bool {
return strings.HasSuffix(name, ".yaml") || strings.HasSuffix(name, ".yml")
}
func decodePromptDefinition(data []byte) (*promptDefinitionFile, error) {
var raw promptDefinitionFile
decoder := yaml.NewDecoder(bytes.NewReader(data))
decoder.KnownFields(true)
if err := decoder.Decode(&raw); err != nil {
return nil, err
}
return &raw, nil
}
func promptDefinitionDataHasID(data []byte, id string) bool {
var raw struct {
ID string `yaml:"id"`
}
if err := yaml.NewDecoder(bytes.NewReader(data)).Decode(&raw); err != nil {
return false
}
return strings.TrimSpace(raw.ID) == id
}
func normalizePromptDefinition(raw *promptDefinitionFile, sourcePath string) (*domain.PromptDefinition, error) {
promptDir := filepath.Dir(sourcePath)
return normalizePromptDefinitionWithContent(raw, func(contentFile string) (string, string, error) {
resolvedPath := strings.TrimSpace(contentFile)
if !filepath.IsAbs(resolvedPath) {
resolvedPath = filepath.Join(promptDir, resolvedPath)
}
resolvedPath = filepath.Clean(resolvedPath)
body, err := os.ReadFile(resolvedPath)
if err != nil {
return "", "", err
}
return string(body), resolvedPath, nil
})
}
func normalizePromptDefinitionFromFS(raw *promptDefinitionFile, fsys fs.FS, sourcePath string) (*domain.PromptDefinition, error) {
promptDir := path.Dir(sourcePath)
return normalizePromptDefinitionWithContent(raw, func(contentFile string) (string, string, error) {
resolvedPath := strings.TrimSpace(contentFile)
if !path.IsAbs(resolvedPath) {
resolvedPath = path.Join(promptDir, resolvedPath)
}
resolvedPath = strings.TrimPrefix(path.Clean(resolvedPath), "/")
body, err := fs.ReadFile(fsys, resolvedPath)
if err != nil {
return "", "", err
}
return string(body), resolvedPath, nil
})
}
func normalizePromptDefinitionWithContent(raw *promptDefinitionFile, readContentFile func(string) (string, string, error)) (*domain.PromptDefinition, error) {
if raw == nil {
return nil, errors.New("prompt definition is nil")
}
@@ -206,7 +401,6 @@ func normalizePromptDefinition(raw *promptDefinitionFile, sourcePath string) (*d
}
templates := make([]domain.PromptMessageTemplate, 0, len(raw.Messages))
promptDir := filepath.Dir(sourcePath)
for i, msg := range raw.Messages {
role := strings.TrimSpace(msg.Role)
if role == "" {
@@ -227,17 +421,11 @@ func normalizePromptDefinition(raw *promptDefinitionFile, sourcePath string) (*d
templateContent := msg.Content
resolvedContentFile := ""
if hasContentFile {
resolvedPath := strings.TrimSpace(msg.ContentFile)
if !filepath.IsAbs(resolvedPath) {
resolvedPath = filepath.Join(promptDir, resolvedPath)
}
resolvedPath = filepath.Clean(resolvedPath)
body, err := os.ReadFile(resolvedPath)
body, resolvedPath, err := readContentFile(msg.ContentFile)
if err != nil {
return nil, fmt.Errorf("prompt %q message %d (%s): failed to read content_file %q: %w", id, i, role, msg.ContentFile, err)
}
templateContent = string(body)
templateContent = body
resolvedContentFile = resolvedPath
}

View File

@@ -8,6 +8,7 @@ import (
"path/filepath"
"strings"
"testing"
"testing/fstest"
"gitea.maximumdirect.net/eric/scriptorium/internal/domain"
)
@@ -324,6 +325,97 @@ output:
})
}
func TestFSRepositoryGetPromptDefinition(t *testing.T) {
repo := NewFSRepository(fstest.MapFS{
"prompts/nested/prompt.yaml": &fstest.MapFile{Data: []byte(`
id: fs-prompt
version: "1.0.0"
inputs:
- name: transcript
required: true
messages:
- role: user
content_file: ./messages/user.tmpl
output:
format: markdown
validation_mode: basic
repair_attempts: 0
`)},
"prompts/nested/messages/user.tmpl": &fstest.MapFile{Data: []byte(`Summarize {{input "transcript"}}.`)},
}, "prompts")
got, err := repo.GetPromptDefinition(context.Background(), "fs-prompt", "")
if err != nil {
t.Fatalf("expected no error, got %v", err)
}
if got.ID != "fs-prompt" {
t.Fatalf("unexpected prompt id: %q", got.ID)
}
if len(got.Templates) != 1 || !strings.Contains(got.Templates[0].Content, `{{input "transcript"}}`) {
t.Fatalf("expected content_file body to be loaded, got %+v", got.Templates)
}
if got.Templates[0].ContentFile != "prompts/nested/messages/user.tmpl" {
t.Fatalf("unexpected content file path: %q", got.Templates[0].ContentFile)
}
}
func TestFSRepositoryRejectsDuplicatePromptIDs(t *testing.T) {
repo := NewFSRepository(fstest.MapFS{
"one.yaml": &fstest.MapFile{Data: []byte(`
id: duplicate-fs-prompt
version: "1.0.0"
messages:
- role: user
content: First.
output:
format: text
validation_mode: none
repair_attempts: 0
`)},
"nested/two.yaml": &fstest.MapFile{Data: []byte(`
id: duplicate-fs-prompt
version: "1.0.0"
messages:
- role: user
content: Second.
output:
format: text
validation_mode: none
repair_attempts: 0
`)},
}, ".")
_, err := repo.GetPromptDefinition(context.Background(), "duplicate-fs-prompt", "")
if !errors.Is(err, ErrInvalidPromptDefinition) {
t.Fatalf("expected ErrInvalidPromptDefinition, got %v", err)
}
if !strings.Contains(err.Error(), "one.yaml") || !strings.Contains(err.Error(), "nested/two.yaml") {
t.Fatalf("expected duplicate paths in error, got %v", err)
}
}
func TestFSRepositoryRejectsUnknownYAMLFields(t *testing.T) {
repo := NewFSRepository(fstest.MapFS{
"not_named_like_id.yaml": &fstest.MapFile{Data: []byte(`
id: strict-fs-prompt
version: "1.0.0"
unknown: true
messages:
- role: user
content: Invalid.
output:
format: text
validation_mode: none
repair_attempts: 0
`)},
}, ".")
_, err := repo.GetPromptDefinition(context.Background(), "strict-fs-prompt", "")
if !errors.Is(err, ErrInvalidYAML) {
t.Fatalf("expected ErrInvalidYAML, got %v", err)
}
}
func assertCacheControl(t *testing.T, got *domain.CacheControl, wantType domain.CacheControlType, wantTTL string) {
t.Helper()
if got == nil {

View File

@@ -5,7 +5,9 @@ import (
"encoding/json"
"errors"
"fmt"
"io/fs"
"os"
"path"
"path/filepath"
"strings"
@@ -18,10 +20,19 @@ type StandardValidator struct {
schemaBaseDir string
}
type FSValidator struct {
fsys fs.FS
root string
}
func NewStandardValidator(schemaBaseDir string) Validator {
return &StandardValidator{schemaBaseDir: schemaBaseDir}
}
func NewFSValidator(fsys fs.FS, root string) Validator {
return &FSValidator{fsys: fsys, root: root}
}
func (v *StandardValidator) Validate(ctx context.Context, artifact *domain.Artifact, contract domain.OutputContract) (domain.ValidationResult, error) {
select {
case <-ctx.Done():
@@ -100,6 +111,88 @@ func (v *StandardValidator) Validate(ctx context.Context, artifact *domain.Artif
}
}
func (v *FSValidator) Validate(ctx context.Context, artifact *domain.Artifact, contract domain.OutputContract) (domain.ValidationResult, error) {
select {
case <-ctx.Done():
return domain.ValidationResult{}, ctx.Err()
default:
}
res := domain.ValidationResult{
Mode: contract.ValidationMode,
SchemaPath: contract.SchemaPath,
RepairAttempts: contract.RepairAttempts,
}
if artifact == nil {
return domain.ValidationResult{}, errors.New("artifact is required for validation")
}
switch contract.ValidationMode {
case domain.ValidationNone:
res.Status = domain.ValidationSkipped
res.IsValid = true
return res, nil
case domain.ValidationBasic:
if strings.TrimSpace(string(artifact.Body)) == "" {
res.Status = domain.ValidationFailed
res.IsValid = false
res.Errors = []string{"output is empty"}
return res, nil
}
res.Status = domain.ValidationPassed
res.IsValid = true
return res, nil
case domain.ValidationJSON:
_, jsonErr := parseJSON(artifact.Body)
if jsonErr != nil {
res.Status = domain.ValidationFailed
res.IsValid = false
res.Errors = []string{fmt.Sprintf("invalid JSON: %v", jsonErr)}
return res, nil
}
res.Status = domain.ValidationPassed
res.IsValid = true
return res, nil
case domain.ValidationJSONSchema:
instance, jsonErr := parseJSON(artifact.Body)
if jsonErr != nil {
res.Status = domain.ValidationFailed
res.IsValid = false
res.Errors = []string{fmt.Sprintf("invalid JSON: %v", jsonErr)}
return res, nil
}
schemaName, schemaDoc, err := v.loadSchemaDocument(contract.SchemaPath)
if err != nil {
return domain.ValidationResult{}, err
}
resourceURL := fsSchemaResourceURL(schemaName)
compiler := jsonschema.NewCompiler()
if err := compiler.AddResource(resourceURL, schemaDoc); err != nil {
return domain.ValidationResult{}, fmt.Errorf("failed to register JSON schema %q: %w", schemaName, err)
}
schema, err := compiler.Compile(resourceURL)
if err != nil {
return domain.ValidationResult{}, fmt.Errorf("failed to compile JSON schema %q: %w", schemaName, err)
}
if err := schema.Validate(instance); err != nil {
res.Status = domain.ValidationFailed
res.IsValid = false
res.Errors = []string{fmt.Sprintf("json schema validation failed: %v", err)}
return res, nil
}
res.Status = domain.ValidationPassed
res.IsValid = true
return res, nil
default:
return domain.ValidationResult{}, fmt.Errorf("unsupported validation mode: %q", contract.ValidationMode)
}
}
func parseJSON(body []byte) (any, error) {
var v any
if err := json.Unmarshal(body, &v); err != nil {
@@ -132,6 +225,20 @@ func (v *StandardValidator) LoadSchemaDocument(ctx context.Context, schemaPath s
return doc, nil
}
func (v *FSValidator) LoadSchemaDocument(ctx context.Context, schemaPath string) (any, error) {
select {
case <-ctx.Done():
return nil, ctx.Err()
default:
}
_, doc, err := v.loadSchemaDocument(schemaPath)
if err != nil {
return nil, err
}
return doc, nil
}
func (v *StandardValidator) resolveSchemaPath(schemaPath string) (string, error) {
if strings.TrimSpace(schemaPath) == "" {
return "", errors.New("schema path is required for json_schema validation")
@@ -149,3 +256,70 @@ func (v *StandardValidator) resolveSchemaPath(schemaPath string) (string, error)
return resolved, nil
}
func (v *FSValidator) loadSchemaDocument(schemaPath string) (string, any, error) {
resolved, err := v.resolveSchemaPath(schemaPath)
if err != nil {
return "", nil, err
}
raw, err := fs.ReadFile(v.fsys, resolved)
if err != nil {
return "", nil, fmt.Errorf("failed to read schema file %q: %w", resolved, err)
}
var doc any
if err := json.Unmarshal(raw, &doc); err != nil {
return "", nil, fmt.Errorf("failed to decode JSON schema %q: %w", resolved, err)
}
return resolved, doc, nil
}
func (v *FSValidator) resolveSchemaPath(schemaPath string) (string, error) {
if strings.TrimSpace(schemaPath) == "" {
return "", errors.New("schema path is required for json_schema validation")
}
if v.fsys == nil {
return "", errors.New("schema filesystem is nil")
}
cleanRoot := cleanFSRoot(v.root)
rootInfo, err := fs.Stat(v.fsys, cleanRoot)
if err != nil {
return "", fmt.Errorf("failed to access schema source %q: %w", cleanRoot, err)
}
cleanSchemaPath := cleanSchemaFSPath(schemaPath)
var resolved string
if rootInfo.IsDir() {
resolved = path.Join(cleanRoot, cleanSchemaPath)
} else {
if cleanSchemaPath != path.Base(cleanRoot) {
return "", fmt.Errorf("schema path %q does not match schema file %q", cleanSchemaPath, path.Base(cleanRoot))
}
resolved = cleanRoot
}
if _, err := fs.Stat(v.fsys, resolved); err != nil {
return "", fmt.Errorf("failed to access schema file %q: %w", resolved, err)
}
return resolved, nil
}
func cleanSchemaFSPath(schemaPath string) string {
cleaned := strings.TrimSpace(schemaPath)
cleaned = strings.TrimPrefix(path.Clean(cleaned), "/")
return cleaned
}
func cleanFSRoot(root string) string {
root = strings.TrimSpace(root)
if root == "" || root == "." {
return "."
}
return strings.TrimPrefix(path.Clean(root), "/")
}
func fsSchemaResourceURL(schemaName string) string {
return "scriptorium-schema:///" + strings.TrimPrefix(path.Clean(schemaName), "/")
}

View File

@@ -5,6 +5,7 @@ import (
"os"
"path/filepath"
"testing"
"testing/fstest"
"gitea.maximumdirect.net/eric/scriptorium/internal/domain"
)
@@ -250,3 +251,77 @@ func TestStandardValidatorLoadSchemaDocumentInvalidJSON(t *testing.T) {
t.Fatal("expected decode error")
}
}
func TestFSValidatorJSONSchemaSuccess(t *testing.T) {
v := NewFSValidator(fstest.MapFS{
"schemas/events.schema.json": &fstest.MapFile{Data: []byte(`{
"$schema": "https://json-schema.org/draft/2020-12/schema",
"type": "object",
"required": ["events"],
"properties": {
"events": {"type": "array"}
}
}`)},
}, "schemas")
res, err := v.Validate(context.Background(), &domain.Artifact{Body: []byte(`{"events":[]}`)}, domain.OutputContract{
ValidationMode: domain.ValidationJSONSchema,
SchemaPath: "events.schema.json",
})
if err != nil {
t.Fatalf("expected no error, got %v", err)
}
if res.Status != domain.ValidationPassed || !res.IsValid {
t.Fatalf("expected passed/valid, got status=%q valid=%v", res.Status, res.IsValid)
}
}
func TestFSValidatorSingleSchemaFileUsesBaseName(t *testing.T) {
v := NewFSValidator(fstest.MapFS{
"events.schema.json": &fstest.MapFile{Data: []byte(`{
"type": "object",
"required": ["events"],
"properties": {
"events": {"type": "array"}
}
}`)},
}, "events.schema.json")
res, err := v.Validate(context.Background(), &domain.Artifact{Body: []byte(`{"events":[]}`)}, domain.OutputContract{
ValidationMode: domain.ValidationJSONSchema,
SchemaPath: "events.schema.json",
})
if err != nil {
t.Fatalf("expected no error, got %v", err)
}
if res.Status != domain.ValidationPassed || !res.IsValid {
t.Fatalf("expected passed/valid, got status=%q valid=%v", res.Status, res.IsValid)
}
_, err = v.Validate(context.Background(), &domain.Artifact{Body: []byte(`{"events":[]}`)}, domain.OutputContract{
ValidationMode: domain.ValidationJSONSchema,
SchemaPath: "other.schema.json",
})
if err == nil {
t.Fatal("expected schema path mismatch error")
}
}
func TestFSValidatorLoadSchemaDocument(t *testing.T) {
v := NewFSValidator(fstest.MapFS{
"schemas/schema.json": &fstest.MapFile{Data: []byte(`{"type":"object"}`)},
}, "schemas")
loader, ok := v.(SchemaDocumentLoader)
if !ok {
t.Fatal("fs validator must implement SchemaDocumentLoader")
}
doc, err := loader.LoadSchemaDocument(context.Background(), "schema.json")
if err != nil {
t.Fatalf("expected no error, got %v", err)
}
obj, ok := doc.(map[string]any)
if !ok || obj["type"] != "object" {
t.Fatalf("unexpected schema document: %#v", doc)
}
}