Implemented support for loading configuration from nested subdirectories
This commit is contained in:
@@ -63,7 +63,9 @@ Validation behavior:
|
|||||||
|
|
||||||
## Prompt Definition Files
|
## Prompt Definition Files
|
||||||
|
|
||||||
Prompt definitions are YAML files in `prompt_dir`.
|
Prompt definitions are YAML files anywhere under `prompt_dir`, including nested subdirectories.
|
||||||
|
|
||||||
|
Subdirectories are organizational only. Callers still select prompts by the YAML `id`, not by file path. For example, `prompts/dnd/recap.yaml` may still declare `id: dnd.recap`, and callers use `--prompt dnd.recap`.
|
||||||
|
|
||||||
Example:
|
Example:
|
||||||
|
|
||||||
@@ -122,7 +124,9 @@ Message rules:
|
|||||||
|
|
||||||
- Repeated roles are allowed.
|
- Repeated roles are allowed.
|
||||||
- `content_file` is resolved relative to the prompt YAML file location.
|
- `content_file` is resolved relative to the prompt YAML file location.
|
||||||
|
- Nested prompt files keep the same relative `content_file` behavior; `./recap.user.md` next to `dnd/recap.yaml` resolves from `dnd/`.
|
||||||
- Prompt decoding is strict; unknown YAML fields are rejected.
|
- Prompt decoding is strict; unknown YAML fields are rejected.
|
||||||
|
- Duplicate prompt IDs are invalid. If multiple files declare the requested prompt ID, Scriptorium fails instead of choosing one.
|
||||||
|
|
||||||
`output` fields:
|
`output` fields:
|
||||||
|
|
||||||
@@ -138,7 +142,9 @@ Repair behavior boundary:
|
|||||||
|
|
||||||
## Profile Definition Files
|
## Profile Definition Files
|
||||||
|
|
||||||
Execution profiles are YAML files in `profile_dir`.
|
Execution profiles are YAML files anywhere under `profile_dir`, including nested subdirectories.
|
||||||
|
|
||||||
|
Subdirectories are organizational only. Callers still select profiles by the YAML `id`, not by file path. For example, `profiles/local/local-quality.yaml` may still declare `id: local-quality`, and callers use `--profile local-quality`.
|
||||||
|
|
||||||
Example:
|
Example:
|
||||||
|
|
||||||
@@ -171,6 +177,7 @@ Profile rules:
|
|||||||
- Profile decoding is strict; unknown YAML fields are rejected.
|
- Profile decoding is strict; unknown YAML fields are rejected.
|
||||||
- Raw `api_key` is rejected; use `api_key_env`.
|
- Raw `api_key` is rejected; use `api_key_env`.
|
||||||
- If `api_key_env` is set, that environment variable must be set when preparing/running.
|
- If `api_key_env` is set, that environment variable must be set when preparing/running.
|
||||||
|
- Duplicate profile IDs are invalid. If multiple files declare the requested profile ID, Scriptorium fails instead of choosing one.
|
||||||
|
|
||||||
Current outbound request behavior:
|
Current outbound request behavior:
|
||||||
|
|
||||||
@@ -184,8 +191,9 @@ Schemas are JSON files, typically in `schema_dir`.
|
|||||||
Rules:
|
Rules:
|
||||||
|
|
||||||
- `output.validation_mode: json_schema` requires `output.schema_path`.
|
- `output.validation_mode: json_schema` requires `output.schema_path`.
|
||||||
- Relative `schema_path` values resolve from `schema_dir`.
|
- Relative `schema_path` values resolve from `schema_dir`, including explicit nested paths such as `dnd/structured_events.schema.json`.
|
||||||
- Absolute `schema_path` values are used directly.
|
- Absolute `schema_path` values are used directly.
|
||||||
|
- Scriptorium does not recursively search schemas by basename; nested schemas must be referenced by their relative path.
|
||||||
- Missing or invalid schema documents cause runtime validation errors.
|
- Missing or invalid schema documents cause runtime validation errors.
|
||||||
- Invalid generated JSON causes validation status `failed` (not a runtime error).
|
- Invalid generated JSON causes validation status `failed` (not a runtime error).
|
||||||
|
|
||||||
@@ -207,6 +215,14 @@ Supported artifact reference types for request inputs are `file` and `inline`.
|
|||||||
- Render example script: `examples/render-markdown-summary.sh`
|
- Render example script: `examples/render-markdown-summary.sh`
|
||||||
- HTTP request example: `examples/http-run.json`
|
- HTTP request example: `examples/http-run.json`
|
||||||
|
|
||||||
|
Example organizational layout:
|
||||||
|
|
||||||
|
```text
|
||||||
|
examples/prompts/dnd/recap.yaml
|
||||||
|
examples/profiles/local/local-quality.yaml
|
||||||
|
examples/schemas/dnd/structured_events.schema.json
|
||||||
|
```
|
||||||
|
|
||||||
## Integration References
|
## Integration References
|
||||||
|
|
||||||
- [Inbound HTTP contract](integrations/http-api.md)
|
- [Inbound HTTP contract](integrations/http-api.md)
|
||||||
|
|||||||
@@ -30,7 +30,7 @@ HTTP adapter:
|
|||||||
|
|
||||||
Filesystem repositories:
|
Filesystem repositories:
|
||||||
|
|
||||||
- Input: prompt/profile YAML files.
|
- Input: prompt/profile YAML files under configured directories.
|
||||||
- Output: normalized domain definitions/profiles or typed errors.
|
- Output: normalized domain definitions/profiles or typed errors.
|
||||||
|
|
||||||
Artifact reader:
|
Artifact reader:
|
||||||
@@ -79,6 +79,9 @@ Execution profile/request settings used through runner:
|
|||||||
Strict decoding and input checks:
|
Strict decoding and input checks:
|
||||||
|
|
||||||
- config/prompt/profile loaders reject unknown YAML fields.
|
- config/prompt/profile loaders reject unknown YAML fields.
|
||||||
|
- prompt/profile repositories scan nested subdirectories recursively.
|
||||||
|
- prompt/profile lookup uses YAML `id` values; subdirectory paths are organizational only.
|
||||||
|
- duplicate prompt/profile IDs are invalid and fail instead of using first-match behavior.
|
||||||
- HTTP DTO decoder rejects unknown JSON fields.
|
- HTTP DTO decoder rejects unknown JSON fields.
|
||||||
- raw API key payload fields are rejected by strict decoding in profile/http paths.
|
- raw API key payload fields are rejected by strict decoding in profile/http paths.
|
||||||
|
|
||||||
@@ -97,6 +100,7 @@ Validator:
|
|||||||
|
|
||||||
- `basic`, `json`, `json_schema` content failures return `ValidationFailed` results.
|
- `basic`, `json`, `json_schema` content failures return `ValidationFailed` results.
|
||||||
- schema load/compile/path failures are runtime errors.
|
- schema load/compile/path failures are runtime errors.
|
||||||
|
- schema lookup uses explicit `schema_path` values relative to `schema_dir`; it does not recursively search by basename.
|
||||||
|
|
||||||
HTTP error mapping:
|
HTTP error mapping:
|
||||||
|
|
||||||
|
|||||||
@@ -7,6 +7,7 @@ import (
|
|||||||
"fmt"
|
"fmt"
|
||||||
"os"
|
"os"
|
||||||
"path/filepath"
|
"path/filepath"
|
||||||
|
"sort"
|
||||||
"strings"
|
"strings"
|
||||||
|
|
||||||
"gitea.maximumdirect.net/eric/scriptorium/internal/domain"
|
"gitea.maximumdirect.net/eric/scriptorium/internal/domain"
|
||||||
@@ -33,40 +34,39 @@ func (r *filesystemRepository) GetProfile(ctx context.Context, id string) (*doma
|
|||||||
return nil, fmt.Errorf("%w: profile id is required", ErrInvalidProfile)
|
return nil, fmt.Errorf("%w: profile id is required", ErrInvalidProfile)
|
||||||
}
|
}
|
||||||
|
|
||||||
files, err := os.ReadDir(r.dir)
|
files, err := r.yamlFiles(ctx)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return nil, fmt.Errorf("failed to read profile directory: %w", err)
|
return nil, fmt.Errorf("failed to read profile directory: %w", err)
|
||||||
}
|
}
|
||||||
|
|
||||||
for _, file := range files {
|
var matches []profileMatch
|
||||||
|
for _, fullPath := range files {
|
||||||
select {
|
select {
|
||||||
case <-ctx.Done():
|
case <-ctx.Done():
|
||||||
return nil, ctx.Err()
|
return nil, ctx.Err()
|
||||||
default:
|
default:
|
||||||
}
|
}
|
||||||
|
|
||||||
if file.IsDir() || (!strings.HasSuffix(file.Name(), ".yaml") && !strings.HasSuffix(file.Name(), ".yml")) {
|
relPath := r.relativePath(fullPath)
|
||||||
continue
|
fileMatch := profileIDFromFileName(filepath.Base(fullPath)) == id
|
||||||
}
|
|
||||||
|
|
||||||
fullPath := filepath.Join(r.dir, file.Name())
|
|
||||||
data, err := os.ReadFile(fullPath)
|
data, err := os.ReadFile(fullPath)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return nil, fmt.Errorf("failed to read profile file %s: %w", file.Name(), err)
|
return nil, fmt.Errorf("failed to read profile file %s: %w", relPath, err)
|
||||||
}
|
}
|
||||||
|
|
||||||
var prof domain.ExecutionProfile
|
var prof domain.ExecutionProfile
|
||||||
decoder := yaml.NewDecoder(bytes.NewReader(data))
|
decoder := yaml.NewDecoder(bytes.NewReader(data))
|
||||||
decoder.KnownFields(true)
|
decoder.KnownFields(true)
|
||||||
if err := decoder.Decode(&prof); err != nil {
|
if err := decoder.Decode(&prof); err != nil {
|
||||||
|
idMatch := fileMatch || profileFileHasID(data, id)
|
||||||
if strings.Contains(err.Error(), "field api_key not found") {
|
if strings.Contains(err.Error(), "field api_key not found") {
|
||||||
if strings.TrimSuffix(strings.TrimSuffix(file.Name(), ".yaml"), ".yml") == id {
|
if idMatch {
|
||||||
return nil, fmt.Errorf("%w: %s", ErrRawAPIKeyNotAllowed, file.Name())
|
return nil, fmt.Errorf("%w: %s", ErrRawAPIKeyNotAllowed, relPath)
|
||||||
}
|
}
|
||||||
continue
|
continue
|
||||||
}
|
}
|
||||||
if strings.TrimSuffix(strings.TrimSuffix(file.Name(), ".yaml"), ".yml") == id {
|
if idMatch {
|
||||||
return nil, fmt.Errorf("%w: %s: %v", ErrInvalidYAML, file.Name(), err)
|
return nil, fmt.Errorf("%w: %s: %v", ErrInvalidYAML, relPath, err)
|
||||||
}
|
}
|
||||||
continue
|
continue
|
||||||
}
|
}
|
||||||
@@ -76,16 +76,88 @@ func (r *filesystemRepository) GetProfile(ctx context.Context, id string) (*doma
|
|||||||
}
|
}
|
||||||
if err := validateProfile(&prof); err != nil {
|
if err := validateProfile(&prof); err != nil {
|
||||||
if errors.Is(err, ErrRawAPIKeyNotAllowed) {
|
if errors.Is(err, ErrRawAPIKeyNotAllowed) {
|
||||||
return nil, fmt.Errorf("%w: %s", err, file.Name())
|
return nil, fmt.Errorf("%w: %s", err, relPath)
|
||||||
}
|
}
|
||||||
return nil, fmt.Errorf("%w: %s: %v", ErrInvalidProfile, file.Name(), err)
|
return nil, fmt.Errorf("%w: %s: %v", ErrInvalidProfile, relPath, err)
|
||||||
}
|
}
|
||||||
return &prof, nil
|
matches = append(matches, profileMatch{
|
||||||
|
profile: &prof,
|
||||||
|
path: relPath,
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
if len(matches) > 1 {
|
||||||
|
paths := make([]string, 0, len(matches))
|
||||||
|
for _, match := range matches {
|
||||||
|
paths = append(paths, match.path)
|
||||||
|
}
|
||||||
|
return nil, fmt.Errorf("%w: duplicate execution profile id %q found in: %s", ErrInvalidProfile, id, strings.Join(paths, ", "))
|
||||||
|
}
|
||||||
|
|
||||||
|
if len(matches) == 1 {
|
||||||
|
return matches[0].profile, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
return nil, ErrProfileNotFound
|
return nil, ErrProfileNotFound
|
||||||
}
|
}
|
||||||
|
|
||||||
|
type profileMatch struct {
|
||||||
|
profile *domain.ExecutionProfile
|
||||||
|
path string
|
||||||
|
}
|
||||||
|
|
||||||
|
func (r *filesystemRepository) yamlFiles(ctx context.Context) ([]string, error) {
|
||||||
|
var files []string
|
||||||
|
err := filepath.WalkDir(r.dir, func(path string, d os.DirEntry, err error) error {
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
select {
|
||||||
|
case <-ctx.Done():
|
||||||
|
return ctx.Err()
|
||||||
|
default:
|
||||||
|
}
|
||||||
|
if d.IsDir() {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
if !isYAMLFile(d.Name()) {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
files = append(files, path)
|
||||||
|
return nil
|
||||||
|
})
|
||||||
|
sort.Strings(files)
|
||||||
|
return files, err
|
||||||
|
}
|
||||||
|
|
||||||
|
func (r *filesystemRepository) relativePath(path string) string {
|
||||||
|
rel, err := filepath.Rel(r.dir, path)
|
||||||
|
if err != nil {
|
||||||
|
return filepath.Clean(path)
|
||||||
|
}
|
||||||
|
return filepath.Clean(rel)
|
||||||
|
}
|
||||||
|
|
||||||
|
func isYAMLFile(name string) bool {
|
||||||
|
return strings.HasSuffix(name, ".yaml") || strings.HasSuffix(name, ".yml")
|
||||||
|
}
|
||||||
|
|
||||||
|
func profileIDFromFileName(name string) string {
|
||||||
|
name = strings.TrimSuffix(name, ".yaml")
|
||||||
|
name = strings.TrimSuffix(name, ".yml")
|
||||||
|
return name
|
||||||
|
}
|
||||||
|
|
||||||
|
func profileFileHasID(data []byte, id string) bool {
|
||||||
|
var raw struct {
|
||||||
|
ID string `yaml:"id"`
|
||||||
|
}
|
||||||
|
if err := yaml.NewDecoder(bytes.NewReader(data)).Decode(&raw); err != nil {
|
||||||
|
return false
|
||||||
|
}
|
||||||
|
return strings.TrimSpace(raw.ID) == id
|
||||||
|
}
|
||||||
|
|
||||||
func validateProfile(p *domain.ExecutionProfile) error {
|
func validateProfile(p *domain.ExecutionProfile) error {
|
||||||
if strings.TrimSpace(p.ID) == "" {
|
if strings.TrimSpace(p.ID) == "" {
|
||||||
return errors.New("id is required")
|
return errors.New("id is required")
|
||||||
|
|||||||
@@ -5,6 +5,7 @@ import (
|
|||||||
"errors"
|
"errors"
|
||||||
"os"
|
"os"
|
||||||
"path/filepath"
|
"path/filepath"
|
||||||
|
"strings"
|
||||||
"testing"
|
"testing"
|
||||||
)
|
)
|
||||||
|
|
||||||
@@ -60,6 +61,75 @@ func TestFilesystemRepository_GetProfile(t *testing.T) {
|
|||||||
}
|
}
|
||||||
})
|
})
|
||||||
|
|
||||||
|
t.Run("valid nested profile", func(t *testing.T) {
|
||||||
|
nestedDir := filepath.Join(tmpDir, "local")
|
||||||
|
if err := os.MkdirAll(nestedDir, 0o755); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
writeProfileTestFile(t, filepath.Join(nestedDir, "nested-local.yaml"), `
|
||||||
|
id: nested-local
|
||||||
|
endpoint: http://localhost:8000/v1
|
||||||
|
model: nested-model
|
||||||
|
temperature: 0.1
|
||||||
|
`)
|
||||||
|
|
||||||
|
p, err := repo.GetProfile(ctx, "nested-local")
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("expected no error, got %v", err)
|
||||||
|
}
|
||||||
|
if p.Model != "nested-model" {
|
||||||
|
t.Fatalf("unexpected model: %q", p.Model)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
|
||||||
|
t.Run("duplicate profile IDs fail as ambiguous", func(t *testing.T) {
|
||||||
|
writeProfileTestFile(t, filepath.Join(tmpDir, "duplicate-profile-a.yaml"), `
|
||||||
|
id: duplicate-profile
|
||||||
|
endpoint: http://localhost:8000/v1
|
||||||
|
model: first-model
|
||||||
|
`)
|
||||||
|
nestedDir := filepath.Join(tmpDir, "duplicates")
|
||||||
|
if err := os.MkdirAll(nestedDir, 0o755); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
writeProfileTestFile(t, filepath.Join(nestedDir, "duplicate-profile-b.yaml"), `
|
||||||
|
id: duplicate-profile
|
||||||
|
endpoint: http://localhost:8000/v1
|
||||||
|
model: second-model
|
||||||
|
`)
|
||||||
|
|
||||||
|
_, err := repo.GetProfile(ctx, "duplicate-profile")
|
||||||
|
if !errors.Is(err, ErrInvalidProfile) {
|
||||||
|
t.Fatalf("expected duplicate profile to return ErrInvalidProfile, got %v", err)
|
||||||
|
}
|
||||||
|
for _, want := range []string{"duplicate execution profile id", "duplicate-profile-a.yaml", filepath.Join("duplicates", "duplicate-profile-b.yaml")} {
|
||||||
|
if !strings.Contains(err.Error(), want) {
|
||||||
|
t.Fatalf("expected error to contain %q, got %v", want, err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
})
|
||||||
|
|
||||||
|
t.Run("nested raw api_key rejected for likely target file", func(t *testing.T) {
|
||||||
|
nestedDir := filepath.Join(tmpDir, "secure")
|
||||||
|
if err := os.MkdirAll(nestedDir, 0o755); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
writeProfileTestFile(t, filepath.Join(nestedDir, "not_named_like_id.yaml"), `
|
||||||
|
id: nested_raw_api_key
|
||||||
|
endpoint: http://localhost:8000/v1
|
||||||
|
model: m
|
||||||
|
api_key: secret
|
||||||
|
`)
|
||||||
|
|
||||||
|
_, err := repo.GetProfile(ctx, "nested_raw_api_key")
|
||||||
|
if !errors.Is(err, ErrRawAPIKeyNotAllowed) {
|
||||||
|
t.Fatalf("expected ErrRawAPIKeyNotAllowed, got %v", err)
|
||||||
|
}
|
||||||
|
if !strings.Contains(err.Error(), filepath.Join("secure", "not_named_like_id.yaml")) {
|
||||||
|
t.Fatalf("expected nested path in error, got %v", err)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
|
||||||
t.Run("invalid yaml", func(t *testing.T) {
|
t.Run("invalid yaml", func(t *testing.T) {
|
||||||
_, err := repo.GetProfile(ctx, "invalid_yaml")
|
_, err := repo.GetProfile(ctx, "invalid_yaml")
|
||||||
if !errors.Is(err, ErrInvalidYAML) {
|
if !errors.Is(err, ErrInvalidYAML) {
|
||||||
@@ -109,3 +179,10 @@ func TestFilesystemRepository_GetProfile(t *testing.T) {
|
|||||||
}
|
}
|
||||||
})
|
})
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func writeProfileTestFile(t *testing.T, path string, content string) {
|
||||||
|
t.Helper()
|
||||||
|
if err := os.WriteFile(path, []byte(strings.TrimLeft(content, "\n")), 0o644); err != nil {
|
||||||
|
t.Fatalf("failed to write profile test file %q: %v", path, err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|||||||
@@ -7,6 +7,7 @@ import (
|
|||||||
"fmt"
|
"fmt"
|
||||||
"os"
|
"os"
|
||||||
"path/filepath"
|
"path/filepath"
|
||||||
|
"sort"
|
||||||
"strings"
|
"strings"
|
||||||
|
|
||||||
"gitea.maximumdirect.net/eric/scriptorium/internal/domain"
|
"gitea.maximumdirect.net/eric/scriptorium/internal/domain"
|
||||||
@@ -62,29 +63,26 @@ func (r *filesystemRepository) GetPromptDefinition(ctx context.Context, id strin
|
|||||||
return nil, fmt.Errorf("%w: prompt id is required", ErrInvalidPromptDefinition)
|
return nil, fmt.Errorf("%w: prompt id is required", ErrInvalidPromptDefinition)
|
||||||
}
|
}
|
||||||
|
|
||||||
files, err := os.ReadDir(r.dir)
|
files, err := r.yamlFiles(ctx)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return nil, fmt.Errorf("failed to read prompt definition directory: %w", err)
|
return nil, fmt.Errorf("failed to read prompt definition directory: %w", err)
|
||||||
}
|
}
|
||||||
|
|
||||||
for _, file := range files {
|
var matches []promptDefinitionMatch
|
||||||
|
for _, fullPath := range files {
|
||||||
select {
|
select {
|
||||||
case <-ctx.Done():
|
case <-ctx.Done():
|
||||||
return nil, ctx.Err()
|
return nil, ctx.Err()
|
||||||
default:
|
default:
|
||||||
}
|
}
|
||||||
|
|
||||||
if file.IsDir() || !isYAMLFile(file.Name()) {
|
relPath := r.relativePath(fullPath)
|
||||||
continue
|
fileMatch := promptIDFromFileName(filepath.Base(fullPath)) == id
|
||||||
}
|
|
||||||
|
|
||||||
fullPath := filepath.Join(r.dir, file.Name())
|
|
||||||
fileMatch := promptIDFromFileName(file.Name()) == id
|
|
||||||
|
|
||||||
raw, err := loadPromptDefinitionFile(fullPath)
|
raw, err := loadPromptDefinitionFile(fullPath)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
if fileMatch {
|
if fileMatch || promptDefinitionFileHasID(fullPath, id) {
|
||||||
return nil, fmt.Errorf("%w: %s: %v", ErrInvalidYAML, file.Name(), err)
|
return nil, fmt.Errorf("%w: %s: %v", ErrInvalidYAML, relPath, err)
|
||||||
}
|
}
|
||||||
continue
|
continue
|
||||||
}
|
}
|
||||||
@@ -92,7 +90,7 @@ func (r *filesystemRepository) GetPromptDefinition(ctx context.Context, id strin
|
|||||||
def, err := normalizePromptDefinition(raw, fullPath)
|
def, err := normalizePromptDefinition(raw, fullPath)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
if fileMatch || strings.TrimSpace(raw.ID) == id {
|
if fileMatch || strings.TrimSpace(raw.ID) == id {
|
||||||
return nil, fmt.Errorf("%w: %s: %v", ErrInvalidPromptDefinition, file.Name(), err)
|
return nil, fmt.Errorf("%w: %s: %v", ErrInvalidPromptDefinition, relPath, err)
|
||||||
}
|
}
|
||||||
continue
|
continue
|
||||||
}
|
}
|
||||||
@@ -103,12 +101,67 @@ func (r *filesystemRepository) GetPromptDefinition(ctx context.Context, id strin
|
|||||||
if version != "" && def.Version != version {
|
if version != "" && def.Version != version {
|
||||||
continue
|
continue
|
||||||
}
|
}
|
||||||
return def, nil
|
matches = append(matches, promptDefinitionMatch{
|
||||||
|
def: def,
|
||||||
|
path: relPath,
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
if len(matches) > 1 {
|
||||||
|
paths := make([]string, 0, len(matches))
|
||||||
|
for _, match := range matches {
|
||||||
|
paths = append(paths, match.path)
|
||||||
|
}
|
||||||
|
if version != "" {
|
||||||
|
return nil, fmt.Errorf("%w: duplicate prompt definition id %q version %q found in: %s", ErrInvalidPromptDefinition, id, version, strings.Join(paths, ", "))
|
||||||
|
}
|
||||||
|
return nil, fmt.Errorf("%w: duplicate prompt definition id %q found in: %s", ErrInvalidPromptDefinition, id, strings.Join(paths, ", "))
|
||||||
|
}
|
||||||
|
|
||||||
|
if len(matches) == 1 {
|
||||||
|
return matches[0].def, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
return nil, ErrPromptDefinitionNotFound
|
return nil, ErrPromptDefinitionNotFound
|
||||||
}
|
}
|
||||||
|
|
||||||
|
type promptDefinitionMatch struct {
|
||||||
|
def *domain.PromptDefinition
|
||||||
|
path string
|
||||||
|
}
|
||||||
|
|
||||||
|
func (r *filesystemRepository) yamlFiles(ctx context.Context) ([]string, error) {
|
||||||
|
var files []string
|
||||||
|
err := filepath.WalkDir(r.dir, func(path string, d os.DirEntry, err error) error {
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
select {
|
||||||
|
case <-ctx.Done():
|
||||||
|
return ctx.Err()
|
||||||
|
default:
|
||||||
|
}
|
||||||
|
if d.IsDir() {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
if !isYAMLFile(d.Name()) {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
files = append(files, path)
|
||||||
|
return nil
|
||||||
|
})
|
||||||
|
sort.Strings(files)
|
||||||
|
return files, err
|
||||||
|
}
|
||||||
|
|
||||||
|
func (r *filesystemRepository) relativePath(path string) string {
|
||||||
|
rel, err := filepath.Rel(r.dir, path)
|
||||||
|
if err != nil {
|
||||||
|
return filepath.Clean(path)
|
||||||
|
}
|
||||||
|
return filepath.Clean(rel)
|
||||||
|
}
|
||||||
|
|
||||||
func loadPromptDefinitionFile(path string) (*promptDefinitionFile, error) {
|
func loadPromptDefinitionFile(path string) (*promptDefinitionFile, error) {
|
||||||
data, err := os.ReadFile(path)
|
data, err := os.ReadFile(path)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
@@ -124,6 +177,20 @@ func loadPromptDefinitionFile(path string) (*promptDefinitionFile, error) {
|
|||||||
return &raw, nil
|
return &raw, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func promptDefinitionFileHasID(path string, id string) bool {
|
||||||
|
data, err := os.ReadFile(path)
|
||||||
|
if err != nil {
|
||||||
|
return false
|
||||||
|
}
|
||||||
|
var raw struct {
|
||||||
|
ID string `yaml:"id"`
|
||||||
|
}
|
||||||
|
if err := yaml.NewDecoder(bytes.NewReader(data)).Decode(&raw); err != nil {
|
||||||
|
return false
|
||||||
|
}
|
||||||
|
return strings.TrimSpace(raw.ID) == id
|
||||||
|
}
|
||||||
|
|
||||||
func normalizePromptDefinition(raw *promptDefinitionFile, sourcePath string) (*domain.PromptDefinition, error) {
|
func normalizePromptDefinition(raw *promptDefinitionFile, sourcePath string) (*domain.PromptDefinition, error) {
|
||||||
if raw == nil {
|
if raw == nil {
|
||||||
return nil, errors.New("prompt definition is nil")
|
return nil, errors.New("prompt definition is nil")
|
||||||
|
|||||||
@@ -68,6 +68,39 @@ func TestFilesystemRepository_GetPromptDefinition(t *testing.T) {
|
|||||||
}
|
}
|
||||||
})
|
})
|
||||||
|
|
||||||
|
t.Run("valid nested file-backed prompt resolves content file relative to nested YAML", func(t *testing.T) {
|
||||||
|
nestedDir := filepath.Join(tmpDir, "dnd", "recap")
|
||||||
|
if err := os.MkdirAll(nestedDir, 0o755); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
writePromptTestFile(t, filepath.Join(nestedDir, "nested_recap.yaml"), `
|
||||||
|
id: nested-recap
|
||||||
|
version: "1.0.0"
|
||||||
|
messages:
|
||||||
|
- role: user
|
||||||
|
content_file: ./nested_recap.user.tmpl
|
||||||
|
output:
|
||||||
|
format: markdown
|
||||||
|
validation_mode: basic
|
||||||
|
repair_attempts: 0
|
||||||
|
`)
|
||||||
|
writePromptTestFile(t, filepath.Join(nestedDir, "nested_recap.user.tmpl"), `Nested recap: {{input "transcript"}}`)
|
||||||
|
|
||||||
|
p, err := repo.GetPromptDefinition(ctx, "nested-recap", "")
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("expected no error, got %v", err)
|
||||||
|
}
|
||||||
|
if len(p.Templates) != 1 {
|
||||||
|
t.Fatalf("expected one template, got %d", len(p.Templates))
|
||||||
|
}
|
||||||
|
if !strings.Contains(p.Templates[0].Content, "Nested recap") {
|
||||||
|
t.Fatalf("expected nested content file body, got %q", p.Templates[0].Content)
|
||||||
|
}
|
||||||
|
if !strings.Contains(p.Templates[0].ContentFile, filepath.Join("dnd", "recap", "nested_recap.user.tmpl")) {
|
||||||
|
t.Fatalf("expected nested content file path, got %q", p.Templates[0].ContentFile)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
|
||||||
t.Run("prompt with default_profile", func(t *testing.T) {
|
t.Run("prompt with default_profile", func(t *testing.T) {
|
||||||
p, err := repo.GetPromptDefinition(ctx, "with-default-profile", "")
|
p, err := repo.GetPromptDefinition(ctx, "with-default-profile", "")
|
||||||
if err != nil {
|
if err != nil {
|
||||||
@@ -84,6 +117,124 @@ func TestFilesystemRepository_GetPromptDefinition(t *testing.T) {
|
|||||||
}
|
}
|
||||||
})
|
})
|
||||||
|
|
||||||
|
t.Run("duplicate prompt IDs fail as ambiguous", func(t *testing.T) {
|
||||||
|
writePromptTestFile(t, filepath.Join(tmpDir, "duplicate_a.yaml"), `
|
||||||
|
id: duplicate-prompt
|
||||||
|
version: "1.0.0"
|
||||||
|
messages:
|
||||||
|
- role: user
|
||||||
|
content: First duplicate.
|
||||||
|
output:
|
||||||
|
format: markdown
|
||||||
|
validation_mode: basic
|
||||||
|
repair_attempts: 0
|
||||||
|
`)
|
||||||
|
nestedDir := filepath.Join(tmpDir, "nested")
|
||||||
|
if err := os.MkdirAll(nestedDir, 0o755); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
writePromptTestFile(t, filepath.Join(nestedDir, "duplicate_b.yaml"), `
|
||||||
|
id: duplicate-prompt
|
||||||
|
version: "2.0.0"
|
||||||
|
messages:
|
||||||
|
- role: user
|
||||||
|
content: Second duplicate.
|
||||||
|
output:
|
||||||
|
format: markdown
|
||||||
|
validation_mode: basic
|
||||||
|
repair_attempts: 0
|
||||||
|
`)
|
||||||
|
|
||||||
|
_, err := repo.GetPromptDefinition(ctx, "duplicate-prompt", "")
|
||||||
|
if !errors.Is(err, ErrInvalidPromptDefinition) {
|
||||||
|
t.Fatalf("expected duplicate prompt to return ErrInvalidPromptDefinition, got %v", err)
|
||||||
|
}
|
||||||
|
for _, want := range []string{"duplicate prompt definition id", "duplicate_a.yaml", filepath.Join("nested", "duplicate_b.yaml")} {
|
||||||
|
if !strings.Contains(err.Error(), want) {
|
||||||
|
t.Fatalf("expected error to contain %q, got %v", want, err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
})
|
||||||
|
|
||||||
|
t.Run("duplicate prompt ID and requested version fails as ambiguous", func(t *testing.T) {
|
||||||
|
writePromptTestFile(t, filepath.Join(tmpDir, "version_duplicate_a.yaml"), `
|
||||||
|
id: duplicate-version-prompt
|
||||||
|
version: "1.0.0"
|
||||||
|
messages:
|
||||||
|
- role: user
|
||||||
|
content: First duplicate version.
|
||||||
|
output:
|
||||||
|
format: markdown
|
||||||
|
validation_mode: basic
|
||||||
|
repair_attempts: 0
|
||||||
|
`)
|
||||||
|
nestedDir := filepath.Join(tmpDir, "versioned")
|
||||||
|
if err := os.MkdirAll(nestedDir, 0o755); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
writePromptTestFile(t, filepath.Join(nestedDir, "version_duplicate_b.yaml"), `
|
||||||
|
id: duplicate-version-prompt
|
||||||
|
version: "1.0.0"
|
||||||
|
messages:
|
||||||
|
- role: user
|
||||||
|
content: Second duplicate version.
|
||||||
|
output:
|
||||||
|
format: markdown
|
||||||
|
validation_mode: basic
|
||||||
|
repair_attempts: 0
|
||||||
|
`)
|
||||||
|
|
||||||
|
_, err := repo.GetPromptDefinition(ctx, "duplicate-version-prompt", "1.0.0")
|
||||||
|
if !errors.Is(err, ErrInvalidPromptDefinition) {
|
||||||
|
t.Fatalf("expected duplicate prompt version to return ErrInvalidPromptDefinition, got %v", err)
|
||||||
|
}
|
||||||
|
for _, want := range []string{"duplicate prompt definition id", "version \"1.0.0\"", "version_duplicate_a.yaml", filepath.Join("versioned", "version_duplicate_b.yaml")} {
|
||||||
|
if !strings.Contains(err.Error(), want) {
|
||||||
|
t.Fatalf("expected error to contain %q, got %v", want, err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
})
|
||||||
|
|
||||||
|
t.Run("non-matching malformed nested prompt is ignored for not found lookup", func(t *testing.T) {
|
||||||
|
nestedDir := filepath.Join(tmpDir, "broken")
|
||||||
|
if err := os.MkdirAll(nestedDir, 0o755); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
writePromptTestFile(t, filepath.Join(nestedDir, "unrelated.yaml"), "id: [")
|
||||||
|
|
||||||
|
_, err := repo.GetPromptDefinition(ctx, "does-not-exist-even-with-broken-nested-file", "")
|
||||||
|
if !errors.Is(err, ErrPromptDefinitionNotFound) {
|
||||||
|
t.Fatalf("expected ErrPromptDefinitionNotFound, got %v", err)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
|
||||||
|
t.Run("strict decode failure in nested prompt matches by YAML ID", func(t *testing.T) {
|
||||||
|
nestedDir := filepath.Join(tmpDir, "strict")
|
||||||
|
if err := os.MkdirAll(nestedDir, 0o755); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
writePromptTestFile(t, filepath.Join(nestedDir, "not_named_like_id.yaml"), `
|
||||||
|
id: nested-strict-error
|
||||||
|
version: "1.0.0"
|
||||||
|
unknown_field: true
|
||||||
|
messages:
|
||||||
|
- role: user
|
||||||
|
content: Invalid because of unknown field.
|
||||||
|
output:
|
||||||
|
format: markdown
|
||||||
|
validation_mode: basic
|
||||||
|
repair_attempts: 0
|
||||||
|
`)
|
||||||
|
|
||||||
|
_, err := repo.GetPromptDefinition(ctx, "nested-strict-error", "")
|
||||||
|
if !errors.Is(err, ErrInvalidYAML) {
|
||||||
|
t.Fatalf("expected ErrInvalidYAML, got %v", err)
|
||||||
|
}
|
||||||
|
if !strings.Contains(err.Error(), filepath.Join("strict", "not_named_like_id.yaml")) {
|
||||||
|
t.Fatalf("expected nested path in error, got %v", err)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
|
||||||
t.Run("version lookup", func(t *testing.T) {
|
t.Run("version lookup", func(t *testing.T) {
|
||||||
_, err := repo.GetPromptDefinition(ctx, "valid-inline", "9.9.9")
|
_, err := repo.GetPromptDefinition(ctx, "valid-inline", "9.9.9")
|
||||||
if !errors.Is(err, ErrPromptDefinitionNotFound) {
|
if !errors.Is(err, ErrPromptDefinitionNotFound) {
|
||||||
@@ -131,6 +282,13 @@ func TestFilesystemRepository_GetPromptDefinition(t *testing.T) {
|
|||||||
})
|
})
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func writePromptTestFile(t *testing.T, path string, content string) {
|
||||||
|
t.Helper()
|
||||||
|
if err := os.WriteFile(path, []byte(strings.TrimLeft(content, "\n")), 0o644); err != nil {
|
||||||
|
t.Fatalf("failed to write prompt test file %q: %v", path, err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
func copyTree(src, dst string) error {
|
func copyTree(src, dst string) error {
|
||||||
return filepath.WalkDir(src, func(path string, d fs.DirEntry, err error) error {
|
return filepath.WalkDir(src, func(path string, d fs.DirEntry, err error) error {
|
||||||
if err != nil {
|
if err != nil {
|
||||||
|
|||||||
@@ -116,6 +116,49 @@ func TestStandardValidatorJSONSchemaSuccess(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func TestStandardValidatorJSONSchemaNestedSchemaPathSuccess(t *testing.T) {
|
||||||
|
tmp := t.TempDir()
|
||||||
|
nestedDir := filepath.Join(tmp, "dnd")
|
||||||
|
if err := os.MkdirAll(nestedDir, 0o755); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if err := os.WriteFile(filepath.Join(nestedDir, "schema.json"), []byte(`{
|
||||||
|
"$schema": "https://json-schema.org/draft/2020-12/schema",
|
||||||
|
"type": "object",
|
||||||
|
"required": ["name"],
|
||||||
|
"properties": {
|
||||||
|
"name": {"type": "string"}
|
||||||
|
}
|
||||||
|
}`), 0644); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
|
||||||
|
v := NewStandardValidator(tmp)
|
||||||
|
|
||||||
|
res, err := v.Validate(context.Background(), &domain.Artifact{Body: []byte(`{"name":"eris"}`)}, domain.OutputContract{
|
||||||
|
ValidationMode: domain.ValidationJSONSchema,
|
||||||
|
SchemaPath: filepath.Join("dnd", "schema.json"),
|
||||||
|
})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("expected no error, got %v", err)
|
||||||
|
}
|
||||||
|
if res.Status != domain.ValidationPassed || !res.IsValid {
|
||||||
|
t.Fatalf("expected passed/valid, got status=%q valid=%v", res.Status, res.IsValid)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestStandardValidatorJSONSchemaNestedSchemaPathMissing(t *testing.T) {
|
||||||
|
v := NewStandardValidator(t.TempDir())
|
||||||
|
|
||||||
|
_, err := v.Validate(context.Background(), &domain.Artifact{Body: []byte(`{"name":"eris"}`)}, domain.OutputContract{
|
||||||
|
ValidationMode: domain.ValidationJSONSchema,
|
||||||
|
SchemaPath: filepath.Join("dnd", "missing.json"),
|
||||||
|
})
|
||||||
|
if err == nil {
|
||||||
|
t.Fatal("expected nested schema load error")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
func TestStandardValidatorJSONSchemaFailure(t *testing.T) {
|
func TestStandardValidatorJSONSchemaFailure(t *testing.T) {
|
||||||
tmp := t.TempDir()
|
tmp := t.TempDir()
|
||||||
schemaPath := filepath.Join(tmp, "schema.json")
|
schemaPath := filepath.Join(tmp, "schema.json")
|
||||||
|
|||||||
Reference in New Issue
Block a user